Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Longitudinal-aware Evaluation of Retrieval-Augmented Generation over Evolving Credit Policy Documents

AbstractWe present a retrieval-augmented generation (RAG) system in production since April 2026 for credit policy question answering, a setting in which the corpus is revised several times per quarter: a single revision changes thresholds, narrows scopes, and restructures tables. Static test sets silently decay under this regime, and scores computed against successive corpus versions cannot separate system drift from corpus drift. We propose a longitudinal-aware evaluation framework that makes RAG quality over an evolving corpus both measurable and attributable. It combines (i) a change-type taxonomy — Stable, Evolved, Added, Removed — built on a deterministic, model-independent alignment of document subsections keyed to how each gold-supporting provision changed between versions, making explicit which scores remain comparable across releases; (ii) a versioned, class-stratified synthetic QA pipeline that keeps the test set current at revision cadence; and (iii) a statement-level LLM judge for compound policy answers; each component is calibrated against subject-matter experts. Across three corpus versions, a conventional longitudinal-unaware protocol would report a recall decline of up to 25.7 points. However, paired comparisons on answer-invariant items show no evidence of regression in the unchanged system.

More from EMNLP 2026

All EMNLP 2026 papers →