Skip to content
conftrace_
2026 EMNLP EMNLP 2026

When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA

AbstractLLMs often fail to handle temporal knowledge conflicts–contradictions that arise when facts evolve over time within their training data. Existing benchmarks often rely on structured sources such as Wikidata, where changes are directly exposed through versioned snapshots, and rarely account for differences in model knowledge cut-offs, limiting fair evaluation across LLMs released at different times. We introduce a novel framework for automatically constructing temporal knowledge probing benchmarks from unstructured, time-stamped corpora by discovering evolving knowledge through salient entities, concepts, and attributes; and using this structure to generate grounded questions with gold answers aligned to model cut-off dates. Using this framework, we construct **evolveQA**, spanning 3 domains: AWS service updates, Azure platform changes, and WHO Disease Outbreak News reports. Evaluating 12 open- and closed-source LLMs across 3 probing formats, we find performance drops of up to 31% on **evolveQA** compared to static knowledge questions, showing that LLMs struggle to recall temporally updated facts even when such knowledge is available in their parameters.

More from EMNLP 2026

All EMNLP 2026 papers →