Skip to content
conftrace_
2026 EMNLP EMNLP 2026

FRED-SemBench: Evaluating Semantic Reliability in LLM Access to Macroeconomic Data

AbstractAccess to authoritative financial data does not ensure semantically correct answers: a language model may retrieve a valid observation associated with the wrong series, transformation, unit, or temporal interpretation. We introduce FRED-SemBench, a 200-question benchmark candidate over independently selected Federal Reserve Economic Data (FRED) concepts that separates retrieval failures from semantic data-access failures. We also instantiate Iris, an ontology-backed semantic layer whose versioned concept contracts bind user intent to provider series, transformations, units, temporal semantics, provenance, and declared coverage. A controlled comparison of an unaided model, FRED and Web access, and Iris reveals a reliability trade-off: raw retrieval substantially improves numerical answering while suppressing abstention and reducing clarification accuracy on underspecified queries. Within its pre-declared support set, Iris preserves the numerical benefit, restores clarification behavior, and attains the strongest strict accuracy, suggesting that this trade-off is not intrinsic to data access. These pilot findings motivate evaluating macroeconomic data access jointly in terms of semantic correctness, boundary behavior, coverage, and operational cost.

More from EMNLP 2026

All EMNLP 2026 papers →