Skip to content
conftrace_
2026 AACL AACL 2026

Failure Modes of Imperfect Relational Context in LLM Document Extraction

AbstractLLM-based extraction pipelines increasingly consume relational context produced upstream, for example by a heuristic linker, a retrieval step, or another model. We study how imperfect relations between document entities affect LLM extraction on FUNSD, a benchmark of scanned forms. Using identical instructions in every condition, four LLMs receive a form’s entities with no relations (Flat), relations from a heuristic linker (Heuristic), gold relations (Oracle), or relations that are almost all wrong (Deranged). In a secondary contrast, gold relations raise accuracy for every model (4.0 to 7.9 percentage points, pp). Heuristic and deranged relations lower accuracy by 8 to 25 pp for three models, while GPT-5 shows significantly smaller effects. Comparing the same questions across conditions shows what lies behind the drops. Wrong edges are associated with answers copied from a different field, and missing edges with more blank answers even though the answer text is still present. These patterns appear under two prompt variants, but the prompts were developed on this corpus and every result is a single generation, so we treat the prompt comparison as exploratory.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from AACL 2026

All AACL 2026 papers →