Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Seeing Failure In Advance: Pre-Generation Failure Diagnosis for Multi-Modal Retrieval-Augmented-Generation

AbstractMultimodal retrieval-augmented generation (MMRAG) systems can fail even when relevant evidence is retrieved, exposing a gap between retrieval and generation. We formulate MMRAG failure diagnosis as an online single-fire decision problem over prefill hidden states, which is distinct from static probing of model internals or post-hoc self-correction. Across InfoSeek, E-VQA, and VISA, we show that hidden states encode both failure occurrence and failure type, with failure detection reaching around 90% accuracy and recall on InfoSeek and E-VQA. We further introduce a Fire-Then-Type prober that separates failure detection from failure typing, improving typed diagnosis over direct classification. Finally, our method matches the response accuracy of final-layer repair on InfoSeek and E-VQA while diagnosing before answer decoding. The available prefill depth varies substantially across models, from 4.6% to 24.3%, and is limited for Qwen3-VL and InternVL on VISA.

More from EMNLP 2026

All EMNLP 2026 papers →