Skip to content
conftrace_
2026 EMNLP EMNLP 2026

E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval

AbstractRetrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external evidence, but existing RAG systems remain poorly matched to visually rich documents. OCR-based methods discard layout and non-textual visual cues, and although vision-language models (VLMs) can better understand native visual information, existing visual retrieval approaches often compress entire pages into global embedding vectors, obscuring fine-grained spatial information such as table cells and local annotations. Meanwhile, discrete Top-K selection in cascaded architectures blocks gradient flow between retrieval and generation, preventing joint optimization.In this work, we propose E2E-VisRAG, an end-to-end visual RAG framework that operates directly on native visual patches. Building upon patch-level visual encoding, we introduce a differentiable Top-K selection mechanism to enable joint training of retrieval and generation, and employ a gated residual fusion module to inject retrieval-enhanced visual features while suppressing noise. This design abandons the traditional paradigms of relying on OCR, global embeddings, and retrieval–generation separation, maximizing the preservation of original document visual semantics. Experiments on Vidore and MMLongBench-Doc show that, under matched 3B generator capacity, E2E-VisRAG outperforms non-end-to-end visual retrieval baselines, improving both Recall@1 and Accuracy@1 over OCR-based, global-embedding, and patch-level retrieval methods.

More from EMNLP 2026

All EMNLP 2026 papers →