Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Grounded Decoding: Retrieval-Anchored Probability Fusion for Faithful RAG

AbstractAs retrieval-augmented generation (RAG) systems scale, ensuring faithful grounding in external evidence remains challenging. Large language models may still rely on parametric knowledge when it conflicts with retrieved information. We propose Grounded Decoding, a training-free decoding framework that improves factual consistency in RAG without modifying model parameters. At each generation step, our method constructs two matched-prompt distributions: a full RAG distribution conditioned on the query, retrieved documents, and generated prefix, and a retrieval-only distribution conditioned solely on the retrieved evidence and the same prefix. We derive the final next-token distribution as the unique solution to a KL-barycenter objective over the probability simplex, resulting in a normalized geometric fusion of the two distributions. The formulation recovers standard RAG when the grounding weight is zero and progressively shifts probability mass toward retrieved evidence as grounding strength increases. We further introduce a conflict-aware adaptive weighting scheme that adjusts grounding strength based on distributional disagreement and retriever confidence. Experiments on ALCE, Natural Questions, and FActScore show consistent improvements in factual accuracy and citation quality over standard RAG and competitive decoding-time baselines while maintaining fluency. These results suggest that probability-level fusion offers an effective and efficient alternative to logit-level intervention for faithful RAG decoding.

More from EMNLP 2026

All EMNLP 2026 papers →