Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Action-Level Credit Assignment for Retrieval-Augmented Reasoning

AbstractComplex question answering over large document collections requires models to identify missing information, issue sub-queries, and iteratively gather evidence before producing an answer. Reinforcement learning offers a natural framework for training such behaviour, but standard GRPO-style training provides only a sparse final-answer reward and assigns the same advantage to every token in a rollout. We introduce PACE-GRPO (Potential-based Action Credit Estimation GRPO), a variant of GRPO that combines potential-based reward shaping with action-level credit assignment. PACE-GRPO computes a state potential from inexpensive retrieval-derived signals, including retrieval utility, coverage, and passage diversity, and uses potential differences to provide step-level feedback. Unlike prior intermediate-reward methods, it requires no gold evidence annotations, teacher traces, reward model, or additional forward passes. Across six multi-hop QA benchmarks, PACE-GRPO learns a tool-use policy that selectively invokes the more expensive retrieval operation only when it is beneficial. This adaptive tool use yields consistent improvements over standard GRPO and strong search-augmented baselines. Our code and data are in https://github.com/lauhaide/pace.

More from EMNLP 2026

All EMNLP 2026 papers →