Skip to content
conftrace_
2026 EMNLP EMNLP 2026

From Base Rollouts to RL Reasoning: A Budgeted Search Perspective

AbstractReinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and search remains unclear. Does RL produce reasoning behavior the base model does not have, or does it shift the rollout distribution toward trajectories the base model can already reach but rarely samples? We study this behaviorally with a Unified Decoding Framework (UDF), which expresses token-level sampling, beam-like search, tree search and sequence-level resampling as executable policies over a shared budgeted operating space. Operating points are scored after generation with pass@k, self-consistency, best-of-N and first-finish success, so the generation policy stays separate from the evaluation metric.Using paired Base/RL checkpoints from SimpleRL-Zoo, we ask whether an RL default-policy curve can be approximated by a structured path of Base operating points (𝜋, NBase) rather than by post-hoc per-budget matching. On Math500, AIME, GPQA and IFEval, the pass@k recovery path follows a Budgeted Operating-Point Transition Rule (BOPTR), NBase ≈ 𝛼 NRL𝛽, with benchmark-conditioned exponents. On Qwen2.5-7B, BOPTR gives the lowest transfer error among the non-oracle rules we test, 3.41 pp. Across models the rule family remains informative, but its parameters and selected policies are model-specific: Qwen models trained on the same difficulty split share operating-point structure, while cross-family models and different training distributions mark the limits of direct transfer.These results support a qualified internalized-search reading: under the recipe we test, much of the measured RL gain corresponds to a change in sampling efficiency, toward operating points the base model can already reach under search. We treat the benchmark-wise scaling patterns as descriptive of this recipe and cohort, report the cases in which they break down, and use UDF and BOPTR as behavioral diagnostics rather than as evidence of parameter-l

More from EMNLP 2026

All EMNLP 2026 papers →