Adaptive Latent Block Insertion via Online Reward Optimization
Abstract
AbstractChain-of-thought prompting improves reasoning by generating intermediate steps as discrete text, but each step is forced to commit to a single token even when the model is uncertain. Continuous latent reasoning offers an alternative: a latent state can preserve multiple candidate interpretations without immediately selecting one token. Existing training-free approaches either place latent states only before decoding or switch between explicit and latent modes without optimizing the inserted state itself. We propose MidThink, a training-free framework that inserts and optimizes latent blocks both at the start of reasoning and adaptively during generation. When a confidence trigger detects sustained mid-stream uncertainty, MidThink inserts a new latent block and refines it with a rollout-based reward computed from short future continuations. We further show that initialization strongly affects whether a mid-stream block preserves a superposition over multiple tokens or collapses toward a single token. Across five mathematical reasoning benchmarks, MidThink improves over training-free latent-reasoning baselines, with the largest gains on the AIME benchmarks. Mechanistic analysis links these gains to three measurable properties of the latent block: distributional entropy across layers, attention centroid position, and stability under reinforcement learning updates.