Stop Rolling the Dice: Active Group Relative Policy Optimization via Intrinsic Metacognition
Abstract
AbstractGroup Relative Policy Optimization (GRPO) is an attractive reinforcement learning paradigm for Large Reasoning Models because its groupwise relative advantages eliminate the need for a value network, but its efficiency is severely limited by zero-variance queries whose rollouts are uniformly correct or uniformly incorrect and therefore consume expensive Chain-of-Thought generation without providing useful learning signals. Existing remedies typically rely on offline difficulty labels or external capability trackers, which either fail to follow the model’s rapidly evolving competence or compromise the end-to-end simplicity of GRPO. We propose Active-GRPO, a prompt-free extension that equips the policy with intrinsic fuzzy metacognition via a lightweight confidence head attached to the final prefill representation. The head predicts ordinal confidence bins and is jointly calibrated against empirical rollout outcomes through a dual-objective optimization framework. It further enables a multi-criterion voting mechanism that filters uninformative queries before rollout generation, thereby concentrating compute on samples near the learning boundary with negligible overhead and without perturbing the reasoning distribution. Experiments on mathematical and logical reasoning benchmarks show that Active-GRPO reduces training computation while improving both reasoning accuracy and intrinsic calibration quality.