Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
Abstract
AbstractOn-policy distillation (OPD) has emerged as a promising paradigm for reasoning-oriented post-training of large language models (LLMs), especially when combined with reinforcement learning from verifiable rewards (RLVR). Existing OPD methods rely on reverse KL (RKL)-based teacher supervision over trajectories sampled from the student policy. However, we identify a critical limitation: under large teacher–student policy divergence, RL-driven exploration often produces trajectories that the teacher assigns low probability, resulting in uninformative negative feedback. To address this, we propose Teacher-Guided Policy Optimization (TGPO). Unlike existing OPD methods that evaluate student-sampled actions using the teacher, TGPO directly guides next-token generation using the teacher’s predicted distribution on student-generated contexts, yielding an on-policy forward-KL objective. Combined with RLVR, this guidance steers exploration toward improved continuations. Experiments on reasoning benchmarks show that TGPO outperforms existing RKL-based OPD methods and remains robust across different teacher models. Furthermore, we find that teacher top-1 predictions suffice for effective guidance, without requiring the full teacher distribution.