Skip to content
conftrace_
2026 EMNLP EMNLP 2026

PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation

AbstractStandard on-policy distillation (OPD) for large language models estimates the reverse-KL objective using student-sampled tokens, yielding an unbiased single-sample Monte Carlo estimator that avoids vocabulary-wide computation. However, we show that this estimator suffers from severe training pathologies in practice: sample inefficiency, unstable generation dynamics, and a substantial performance gap compared to exact full-vocabulary OPD. Reward-level diagnosis traces these pathologies to the log-ratio reward, which is unbounded by construction, producing extremely high-variance gradients concentrated at early positions and persisting throughout training; standard post-hoc scaling fail as they operate only after this distortion occurs. To solve this problem, we propose *PowerOPD*: a family of natively bounded, sign-consistent rewards from the Box-Cox power transformation, parameterized by 𝛼 > 0, of which the log-ratio is the degenerate 𝛼 → 0 limit. Across six mathematical reasoning benchmarks and four Qwen3 teacher–student pairs, PowerOPD achieves benchmark-averaged Avg@8/Pass@8 gains of up to +6.37/+5.71 over vanilla OPD, +3.01/+3.54 over post-hoc stabilization, and +2.59/+8.90 over full-vocabulary OPD, while reducing wall-clock time by 59.2% and peak GPU memory by 23.1%. Larger 𝛼 generally improves accuracy, consistently shortens responses, and keeps gradient norms more than 3,000× smaller than vanilla OPD.

More from EMNLP 2026

All EMNLP 2026 papers →