Skip to content
conftrace_
2026 EMNLP EMNLP 2026

When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

AbstractWeak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train-test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution while failing to transfer across preference datasets. We provide evidence consistent with a representational failure mode in which weak-supervised fine-tuning may move the strong model toward source-domain features instead of maintaining broadly transferable preference representations. To mitigate this, we propose *Representation Anchoring* (Anchor), a simple yet effective regularizer that constrains excessive drift from the pretrained strong model’s representation space during fine-tuning, while still allowing task-relevant adaptation. Across preference domains, datasets, and model families, Anchor improves out-of-distribution transfer while maintaining competitive in-distribution performance. Together, our evaluation protocol, transfer-aware metrics, and method expose hidden brittleness in current W2S reward modeling and provide a practical path toward more robust preference transfer.

More from EMNLP 2026

All EMNLP 2026 papers →