Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Mitigating Exploration Bias in RL for Multi-Instruction Following

AbstractRL has emerged as a powerful paradigm for enhancing the instruction following capabilities of LLMs. While existing training recipes achieve substantial gains, we find that they suffer from **exploration bias towards easy instructions** when the training data has multiple instructions in a prompt. This bias is caused by two main reasons: 1) **the policy model’s initial ability to satisfy hard instructions is too low** to trigger successful exploration during RL training, so the optimization is biased to easy instructions and 2) canonical RL training recipes typically employ a cumulative reward (the number of instructions fulfilled), **treating all the instructions equally**; the policy model is biased to fulfill the easy instructions to get the same amount of rewards. To address these, we first propose two metrics to measure the exploration bias of instruction following and then introduce a two-stage framework to alleviate it: 1) Behavioral Bootstrapping: a lightweight rejection sampling fine-tuning before RL to activate hard instructions and 2) Scarcity-Aware Rewards: a new RL reward function that assigns rewards to instructions based on their empirical scarcity. Experiments show that the proposed metrics are highly correlated with model performance and our methods unleash the potential of RL training: our best models outperform the baselines by a significant margin across three verifiable instruction following benchmarks.

More from EMNLP 2026

All EMNLP 2026 papers →