Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond SFT-Optimality: Cold-Start Data Selection for Reinforcement Learning via Gradient Subspace Coverage

AbstractCold-start supervised fine-tuning (SFT) is a critical initialization step before reinforcement learning from verifiable rewards (RLVR). Yet existing cold-start data selection methods largely inherit criteria from supervised instruction tuning—sample quality, difficulty, perplexity, or diversity—implicitly assuming that a stronger SFT warm start is also a better starting point for RL. We argue that this assumption does not always hold in the SFT-to-RL pipeline: cold-start SFT not only reduces supervised loss but also shapes the initial update geometry that downstream reward optimization inherits. Building on this observation, we propose RL-Proxy-Coverage. Our method estimates a low-rank, reward-relevant gradient subspace from a small set of representative RL anchor prompts and verifier rewards, and selects, as the cold-start subset, those candidate SFT examples whose gradients provide non-redundant coverage of this subspace. In controlled mathematical-reasoning experiments, RL-Proxy-Coverage outperforms representative SFT-centric, hard-diverse, and reward-aware baselines at the early-to-mid RL checkpoints. Our findings suggest a new view of cold-start data: it should be regarded not merely as supervised examples for improving SFT, but as an initialization resource that shapes the starting point of downstream reward optimization.

More from EMNLP 2026

All EMNLP 2026 papers →