EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing
Abstract
AbstractProcess reward models (PRMs) are widely used in language-model training with dense, step-level supervision. They assume that PRM scores remain stable proxies for step correctness under label-preserving transformations that alter reasoning structure while preserving final answers. However, this assumption remains insufficiently validated. Such transformations may alter the relationship between PRM scores and correctness, producing different failure modes across models. To address this gap, we introduce EST-PRM, a stress-testing framework for dense process rewards. It evaluates three transformations: (1) step inflation, (2) dependency-aware step reordering, and (3) confidence markers. We further define a vulnerability decomposition that separates reward inflation from loss of correctness sensitivity. We evaluate five PRM-style models on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench. Results reveal substantial differences in vulnerability patterns across models. Math-Shepherd exhibits the strongest sensitivity to position-based perturbations, with a Pearson correlation drop of 0.139 ± 0.031 and a 29.7 ± 4.2% score inflation rate, while Qwen2.5-Math-PRM is most affected by step inflation, reaching a 47.6 ± 4.3% inflation rate. Confidence-based perturbations further distort reward calibration, revealing inconsistencies in correctness estimation. Finally, we evaluate three mitigation strategies and identify trade-offs between robustness coverage and false-positive rates.