SCRIBE: Learning Reusable Skill Rubrics for Process Reward Models
Abstract
AbstractTraining reliable tool-augmented agents remains challenging due to the difficulty of credit assignment in multi-step reasoning. Process Reward Models (PRMs) provide a natural solution by assigning rewards to intermediate reasoning steps or trajectories, but standard LLM-based judges often produce inconsistent signals due to underspecified evaluation criteria. Recent rubric-based reward methods improve reward quality by making scoring criteria explicit, yet they typically rely on reference answers and construct instance-level rubrics that are difficult to reuse across examples. In this work, we propose SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework for reusable subgoal-level rubric rewards. SCRIBE decomposes reasoning trajectories into intermediate subgoals, clusters semantically similar reasoning behaviors into reusable Skill Prototypes, and grounds reward evaluation in prototype-level scoring criteria. This transforms reward modeling from instance-specific open-ended evaluation into reusable process-level verification. Empirical results show that SCRIBE consistently outperforms strong RL and rubric-based reward baselines across mathematical reasoning and tool-use benchmarks; notably, it improves the AIME25 score of a Qwen3-4B model from 43.3% to 63.3%. Further analysis shows that SCRIBE reduces dependence on reference answers and provides more stable, cross-instance consistent reward judgments, offering a scalable approach to reliable process reward modeling for tool-augmented agents.