RLMA: Reinforcement Learning with PolyMetric Alignment
Abstract
AbstractRLHF, RLAIF, and RLVR align LLMs with human instructions but rely on costly human labels, potentially biased LLM judges, or task-specific verifiers. We propose PolyMetric Alignment (RLMA), a reference-based reward framework for supervised-answer tasks that fuses heterogeneous evaluation metrics against a given reference answer using a Residual Set Transformer (RST) trained with a Bradley-Terry objective. Empirically, RLMA is stronger than all three paradigms, achieving 98.0% of the Pass@K oracle on a 9-pool benchmark, with gains of +17.6 pp over the RLA single-metric baseline and +7.4 to +15.4 pp over RLHF, RLAIF, and RLVR. RLMA also generalizes broadly, reaching 97.2% of oracle on unseen GSM8K and, when used as a DPO reward, improving TriviaQA Pass@1 by +4.80 pp on Qwen-2.5-7B-Instruct. Finally, RLMA is efficient to deploy. Its 113K-parameter gate requires only 0.33 ms per candidate, roughly 300x faster than a single 8B reward-model forward.