Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Visual Preference Optimization with Rubric Rewards

AbstractThe effectiveness of Direct Preference Optimization (DPO) depends on data pairs that capture task-relevant quality differences. Existing data pipelines often rely on *off-policy* perturbations or *coarse* outcome-based signals, which are not well suited to fine-grained visual reasoning. We propose *rDPO*, a rubric-based data pipeline for DPO-based visual preference optimization. For each image-instruction pair, we create a checklist-style rubric of essential and additional criteria to score responses from any possible policies. The instruction-rubric pool is built *offline* and reused during the construction of *on-policy* preference data. On public reward modeling benchmarks, rubric-based prompting massively improves a 30B-A3B judge and brings it close to GPT-5.4. On public downstream benchmarks, rubric-based filtering raises the macro average to 82.69, whereas outcome-based filtering drops it to 75.82 from 81.14. In scalability evaluation on a comprehensive benchmark, rDPO achieves 61.01, markedly outperforming the style-constrained baseline (52.36) and surpassing the 59.48 base model. Together, these results show that visual preference optimization benefits from combining on-policy data construction with instance-specific criterion-level feedback.

More from EMNLP 2026

All EMNLP 2026 papers →