VERGE: Verification-Enhanced Generation of Multi-Hop Evaluation Datasets for Task-Specific RAG
Abstract
AbstractEvaluating retrieval-augmented generation (RAG) on custom knowledge bases requires task-specific benchmarks of multi-chunk evidence synthesis, yet most existing generation pipelines apply quality control only after the full question set has been solved. We present VERGE, which moves quality control inside the generation loop: a verification agent targets, per item and without an external solver pass, an operational multi-chunk criterion and a QA-integrity criterion, refining items that fail and discarding those it cannot repair. From 11,052 candidates across six domains, VERGE yields 2,854 verified multiple-choice items targeting aggregation- and comparison-style synthesis over topically coherent evidence. We contribute a method-agnostic validation protocol — exhaustive strict-subset ablation, chunk-removal ablation, format-unconfounded open-ended re-evaluation, and inter-chunk-similarity stratification — and use it to localise where the multi-chunk dependence holds. On a stratified sample (n = 250), 69.8% of items not answerable closed-book cannot be answered from any strict subset of their supporting chunks (strict-necessity violation rate 15.2%). A human annotation study finds the verifier reliable (Cohen’s 𝜅 = 0.903) and VERGE items higher in QA integrity and distractor quality than the closest prior method (p < 0.001).