MetaBench: Benchmarking Two-Stage Reranking and Top-N Screening for Meta-Analyses
Abstract
AbstractMeta-analyses are central to evidence-based medicine, but their production remains slow because reviewers must search PubMed and screen large candidate sets under PICOS eligibility constraints. We introduce MetaBench, a large-scale benchmark for early-stage PubMed title/abstract citation screening in meta-analysis study selection, formulated as a two-stage reranking and Top-N screening problem with effort-aware evaluation. MetaBench contains 7,801 PubMed meta-analyses, 120,930 PMID-linked included studies, and reproducible candidate pools with 4.2M unique title/abstract records constructed from replayed PubMed strategies or expert-validated training-only reconstructions. To improve evaluation fidelity, we introduce a coverage-based tiering protocol and reserve Tier 1 (1,297 topics) as a high-integrity test set. MetaBench evaluates candidate reranking with Recall/nDCG and Effort@R/NormEffort@R, and budgeted Top-N screening with sensitivity, specificity, precision, and F-scores. The main benchmark intentionally focuses on scalable title/abstract screening rather than full-text eligibility adjudication; PMCID metadata supports an auxiliary full-text track on an OA-constrained subset. Experiments show that MetaBench supervision improves dense reranking, reduces screening effort, and strengthens LLM-based inclusion decisions. An anonymized codebase and dataset sample are included with this ARR submission; the full dataset and codebase will be publicly released upon acceptance.