Skip to content
conftrace_
2026 EMNLP EMNLP 2026

When Audit Quality Fails to Predict Downstream Utility: Synthetic-Data Selection for Low-Resource Classification

AbstractQuality-aware synthetic-data selection assumes that examples rated highly by an LLM judge will help downstream learning. We test this assumption in a controlled replay across two low-resource African-language classification tasks, four languages, and seven selectors. Six selectors retain 64 examples per task-language cell. CoSDA-V2 yields the highest judged label correctness (0.904 versus 0.767 for naive) and the lowest measured shortcut score, but AlpaGasus leads CoSDA-V2 on downstream Macro-F1 in both the main replay (0.202 versus 0.163) and the three-seed refit (0.193 versus 0.157).Adding the audit’s strict gate to AlpaGasus reduces Macro-F1 by 0.029, driven by one cell where only 31 of 64 candidates survive. Because the gate changes pool composition and training-set size, this comparison measures their joint effect. LESS also falls below naive when unconstrained top-k collapses label balance, but recovers with quotas. At this budget, retention constraints account for more of the observed differences than a consistent ranking advantage. In this replay, audit scores do not reliably predict downstream utility. We therefore recommend reporting both metrics for the same retained sets.

More from EMNLP 2026

All EMNLP 2026 papers →