Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Out-of-Distribution Checkpoint Selection Has a Resolution Problem: Auditing Sparse Exact Match with Token Likelihood

AbstractExact match (EM) can make checkpoint selection ambiguous when a small labeled out of distribution (OOD) validation set leaves several checkpoints tied at zero or at the maximum. Without an explicit tie rule, the reported metric does not identify which checkpoint produced the final test score. We study this validation resolution problem in semantic parsing and morphological reinflection. Our protocol reports the reference budget, token likelihood choice under teacher forcing, and latest tied candidate, while using EM for a unique nonzero maximum. Across 600 draws from 30 runs, 8 references leave every early checkpoint at zero in 48% of draws and the maximum tied in 57%. Within tied draws, mean full split regret falls from 0.063 under random selection to 0.050 with token likelihood. The latest tied candidate performs better on average for mostly monotone trajectories, but token likelihood independently orders candidates that hard EM cannot distinguish. These results support reporting both orderings when small labeled OOD validation sets guide checkpoint selection.

More from EMNLP 2026

All EMNLP 2026 papers →