Skip to content
conftrace_
2026 EMNLP EMNLP 2026

KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty

AbstractMath reasoning benchmarks have prolifer- ated, yet most lack a per-item difficulty sig- nal grounded in actual human performance. We introduce KCSAT-ML, a decade (2014– 2025) of Korean College Scholastic Ability Test (KCSAT; Suneung) mathematics: 664 problems with a 339-item core set carrying official per-item error rates from nationwide cohorts of hundreds of thousands of examinees. We pair the benchmark with Difficulty-aligned Reasoning Gain (DRG): a score-orthogonal metric that asks whether a model’s mistakes concentrate on the items humans found hard, or on items humans found easy. Together they expose, across a wide range of VLMs (and LLMs with OCR), three patterns: (i) low- budget accuracy collapses on the high-human- error tail at every model size; (ii) test-time scaling (TTS) raises token use roughly linearly with cohort error rate, while accuracy gains follow a non-monotonic curve; (iii) within a single family, TTS flips between anti-scaling on the hardest items and overthinking on eas- ier ones — two faces of the same alignment failure. With the proposed DRG metric, we find that models with near-identical accuracy can sit at near-opposite values: one model gets wrong what humans also find hard, while another solves the hardest items yet fails on items humans find easy, which is a critical con- trast that aggregate accuracy hides. Our code and dataset builder is fully open-sourced at https://github.com/naver-ai/KCSAT-ML.

More from EMNLP 2026

All EMNLP 2026 papers →