Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Rethinking Evaluation Practices in Automated Essay Scoring: A Quantitative Analysis over the Past Decade

AbstractAutomated Essay Scoring (AES) aims to automatically assign quality scores to written essays and has been studied for several decades. Despite substantial methodological progress, little attention has been given to how AES systems are evaluated. In this position paper, we argue that several common AES evaluation practices are misaligned with broader machine learning evaluation standards. To examine current practices, we systematically analyzed 101 AES papers published in leading venues over the past decade, focusing on experimental setup, reproducibility, model development, and performance reporting and analysis. Consequently, we identify recurring shortcomings and highlight ways for more rigorous and transparent evaluation practices.

More from EMNLP 2026

All EMNLP 2026 papers →