Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond Task-Based Evaluation: Toward Situated Benchmarks for Educational LLMs

AbstractThis paper argues that evaluation of educational LLMs should move beyond decontextualized task-based benchmarks toward situated benchmarks centered on pedagogical action. Prior work in human-computer interaction and the learning sciences has long emphasized situated use, user context, and interaction processes; our focus is narrower and complementary: how these concerns should reshape benchmark design for educational LLMs. Existing evaluations already cover question generation, knowledge answering, error diagnosis, feedback generation, and teaching analysis, but their dominant unit remains the single input-output episode, making it difficult to compare whether a model genuinely supports learning progress, instructional judgment, or subsequent revision in context. Educational activity is processual, context-dependent, and development-oriented, so explanations, feedback, and hints can only be evaluated properly in relation to learner state, interaction history, and pedagogical goals. We review a landscape still dominated by task-based paradigms and argue, from educational conceptions, educational values, and research methodology, that next-generation educational benchmarks should make pedagogical function in context a core target. We further propose layered, scalable, and benchmarkable situated protocols built through simulation-first development and follow-up empirical calibration, extending evaluation beyond task quality to contextual fit and downstream consequences for learning and teaching decisions.

More from EMNLP 2026

All EMNLP 2026 papers →