Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Evaluating LLM Autograders Without Ground Truth

AbstractLLM autograders are usually evaluated against human “ground truth,” a single reference label provided by one annotator or constructed by aggregating multiple annotations. However, human grading is often variable and subjective. We instead propose to evaluate LLM graders by treating humans and LLMs as two grading systems and characterize their agreement and disagreement. We operationalize this approach using distance correlation and generalizability theory. Across two programming-grading datasets, we find higher Human–LLM agreement on more objectively specified rubric items. On items involving greater subjective judgment, LLMs exhibit strong internal consensus, diverging more from human graders.

More from EMNLP 2026

All EMNLP 2026 papers →