2026
EMNLP
EMNLP 2026
Evaluating LLM Autograders Without Ground Truth
Abstract
AbstractLLM autograders are usually evaluated against human “ground truth,” a single reference label provided by one annotator or constructed by aggregating multiple annotations. However, human grading is often variable and subjective. We instead propose to evaluate LLM graders by treating humans and LLMs as two grading systems and characterize their agreement and disagreement. We operationalize this approach using distance correlation and generalizability theory. Across two programming-grading datasets, we find higher Human–LLM agreement on more objectively specified rubric items. On items involving greater subjective judgment, LLMs exhibit strong internal consensus, diverging more from human graders.