Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Formatting and Surface Feature Confounds in AI Text Detection: A Clever Hans Effect

AbstractAI-text detectors are typically evaluated on benchmarks that pair human-written text with text generated by language models. However, differences in how these two classes are constructed can introduce surface-form cues that make the benchmark easier to solve. We show that surface form alone strongly separates human from machine-generated text across three detection benchmarks, but the learned surface patterns vary substantially across benchmarks. Detector scores are also substantially predictable from surface form, although detectors retain signal beyond it, and controlled reformatting reveals partial and detector-specific causal effects. Our findings show that benchmark performance alone does not establish that a detector has identified a stable and transferable signal of machine generation.

More from EMNLP 2026

All EMNLP 2026 papers →