Skip to content
conftrace_
2025 ACL ACL 2025

Measuring Label Ambiguity in Subjective Tasks using Predictive Uncertainty Estimation

Abstract

AbstractHuman annotations in natural language corpora vary due to differing human perspectives. This is especially prevalent in subjective tasks. In these datasets, certain data samples are more prone to label variation and can be indicated as ambiguous samples.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

Richard Alies, Elena Merdjanovska, Alan Akbik

Topics

Machine Learning > Core Methods > Classification Natural Language Processing > Applications > Text Classification Machine Learning > Learning Types > Uncertainty Quantification

Keywords

uncertainty estimation predictive uncertainty human annotation label ambiguity subjective task
Download PDF

More from ACL 2025

All ACL 2025 papers →
Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases
Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data
Understanding Disagreement: An Annotation Study of Sentiment and Emotional Language in Environmental Communication
Disagreements in analyses of rhetorical text structure: A new dataset and first analyses
Subjectivity in the Annotation of Bridging Anaphora
The revision of linguistic annotation in the Universal Dependencies framework: a look at the annotators’ behavior