inter-annotator agreement

KaLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks CVPR 2026 LLMs as Span Annotators: A Comparative Study of LLMs and Humans EACL 2026 Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation & The Evaluation Gap ACL 2026 A European Portuguese corpus annotated for verbal idioms NAACL 2025 A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment ACL 2025 Annotating Spatial Descriptions in Literary and Non-Literary Text ACL 2025 Annotating the Annotators: Analysis, Insights and Modelling from an Annotation Campaign on Persuasion Techniques Detection ACL 2025 Another Approach to Agreement Measurement and Prediction with Emotion Annotations ACL 2025 Automating Annotation Guideline Improvements using LLMs: A Case Study COLING 2025 Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement EMNLP 2025 Context Is Ubiquitous, but Rarely Changes Judgments: Revisiting Document-Level MT Evaluation EMNLP 2025 Creating Hierarchical Relations in a Multilingual Event-type Ontology ACL 2025 Disagreement in Metaphor Annotation of Mexican Spanish Science Tweets COLING 2025 Disagreements in analyses of rhetorical text structure: A new dataset and first analyses ACL 2025 Disentangling Subjectivity and Uncertainty for Hate Speech Annotation and Modeling using Gaze EMNLP 2025 Enhancing Human Evaluation in Machine Translation with Comparative Judgement ACL 2025 EventFull: Complete and Consistent Event Relation Annotation NAACL 2025 Few-Shot Prompting, Full-Scale Confusion: Evaluating Large Language Models for Humor Detection in Croatian Tweets ACL 2025 Guidelines for Fine-grained Sentence-level Arabic Readability Annotation ACL 2025 High-Quality Complex Text-to-SQL Data Generation through Chain-of-Verification IJCNLP 2025 ICLE-RC: International Corpus of Learner English for Relative Clauses ACL 2025 Insights into developing analytical categorization schemes: three problem types related to annotation agreement NAACL 2025 LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation EMNLP 2025 LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests EMNLP 2025 ReproHum #0729-04: Partial reproduction of the human evaluation of the MemSum and NeuSum summarisation systems ACL 2025