inter-annotator agreement

PyRater: A Python Toolkit for Annotation Analysis COLING 2024 ReproHum: #0033-03: How Reproducible Are Fluency Ratings of Generated Text? A Reproduction of August et al. 2022 COLING 2024 ReproHum #0033-3: Comparable Relative Results with Lower Absolute Values in a Reproduction Study COLING 2024 ReproHum #0087-01: A Reproduction Study of the Human Evaluation of the Coverage of Fact Checking Explanations COLING 2024 ShadowSense: A Multi-annotated Dataset for Evaluating Word Sense Induction COLING 2024 Synchronizing Approach in Designing Annotation Guidelines for Multilingual Datasets: A COVID-19 Case Study Using English and Japanese Tweets ACL 2024 The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels NAACL 2024 To Leave No Stone Unturned: Annotating Verbal Idioms in the Parallel Meaning Bank COLING 2024 Universal Dependencies: Extensions for Modern and Historical German COLING 2024 AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite EMNLP 2023 A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization ACL 2023 Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings ACL 2023 Common Law Annotations: Investigating the Stability of Dialog System Output Annotations ACL 2023 DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection SemEval 2023 Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty SemEval 2023 Harmful Language Datasets: An Assessment of Robustness ACL 2023 Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign EMNLP 2023 Identifying Visual Depictions of Animate Entities in Narrative Comics: An Annotation Study ACL 2023 LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization EACL 2023 Medieval Social Media: Manual and Automatic Annotation of Byzantine Greek Marginal Writing ACL 2023 Method for Designing Semantic Annotation of Sepsis Signs in Clinical Text ACL 2023 MIND at SemEval-2023 Task 11: From Uncertain Predictions to Subjective Disagreement SemEval 2023 Quirk or Palmer: A Comparative Study of Modal Verb Frameworks with Annotated Datasets CoNLL 2023 Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation ACL 2023 Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling ACL 2023