annotation quality

Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign EMNLP 2023 How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation? ACL 2023 Incorporating Worker Perspectives into MTurk Annotation Practices for NLP EMNLP 2023 K-HATERS: A Hate Speech Detection Corpus in Korean with Target-Specific Ratings EMNLP 2023 Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection EMNLP 2023 Rethinking Annotation: Can Language Learners Contribute? ACL 2023 Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling ACL 2023 Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance EMNLP 2023 Verifying Annotation Agreement without Multiple Experts: A Case Study with Gujarati SNACS ACL 2023 Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models EMNLP 2022 DiaBiz.Kom - towards a Polish Dialogue Act Corpus Based on ISO 24617-2 Standard COLING 2022 Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED ACL 2022 Establishing Annotation Quality in Multi-label Annotations COLING 2022 Evaluating Diversity of Multiword Expressions in Annotated Text COLING 2022 GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation EMNLP 2022 Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations ACL 2022 k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations ACL 2022 On reporting scores and agreement for error annotation tasks EMNLP 2022 On the Diversity and Limits of Human Explanations NAACL 2022 The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation EMNLP 2022 Toward More Meaningful Resources for Lower-resourced Languages ACL 2022 ActiveAnno: General-Purpose Document-Level Annotation Tool with Active Learning Integration NAACL 2021 Adaptiope: A Modern Benchmark for Unsupervised Domain Adaptation WACV 2021 Agree to Disagree: Analysis of Inter-Annotator Disagreements in Human Evaluation of Machine Translation Output CoNLL 2021 Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability ACL 2021