Evaluation

Anatomy of OntoGUM—Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms EMNLP 2021 A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences AAAI 2021 Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization EMNLP 2021 Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task EMNLP 2021 Are We Summarizing the Right Way? A Survey of Dialogue Summarization Data Sets EMNLP 2021 Assessing the Reliability of Word Embedding Gender Bias Measures EMNLP 2021 A Systematic Review of Reproducibility Research in Natural Language Processing EACL 2021 A Targeted Assessment of Incremental Processing in Neural Language Models and Humans ACL 2021 Benchmarking: Past, Present and Future ACL 2021 Better than Average: Paired Evaluation of NLP systems ACL 2021 Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers EMNLP 2021 Boundary IoU: Improving Object-Centric Image Segmentation Evaluation CVPR 2021 Characterizing Fairness Over the Set of Good Models Under Selective Labels ICML 2021 Chinese WPLC: A Chinese Dataset for Evaluating Pretrained Language Models on Word Prediction Given Long-Range Context EMNLP 2021 CIDEr-R: Robust Consensus-based Image Description Evaluation EMNLP 2021 Common Sense Bias in Semantic Role Labeling EMNLP 2021 Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation EMNLP 2021 Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias EMNLP 2021 CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification EMNLP 2021 COSMic: A Coherence-Aware Generation Metric for Image Descriptions EMNLP 2021 Counterfactual Matters: Intrinsic Probing For Dialogue State Tracking EMNLP 2021 Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability ACL 2021 cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE EMNLP 2021 DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues EMNLP 2021 Detecting Post-Edited References and Their Effect on Human Evaluation EACL 2021