Evaluation

Model Analysis & Evaluation for Ambiguous Question Answering ACL 2023 ModelDiff: A Framework for Comparing Learning Algorithms ICML 2023 MoQA: Benchmarking Multi-Type Open-Domain Question Answering ACL 2023 Morphological Inflection: A Reality Check ACL 2023 Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking ACL 2023 mSCAN: A Dataset for Multilingual Compositional Generalisation Evaluation EMNLP 2023 MTEB: Massive Text Embedding Benchmark EACL 2023 MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation ACL 2023 Multi-Dimensional Evaluation of Text Summarization with In-Context Learning ACL 2023 NarrativeXL: a Large-scale Dataset for Long-Term Memory Models EMNLP 2023 Never mind the metrics---what about the uncertainty? Visualising binary confusion matrix metric distributions to put performance in perspective ICML 2023 NEWTON: Are Large Language Models Capable of Physical Reasoning? EMNLP 2023 NICO++: Towards Better Benchmarking for Domain Generalization CVPR 2023 NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist ACL 2023 NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark EMNLP 2023 Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP ACL 2023 No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference ACL 2023 OLEA: Tool and Infrastructure for Offensive Language Error Analysis in English EACL 2023 On Calibrating Semantic Segmentation Models: Analyses and an Algorithm CVPR 2023 On Evaluating Multilingual Compositional Generalization with Translated Datasets ACL 2023 On Event Individuation for Document-Level Information Extraction EMNLP 2023 On the bias of K-fold cross validation with stable learners AISTATS 2023 On the Blind Spots of Model-Based Evaluation Metrics for Text Generation ACL 2023 On the Calibration of Large Language Models and Alignment EMNLP 2023 On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research EMNLP 2023