Evaluation

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark EMNLP 2023 On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research ACL 2023 On the Blind Spots of Model-Based Evaluation Metrics for Text Generation ACL 2023 On the Calibration of Large Language Models and Alignment EMNLP 2023 On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing ACL 2023 Prompting is not a substitute for probability measurements in large language models EMNLP 2023 Pseudointelligence: A Unifying Lens on Language Model Evaluation EMNLP 2023 RDumb: A simple approach that questions our progress in continual test-time adaptation NeurIPS 2023 ReCode: Robustness Evaluation of Code Generation Models ACL 2023 Re-Examining Summarization Evaluation across Multiple Quality Criteria EMNLP 2023 Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale ACL 2023 REV: Information-Theoretic Evaluation of Free-Text Rationales ACL 2023 Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation ACL 2023 RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering EMNLP 2023 Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations CVPR 2023 SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality NeurIPS 2023 Toward Human-Like Evaluation for Natural Language Generation with Error Analysis ACL 2023 Towards Inferential Reproducibility of Machine Learning Research ICLR 2023 Training Meta-Surrogate Model for Transferable Adversarial Attack AAAI 2023 UINAUIL: A Unified Benchmark for Italian Natural Language Understanding ACL 2023 VIPHY: Probing “Visible” Physical Commonsense Knowledge EMNLP 2023 What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers? ICLR 2023 What’s the Meaning of Superhuman Performance in Today’s NLU? ACL 2023 Why Is the Winner the Best? CVPR 2023 Zero-Shot Model Diagnosis CVPR 2023