Evaluation

RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-Centric Learning WACV 2024 Robust Evaluation Measures for Evaluating Social Biases in Masked Language Models AAAI 2024 RoleAgent: Building, Interacting, and Benchmarking High-quality Role-Playing Agents from Scripts NeurIPS 2024 RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning EMNLP 2024 RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs EMNLP 2024 Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? ACL 2024 SafetyBench: Evaluating the Safety of Large Language Models ACL 2024 Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? NeurIPS 2024 SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models ACL 2024 ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws EMNLP 2024 SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading EMNLP 2024 SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes ACL 2024 Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers? EMNLP 2024 Self-Evaluation of Large Language Model based on Glass-box Features EMNLP 2024 SETLEXSEM CHALLENGE: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models NeurIPS 2024 Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research ACL 2024 Small Objects Matters in Weakly-Supervised Semantic Segmentation WACV 2024 SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration EMNLP 2024 Socio-political Events of Conflict and Unrest: A Survey of Available Datasets EACL 2024 Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets EMNLP 2024 Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization? ACL 2024 SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation ACL 2024 Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? EMNLP 2024 STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions EMNLP 2024 STORYSUMM: Evaluating Faithfulness in Story Summarization EMNLP 2024