benchmark evaluation

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents NeurIPS 2024 Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding CVPR 2024 T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models NeurIPS 2024 TAPVid-3D: A Benchmark for Tracking Any Point in 3D NeurIPS 2024 TAXI: Evaluating Categorical Knowledge Editing for Language Models ACL 2024 TelBench: A Benchmark for Evaluating Telco-Specific Large Language Models EMNLP 2024 TempCompass: Do Video LLMs Really Understand Videos? ACL 2024 The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance EMNLP 2024 THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation RSS 2024 The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding CVPR 2024 The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention EMNLP 2024 The Greatest Good Benchmark: Measuring LLMs’ Alignment with Utilitarian Moral Dilemmas EMNLP 2024 The Instinctive Bias: Spurious Images lead to Illusion in MLLMs EMNLP 2024 The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models ACL 2024 The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding NeurIPS 2024 The State of the Art of Large Language Models on Chartered Financial Analyst Exams EMNLP 2024 THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models CVPR 2024 TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models ACL 2024 TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models ACL 2024 Time Travel in LLMs: Tracing Data Contamination in Large Language Models ICLR 2024 ToMBench: Benchmarking Theory of Mind in Large Language Models ACL 2024 ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models EMNLP 2024 To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination ICLR 2024 Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation? NeurIPS 2024 Toward Informal Language Processing: Knowledge of Slang in Large Language Models NAACL 2024