Evaluation

Using Next Sentence Prediction to Test ChatGPT’s Text Comprehension (Student Abstract) AAAI 2025 UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench ACL 2025 UTER: Capturing the Human Touch in Evaluating Morphologically Rich and Low-Resource Languages NAACL 2025 Value Portrait: Assessing Language Models’ Values through Psychometrically and Ecologically Valid Items ACL 2025 VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning EMNLP 2025 VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation ACL 2025 VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation CVPR 2025 Video-Bench: Human-Aligned Video Generation Benchmark CVPR 2025 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis CVPR 2025 Vision-Language Models Do Not Understand Negation CVPR 2025 VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues ACL 2025 VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models CVPR 2025 VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models ICCV 2025 VMLU Benchmarks: A comprehensive benchmark toolkit for Vietnamese LLMs ACL 2025 Wait, that’s not an option: LLMs Robustness with Incorrect Multiple-Choice Options ACL 2025 We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? ACL 2025 We Need to Measure Data Diversity in NLP — Better and Broader EMNLP 2025 “What do you call a dog that is incontrovertibly true? Dogma”: Testing LLM Generalization through Humor ACL 2025 What’s the most important value? INVP: INvestigating the Value Priorities of LLMs through Decision-making in Social Scenarios COLING 2025 When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following EMNLP 2025 When LRP Diverges from Leave-One-Out in Transformers EMNLP 2025 When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA EMNLP 2025 Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases EMNLP 2025 Where is this coming from? Making groundedness count in the evaluation of Document VQA models NAACL 2025 Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above ACL 2025