llm evaluation

Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints ACL 2025 Fingerprinting LLMs through Survey Item Factor Correlation: A Case Study on Humor Style Questionnaire EMNLP 2025 FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models NAACL 2025 GerMedIQ: A Resource for Simulated and Synthesized Anamnesis Interview Responses in German ACL 2025 Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations EMNLP 2025 Hallucination Detection in Structured Query Generation via LLM Self-Debating EMNLP 2025 HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Assistant Scenarios ACL 2025 HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices ACL 2025 How well do LLMs reason over tabular data, really? ACL 2025 Human and LLM-Based Resume Matching: An Observational Study NAACL 2025 Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge ACL 2025 Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding EMNLP 2025 Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models ACL 2025 Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges ACL 2025 Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge EMNLP 2025 Know Thyself: Validating Knowledge Awareness of LLM-based Persona Agents NAACL 2025 Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark COLING 2025 Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundaries NAACL 2025 LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation ACL 2025 LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs NAACL 2025 LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models ACL 2025 LSC-Eval: A General Framework to Evaluate Methods for Assessing Dimensions of Lexical Semantic Change Using LLM-Generated Synthetic Data ACL 2025 MathConstruct: Challenging LLM Reasoning with Constructive Proofs ICML 2025 MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance ACL 2025 Measuring Bias and Agreement in Large Language Model Presupposition Judgments ACL 2025