Evaluation

StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code ACL 2024 Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US EMNLP 2024 SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents NeurIPS 2024 SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists EMNLP 2024 TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases NeurIPS 2024 TAIL: A Toolkit for Automatic and Realistic Long-Context Large Language Model Evaluation EMNLP 2024 TaskBench: Benchmarking Large Language Models for Task Automation NeurIPS 2024 Testing Semantic Importance via Betting NeurIPS 2024 Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning NeurIPS 2024 T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step ACL 2024 TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction ACL 2024 The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance EMNLP 2024 The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models ACL 2024 The Effect of Sampling Temperature on Problem Solving in Large Language Models EMNLP 2024 The effects of distance on NPI illusive effects in BERT EMNLP 2024 The Greatest Good Benchmark: Measuring LLMs’ Alignment with Utilitarian Moral Dilemmas EMNLP 2024 The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes CVPR 2024 Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability EMNLP 2024 The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models EMNLP 2024 The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models ACL 2024 The State of Relation Extraction Data Quality: Is Bigger Always Better? ACL 2024 The State of the Art of Large Language Models on Chartered Financial Analyst Exams EMNLP 2024 TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models ACL 2024 ToMBench: Benchmarking Theory of Mind in Large Language Models ACL 2024 ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models EMNLP 2024