Evaluation

On the Worst Prompt Performance of Large Language Models NeurIPS 2024 Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark ACL 2024 Order Effects in Annotation Tasks: Further Evidence of Annotation Sensitivity EACL 2024 Overcoming Common Flaws in the Evaluation of Selective Classification Systems NeurIPS 2024 Paloma: A Benchmark for Evaluating Language Model Fit NeurIPS 2024 PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining NeurIPS 2024 PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data EMNLP 2024 PEDANTS: Cheap but Effective and Interpretable Answer Equivalence EMNLP 2024 Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering NeurIPS 2024 PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations NeurIPS 2024 PizzaCommonSense: A Dataset for Commonsense Reasoning about Intermediate Steps in Cooking Recipes EMNLP 2024 Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning EMNLP 2024 POSIX: A Prompt Sensitivity Index For Large Language Models EMNLP 2024 Post-Hoc Reversal: Are We Selecting Models Prematurely? NeurIPS 2024 Power of knockoff: The impact of ranking algorithm, augmented design, and symmetric statistic JMLR 2024 PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion EMNLP 2024 Precise Model Benchmarking with Only a Few Observations EMNLP 2024 Predicting generalization performance with correctness discriminators EMNLP 2024 Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method EMNLP 2024 PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action NeurIPS 2024 Probing Language Models for Pre-training Data Detection ACL 2024 ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution ACL 2024 Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models EMNLP 2024 PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities ACL 2024 QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation EMNLP 2024