ACL 2025

4,141 papers

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, Dieuwke Hupkes

HuGME: A benchmark system for evaluating Hungarian generative LLMs

Noémi Ligeti-Nagy, Gabor Madarasz, Flora Foldesi, Mariann Lengyel, Matyas Osvath

CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization

Brihi Joshi, Sriram Venkatapathy, Mohit Bansal, Nanyun Peng, Haw-Shiuan Chang

Using LLM Judgements for Sanity Checking Results and Reproducibility of Human Evaluations in NLP

Rudali Huidrom, Anya Belz

Clustering Zero-Shot Uncertainty Estimations to Assess LLM Response Accuracy for Yes/No Q&A

Christopher T. Franck, Amy Vennos, W. Graham Mueller, Daniel Dakota

(Towards) Scalable Reliable Automated Evaluation with Large Language Models

Bertil Braun, Martin Forell

Metric assessment protocol in the context of answer fluctuation on MCQ tasks

Ekaterina Goliakova, Xavier Renard, Marie-jeanne Lesot, Thibault Laugel, Christophe Marsala

Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs

Minsuh Joo, Hyunsoo Cho

Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs

Jing Yang Lee, Kong Aik Lee, Woon-Seng Gan

IRSum: One Model to Rule Summarization and Retrieval

Sotaro Takeshita, Simone Paolo Ponzetto, Kai Eckert

Are Bias Evaluation Methods Biased ?

Lina Berrayana, Sean Rooney, Luis Garces-Erice, Ioana Giurgiu

Can Perplexity Predict Finetuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali

Nishant Luitel, Nirajan Bekoju, Anand Kumar Sah, Subarna Shakya

Measure only what is measurable: towards conversation requirements for evaluating task-oriented dialogue systems

Emiel van Miltenburg, Anouck Braggaar, Emmelyn Croes, Florian Kunneman, Christine Liebrecht

SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities

Noga BenYoash, Menachem Brief, Oded Ovadia, Gil Shenderovitz, Moshik Mishaeli

Learning and Evaluating Factual Clarification Question Generation Without Examples

Matthew Toles, Yukun Huang, Zhou Yu

Evaluating LLMs with Multiple Problems at once

Zhengxiang Wang, Jordan Kodner, Owen Rambow

Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?

Evangelia Gogoulou, Shorouq Zahra, Liane Guillou, Luise Dürlich, Joakim Nivre

Selective Shot Learning for Code Explanation

Paheli Bhattacharya, Rishabh Gupta

Free-text Rationale Generation under Readability Level Control

Yi-Sheng Hsu, Nils Feldhus, Sherzod Hakimov

Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons

Isik Baran Sandan, Tu Anh Dinh, Jan Niehues

Are LLMs (Really) Ideological? An IRT-based Analysis and Alignment Tool for Perceived Socio-Economic Bias in LLMs

Jasmin Wachter, Michael Radloff, Maja Smolej, Katharina Kinder-Kurlanda

ARGENT: Automatic Reference-free Evaluation for Open-Ended Text Generation without Source Inputs

Xinyue Zhang, Agathe Zecevic, Sebastian Zeki, Angus Roberts

HEDS 3.0: The Human Evaluation Data Sheet Version 3.0

Anya Belz, Craig Thomson

Does Biomedical Training Lead to Better Medical Performance?

Amin Dada, Osman Alperen Koraş, Marie Bauer, Jean-philippe Corbeil, Amanda Butler Contreras

The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation

Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza

Spatial Representation of Large Language Models in 2D Scene

Wenya Wu, Weihong Deng

Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans

Javier Conde, Miguel González Saiz, María Grandury, Pedro Reviriego, Gonzalo Martínez

Towards Comprehensive Evaluation of Open-Source Language Models: A Multi-Dimensional, User-Driven Approach

Qingchen Yu

Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora

Erik Derner, Sara Sansalvador De La Fuente, Yoan Gutiérrez, Paloma Moreda Pozo, Nuria M Oliver

Adapting Psycholinguistic Research for LLMs: Gender-inclusive Language in a Coreference Context

Marion Bartl, Thomas Brendan Murphy, Susan Leavy

Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language

Kristin Gnadt, David Thulke, Simone Kopeinik, Ralf Schlüter

Towards Massive Multilingual Holistic Bias

Xiaoqing Tan, Prangthip Hansanti, Arina Turkatenko, Joe Chuang, Carleigh Wood

Disentangling Biased Representations: A Causal Intervention Framework for Fairer NLP Models

Yangge Qian, Yilong Hu, Siqi Zhang, Xu Gu, Xiaolin Qin

Ableism, Ageism, Gender, and Nationality bias in Norwegian and Multilingual Language Models

Martin Sjåvik, Samia Touileb

Examining the Cultural Encoding of Gender Bias in LLMs for Low-Resourced African Languages

Abigail Oppong, Hellina Hailu Nigatu, Chinasa T. Okolo

GenWriter: Reducing Gender Cues in Biographies through Text Rewriting

Shweta Soundararajan, Sarah Jane Delany

Some Myths About Bias: A Queer Studies Reading Of Gender Bias In NLP

Filipa Calado

Fine-Tuning vs Prompting Techniques for Gender-Fair Rewriting of Machine Translations

Paolo Mainardi, Federico Garcea, Alberto Barrón-Cedeño

GENDEROUS: Machine Translation and Cross-Linguistic Evaluation of a Gender-Ambiguous Dataset

Janiça Hackenbuchner, Eleni Gkovedarou, Joke Daems

From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in Maltese Language Models

Melanie Galea, Claudia Borg

One Size Fits None: Rethinking Fairness in Medical AI

Roland Roller, Michael Hahn, Ajay Madhavan Ravichandran, Bilgin Osmanodja, Florian Oetke

Assessing Gender Bias of Pretrained Bangla Language Models in STEM and SHAPE Fields

Noor Mairukh Khan Arnob, Saiyara Mahmud, Azmine Toushik Wasi

ArGAN: Arabic Gender, Ability, and Nationality Dataset for Evaluating Biases in Large Language Models

Ranwa Aly, Yara Allam, Rana Gaber, Christine Basta

A Diachronic Analysis of Human and Model Predictions on Audience Gender in How-to Guides

Nicola Fanton, Sidharth Ranjan, Titus Von Der Malsburg, Michael Roth

Measuring Gender Bias in Language Models in Farsi

Hamidreza Saffari, Mohammadamin Shafiei, Donya Rooein, Debora Nozza

Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models

Aleksandra Sorokovikova, Pavel Chizhov, Iuliia Eremenko, Ivan P. Yamshchikov

Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages

Lance Calvin Lim Gamboa, Yue Feng, Mark G. Lee

Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs

Elisa Forcada Rodríguez, Olatz Perez-de-Vinaspre, Jon Ander Campos, Dietrich Klakow, Vagrant Gautam

Can Explicit Gender Information Improve Zero-Shot Machine Translation?

Van-Hien Tran, Huy Hien Vu, Hideki Tanaka, Masao Utiyama

Characterizing non-binary French: A first step towards debiasing gender inference

Marie Flesch, Heather Burnett

Tag-First: Mitigating Distributional Bias in Synthetic User Profiles through Controlled Attribute Generation

Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez Santiago

GG-BBQ: German Gender Bias Benchmark for Question Answering

Shalaka Satheesh, Katrin Klug, Katharina Beckh, Héctor Allende-Cid, Sebastian Houben

Wanted: Personalised Bias Warnings for Gender Bias in Language Models

Chiara Di Bonaventura, Michelle Nwachukwu, Maria Stoica

Strengths and Limitations of Word-Based Task Explainability in Vision Language Models: a Case Study on Biological Sex Biases in the Medical Domain

Lorenzo Bertolini, Valentin Comte, Victoria Ruiz-Serra, Lia Orfei, Mario Ceresa

WoNBias: A Dataset for Classifying Bias & Prejudice Against Women in Bengali Text

Md. Raisul Islam Aupi, Nishat Tafannum, Md. Shahidur Rahman, Kh Mahmudul Hassan, Naimur Rahman

Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation

Hadi Mohammadi, Tina Shahedi, Pablo Mosteiro, Massimo Poesio, Ayoub Bagheri

Mind the Gap: Gender-based Differences in Occupational Embeddings

Olga Kononykhina, Anna-Carolina Haensch, Frauke Kreuter

Gender Bias in Nepali-English Machine Translation: A Comparison of LLMs and Existing MT Systems

Supriya Khadka, Bijayan Bhattarai

Introducing MARB — A Dataset for Studying the Social Dimensions of Reporting Bias in Language Models

Tom Södahl Bladsjö, Ricardo Muñoz Sánchez

Power(ful) Associations: Rethinking “Stereotype” for NLP

Hannah Devinney

Detecting Bias and Intersectional Bias in Italian Word Embeddings and Language Models

Alexandre Puttick, Mascha Kurpicz-Briki

Intersectional Bias in Japanese Large Language Models from a Contextualized Perspective

Hitomi Yanaka, Xinqi He, Lu Jie, Namgi Han, Sunjin Oh

JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models

Hitomi Yanaka, Namgi Han, Ryoma Kumon, Lu Jie, Masashi Takeshita

Low-resource Buryat-Russian neural machine translation

Dari Baturova, Sarana Abidueva, Dmitrii Lichko, Ivan Bondarenko

Field to Model: Pairing Community Data Collection with Scalable NLP through the LiFE Suite

Karthick Narayanan R, Siddharth Singh, Saurabh Singh, Aryan Mathur, Ritesh Kumar

A Practical Tool to Help Automate Interlinear Glossing: a Study on Mukrī Kurdish

Hiwa Asadpour, Shu Okabe, Alexander Fraser

Searchable Language Documentation Corpora: DoReCo meets TEITOK

Maarten Janssen, Frank Seifart

KazBench-KK: A Cultural-Knowledge Benchmark for Kazakh

Sanzhar Umbet, Sanzhar Murzakhmetov, Beksultan Sagyndyk, Kirill Yakunin, Timur Akishev

Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages

Siyu Liang, Gina-Anne Levow

What Causes Knowledge Loss in Multilingual Language Models?

Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti, Genta Indra Winata

Automatic Phone Alignment of Code-switched Urum–Russian Field Data

Emily Ahn, Eleanor Chodroff, Gina-Anne Levow

AIC CTU@FEVER 8: On-premise fact checking through long context RAG

Herbert Ullrich, Jan Drchal

Graph-of-Thoughts for Fact-Checking with Large Language Models

Sascha Rolinger, Jin Liu

Fathom: A Fast and Modular RAG Pipeline for Fact-Checking

Farrukh Bin Rashid, Saqib Hakak

SANCTUARY: An Efficient Evidence-based Automated Fact Checking System

Arbaaz Dharmavaram, Saqib Hakak

OldJoe at AVeriTeC: In-context learning for fact-checking

Farah Ftouhi, Russel Dsouza, Lance Calvin Lim Gamboa, Asim Abbas, Mubashir Ali

Exploring Semantic Filtering Heuristics For Efficient Claim Verification

Max Upravitelev, Premtim Sahitaj, Arthur Hilbert, Veronika Solopova, Jing Yang

Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification

Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, Kunwoo Park

The 2nd Automated Verification of Textual Claims (AVeriTeC) Shared Task: Open-weights, Reproducible and Efficient Systems

Mubashara Akhtar, Rami Aly, Yulong Chen, Zhenyun Deng, Michael Schlichtkrull

SemQA: Evaluating Evidence with Question Embeddings and Answer Entailment for Fact Verification

Kjetil Indrehus, Caroline Vannebo, Roxana Pop

EMULATE: A Multi-Agent Framework for Determining the Veracity of Atomic Claims by Emulating Human Actions

Spencer Hong, Meng Luo, Xinyi Wan

(Fact) Check Your Bias

Eivind Morris Bakke, Nora Winger Heggelund

GQC: LLM-Based Grouped QA Consolidation for Open-Domain Fact Verification at AVeriTeC

Dongzhuoran Zhou, Roxana Pop, Yuqicheng Zhu, Evgeny Kharlamov

The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucination

Yuji Zhang, Sha Li, Cheng Qian, Jiateng Liu, Pengfei Yu

Correcting Hallucinations in News Summaries: Exploration of Self-Correcting LLM Methods with External Knowledge

Juraj Vladika, Ihsan Soydemir, Florian Matthes

FACT5: A Novel Benchmark and Pipeline for Nuanced Fact-Checking of Complex Statements

Shayan Chowdhury, Sunny Fang, Smaranda Muresan

RAG based Question Answering of Korean Laws and Precedents

Kiho Seo, Takehito Utsuro

Less Can be More: An Empirical Evaluation of Small and Large Language Models for Sentence-level Claim Detection

Andrew Bell

When Scale Meets Diversity: Evaluating Language Models on Fine-Grained Multilingual Claim Verification

Hanna Shcharbakova, Tatiana Anikina, Natalia Skachkova, Josef van Genabith

Multilingual Symptom Detection on Social Media: Enhancing Health-related Fact-checking with LLMs

Saidah Zahrotul Jannah, Elyanah Aco, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

Portuguese Automated Fact-checking: Information Retrieval with Claim extraction

Juliana Gomes, Eduardo Garcia, Arlindo Rodrigues Galvão Filho

Language Model Re-rankers are Fooled by Lexical Similarities

Lovisa Hagström, Ercong Nie, Ruben Halifa, Helmut Schmid, Richard Johansson

Automated Claim–Evidence Extraction for Political Discourse Analysis: A Large Language Model Approach to Rodong Sinmun Editorials

Gyuri Choi, Hansaem Kim

Can Reasoning LLMs Synthesize Complex Climate Statements?

Yucheng Lu

LLM-Driven Estimation of Personal Carbon Footprint from Dialogues

Shuqin Li, Huifang Du, Haofen Wang

Transforming adaptation tracking: benchmarking Transformer-based NLP approaches to retrieve adaptation-relevant information from climate policy text

Jetske Bonenkamp, Robbert Biesbroek, Ioannis N. Athanasiadis

Interactive platform for the exploration of large-scale ‘living’ systematic maps

Tim Repke

Listen to the Context: Towards Faithful Large Language Models for Retrieval Augmented Generation on Climate Questions

David Thulke, Jakob Kemmler, Christian Dugast, Hermann Ney

Robust Table Information Extraction from Sustainability Reports: A Time-Aware Hybrid Two-Step Approach

Hendrik Weichel, Martin Simon, Jörg Schäfer

CPIQA: Climate Paper Image Question Answering Dataset for Retrieval-Augmented Generation with Context-based Query Expansion

Rudra Mutalik, Abiram Panchalingam, Loitongbam Gyanendro Singh, Timothy J. Osborn, Ed Hawkins