Skip to content
conftrace_
2026 AACL AACL 2026

BAEmpEval: A Framework for Probing Large Audio Language Models on Multidimensional Empathetic Response Assessment

AbstractLarge Audio Language Models (LALMs) are increasingly applied in affective and conversational AI, yet their ability to align with human empathetic judgments in spoken dialogue remains poorly understood. We introduce BAEmpEval, a multimodal evaluation framework for analyzing empathetic response judgment in instruction-following LALMs under subjective conversational settings. The framework contains 5,436 dialogue pairs collected from three speech sources, where models evaluate listener responses across six fine-grained dimensions spanning both vocal perception (Vocal Warmth, Perceived Genuineness, Prosodic Attunement) and conversational response strategies (Exploration, Interpretation, Emotional Reaction). Experiments reveal a challenging setting: experiments across recent LALMs reveal uneven alignment with human relational judgments across empathy dimensions and speech conditions. Our findings suggest that empathetic response judgment remains a challenging multidimensional perceptual task for current LALMs and highlight the importance of evaluation frameworks that account for subjective relational variability. We release the dataset and evaluation code to support future research in human-centric AI.

More from AACL 2026

All AACL 2026 papers →