Encoded but Not Express: Probing Subtle Speech-Conditioned Emotion in Large Audio Language Models
Abstract
AbstractLarge Audio Language Models (LALMs) exhibit strong speech understanding ability, yet it remains unclear whether they meaningfully utilize subtle speech-conditioned emotional cues beyond coarse acted emotions. In this work, we investigate whether the evaluated LALMs exhibit measurable sensitivity to fine-grained emotional variation in speech through controlled perturbation experiments, fine-grained emotion elicitation tasks, and representation geometry analysis. Our results show that vocal delivery variations systematically influence both model behavior and hidden representations, indicating that the evaluated LALMs already encode non-trivial speech-conditioned affective information. However, models struggle under emotionally ambiguous settings, particularly for semantically related fine-grained emotions. Representation analysis further suggests that subtle emotional structure is partially preserved despite weak downstream utilization. Overall, our findings suggest that the evaluated LALMs do not completely ignore subtle speech emotion cues, but instead utilize them inconsistently under ambiguous affective conditions