Asymmetric Multimodal Learning with Residual Decomposition for Multimodal Intent Recognition
Abstract
AbstractMultimodal intent recognition predicts a speaker’s intent from text, audio, and video, enabling conversational AI and human–computer interaction. However, existing methods share two limitations: (I) the three modalities are assigned equivalent roles even though text alone carries most of the predictive signal; (II) nonverbal cues that complement text are mixed with cues that merely repeat it. We propose Asymmetric Multimodal Learning with Residual Decomposition (AML), in which text produces the primary prediction and nonverbal signals enter only as a correction. The correction splits into Hierarchical Modality Calibration (HMC), which removes cues already present in text, and Asymmetric Multimodal Disentanglement (AMD), which carries cues that add new information. Extensive experiments on MIntRec2.0 and IEMOCAP-DA show that AML consistently surpasses prior state-of-the-art methods across all reported metrics, and additional analyses confirm the structural advantages of the asymmetric design.