Beyond the Efficiency Paradox: Unlocking Multimodal Subspaces via Cumulative Information Span
Abstract
AbstractIn real-world multimodal applications, different modalities often exhibit weakly aligned adaptation requirements, making it challenging for a shared adapter to effectively capture heterogeneous signals. Existing approaches typically use a unified low-rank adapter for concatenated multimodal representations, leading to competition among modality-specific adaptations and diminishing returns as the adapter rank increases. To address this limitation, we propose msLoRA, a modality-specific adaptation framework for efficient multimodal fine-tuning. msLoRA assigns independent low-rank adapters to different modalities, allowing them to capture specialized adaptation directions while maintaining efficient parameterization. Our theoretical analysis introduces the concept of cumulative information span, demonstrating that modality-specific adapters can expand the effective adaptation space and avoid the cross-modal disagreement typical of shared adapters. We evaluate msLoRA using multiple prominent open-source LLMs, including the Llama, Qwen, and Mistral families, across four multimodal tasks, showing consistent improvements in scalability and effectiveness for low-rank adaptation in multimodal LLMs. The source code is available at https://github.com/XiuzeZhou/msLoRA/.