LAK-LoRA: Layer-Wise Adaptive Expert Allocation and Principal Component–Based Feature Separation for Medical Multi-Task Learning
Abstract
AbstractMulti-task learning in the medical domain remains a significant challenge for large language models (LLMs), plagued by conflicting task objectives and substantial computational overhead associated with full fine-tuning. Although parameter-efficient tuning methods help reduce these resource demands, their effectiveness is limited by two critical issues: (i) rigid allocation of expert capacity across layers, and (ii) lack of regularization mechanisms to partition shared components from task-specific signals. To address these limitations, we propose LAK-LoRA, a novel architecture that enables adaptive, depth-aware scaling of expert capacity, allowing more efficient resource allocation throughout the model. Furthermore, to resolve conflicting task objectives, LAK-LoRA divides experts into two distinct groups dedicated to modeling general and task-specific features, respectively. This separation is implemented via a feature separator module, which disentangles representations into shared and task-specific subspaces before assigning them to the appropriate experts. To foster diversity and minimize representational redundancy, orthogonality constraints are imposed among specialized experts. Extensive experiments on benchmark medical datasets demonstrate that LAK-LoRA achieves the best average performance.