HLoRA: Hierarchical Regularization on Combined Parameter Space for LoRA Fine-Tuning
Abstract
AbstractLow-Rank Adaptation (LoRA) is widely used for fine-tuning large language models, but suffers from catastrophic forgetting. Existing continual learning methods mitigate this by estimating per-parameter importance independently. However, this assumption breaks down under LoRA’s factorization 𝛥 W = BA: infinitely many pairs (B, A) yield the same weight update, making importance scores on A and B separately unreliable. We propose HLoRA, which computes path-integral importance in the combined weight update space, bypassing the unreliable separate estimation on A and B. HLoRA applies hierarchical regularization at two levels: element-wise constraints preserve parameters critical for general knowledge, and layer-wise coefficients adapt regularization strength based on each layer’s aggregate importance. Experiments on scientific, medical, and physical tasks using GPT-J and LLaMA-3 show that HLoRA mitigates catastrophic forgetting while enhancing adaptability, running approximately 20x faster and requiring only 10%–15% of the storage compared to prior methods. Code is released on https://anonymous.4open.science/r/HLoRA-915F.