Rethinking Gradient Flow Through Frozen Blocks for Memory-Efficient Block-Coordinate Training
Abstract
AbstractBlock-coordinate descent (BCD) has emerged as an effective approach for memory-efficient fine-tuning of large language models. During training, BCD randomly activates a small subset of blocks, such as layers or modules, and updates only those blocks while keeping the rest frozen. Although effective, prior work has paid little attention to the frozen blocks, which must still retain forward activations to propagate gradients during the backward pass. Our analysis reveals that gradients through these frozen blocks are highly sparse at the token level: a small subset of tokens carries nearly all of the signal required by the active blocks. Building on this observation, we introduce frozen-block gradient approximation with sparse tokens (Frost), which approximates gradients through inactive blocks using only a sparse subset of tokens and integrates seamlessly with standard BCD optimizers. Across diverse fine-tuning scenarios, model backbones, and modalities, Frost substantially improves the memory efficiency of BCD while preserving downstream performance with negligible degradation.