Gradient Starvation in Low-Rank Language Model Pretraining
Abstract
AbstractLow-rank (LR) pretraining underperforms full-rank (FR) pretraining, but why the gap persists remains unclear. Using GPT-2 on BabyLM, we show the representational split between them is not fixed at initialisation, spectrum-matched initialisation eliminates the representational gap at step zero, but it’s reasserted within a few training steps, implicating training dynamics over initialisation. We further show gradient dimensionality is a causally separable driver of this split, not merely a restatement of the weight factorization. Restricting or expanding gradient dimensionality of LR models directly shifts representational alignment with FR counterpart independent of weight rank. Finally, optimisation success and representational alignment dissociate, that is, spectral normalisation and other interventions improve perplexity without recovering FR-LR similarity. Also, we observered that representational alignment predicts downstream competence better than perplexity does. These results support gradient starvation as a genuine, partial mechanism, and argue efficient pretraining should be evaluated by both representational trajectory and performance, not solely the latter.