Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Learning New Facts with QLoRA: An Acquisition-Retention Frontier

AbstractParameter-efficient fine-tuning (PEFT) is often assumed to preserve pretrained capabilities because it updates only a small number of parameters. We show that this assumption depends strongly on adapter capacity. We study factual acquisition in a controlled OpenStreetMap-derived benchmark where Qwen3-4B must learn anonymized geographic relations while retaining unrelated capabilities. Comparing FFT with QLoRA ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition–retention frontier. Low-rank QLoRA preserves OOD performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at increasing cost to unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime. Distributional, weight-space, and spectral diagnostics mirror this behavioral trade-off, with higher-rank QLoRA moving farther from the pretrained model. A separate math adaptation experiment shows a weaker frontier, suggesting that the effect is most pronounced when adaptation must install new factual associations rather than reinforce skills already supported by pretraining.

More from EMNLP 2026

All EMNLP 2026 papers →