Skip to content
conftrace_
2026 EMNLP EMNLP 2026

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

AbstractPost-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose *CoSpaDi*, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. *CoSpaDi* factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across Llama and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.

More from EMNLP 2026

All EMNLP 2026 papers →