Skip to content
conftrace_
2025 ICML ICML 2025

Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Thiziri Nait Saada, Alireza Naderi, Jared Tanner

Topics

Machine Learning > Optimization & Theory > Optimization Deep Learning > Architectures > Transformers Deep Learning > Learning Types > Representation Learning

Keywords

random matrix theory softmax attention singular value gradient propagation spectral gap rank collapse
Download PDF