Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers

Freya Behrens; Luca Biggio; Lenka Zdeborová

2025 ICML ICML 2025

Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Freya Behrens, Luca Biggio, Lenka Zdeborová

Topics

Artificial Intelligence > Core AI > Interpretability Machine Learning > Core Methods > Representation Learning Deep Learning > Architectures > Transformers

Keywords

softmax embedding dimension feed-forward layer attention counting histogram task

Download PDF