Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape

Juno Kim; Taiji Suzuki

2024 ICML ICML 2024

Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Juno Kim, Taiji Suzuki

Topics

Machine Learning > Core Methods > Representation Learning Machine Learning > Optimization & Theory > Optimization Artificial Intelligence > Core AI > Large Language Models

Keywords

in-context learning transformer wasserstein gradient flow nonlinear feature mean-field dynamics improvement rates

Download PDF