Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?

Khashayar Gatmiry; Nikunj Saunshi; Sashank J. Reddi; Stefanie Jegelka; Sanjiv Kumar

2024 ICML ICML 2024

Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

❓ The Questioner

Authors

Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi, Stefanie Jegelka, Sanjiv Kumar

Topics

Machine Learning > Core Methods > Regression Machine Learning > Optimization & Theory > Learning Theory Deep Learning > Architectures > Transformers

Keywords

gradient descent convergence preconditioners fix-point iterative algorithms linear looped transformers population training loss

Download PDF