Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

David T Hoffmann; Simon Schrodi; Jelena Bratulić; Nadine Behrmann; Volker Fischer; Thomas Brox

2024 ICML ICML 2024

Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

David T Hoffmann, Simon Schrodi, Jelena Bratulić, Nadine Behrmann, Volker Fischer, Thomas Brox

Topics

Machine Learning > Optimization & Theory > Neural Network Optimization Natural Language Processing > Generation > Language Modeling Artificial Intelligence > Core AI > Large Language Models

Keywords

transformer softmax training loss self attention multi-step task eureka-moments

Download PDF