Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective

Wu Lin; Felix Dangel; Runa Eschenhagen; Juhan Bae; Richard E. Turner; Alireza Makhzani

2024 ICML ICML 2024

Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

❓ The Questioner

Authors

Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E. Turner, Alireza Makhzani

Topics

Machine Learning > Optimization & Theory > Optimization Machine Learning > Application Areas > Efficient Computing Deep Learning > Architectures > Transformers

Keywords

adaptive gradient method generalization gap half precision adam(w) second-order perspective square-root

Download PDF