How to set AdamW’s weight decay as you scale model and dataset size

Xi Wang; Laurence Aitchison

2025 ICML ICML 2025

How to set AdamW’s weight decay as you scale model and dataset size

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Xi Wang, Laurence Aitchison

Topics

Machine Learning > Optimization & Theory > Neural Network Optimization Deep Learning > Techniques > Model Architecture Deep Learning > Optimization & Theory > Optimization

Keywords

weight decay exponential moving average learning rate scaling adamw mup ema timescale

Download PDF