conftrace_

mup

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

macro-variables (1) micro-variables (1) width-dominance regime (1) ema timescale (1) adamw (1) early training stage (1) learning rate scaling (1) exponential moving average (1) maximal update parameterization (1) weight decay (1)

Papers

How to set AdamW’s weight decay as you scale model and dataset size ICML 2025 On the Provable Separation of Scales in Maximal Update Parameterization ICML 2025