Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

Shikai Qiu; Lechao Xiao; Andrew Gordon Wilson; Jeffrey Pennington; Atish Agarwala

2025 ICML ICML 2025

Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, Atish Agarwala

Topics

Machine Learning > Optimization & Theory > Neural Network Optimization Deep Learning > Architectures > Transformers Artificial Intelligence > Core AI > Large Language Models

Keywords

learning rate decay compute-optimal training neural scaling law scaling collapse sgd noise dynamics supercollapse

Download PDF