The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training

Jinbo Wang; Mingze Wang; Zhanpeng Zhou; Junchi Yan; Weinan E; Lei Wu

2025 ICML ICML 2025

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Jinbo Wang, Mingze Wang, Zhanpeng Zhou, Junchi Yan, Weinan E, Lei Wu

Topics

Machine Learning > Optimization & Theory > Optimization Machine Learning > Application Areas > Efficient Computing Artificial Intelligence > Core AI > Large Language Models

Keywords

llm pretraining gpt2 llama adamw blockwise learning rate sharpness disparity

Download PDF