Skip to content
conftrace_

learning rate warmup

11 papers

Explore in graph

Co-occurring keywords · papers mentioning both

adam optimizer (3) learning rate (3) gradient descent (3) neural network optimization (3) transformer optimization (2) layer normalization (2) warmup schedule (2) adaptive learning rate (2) gpt training (1) transformer training (1)

Papers

Taming Transformer Without Using Learning Rate Warmup ICLR 2025 The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence ACL 2025 Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training NeurIPS 2024 Random Function Descent NeurIPS 2024 When Will Gradient Regularization Be Harmful? ICML 2024 Why Warmup the Learning Rate? Underlying Mechanisms and Improvements NeurIPS 2024 Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling AAAI 2021 On the Adequacy of Untuned Warmup for Adaptive Optimization AAAI 2021 Improving Transformer Optimization Through Better Initialization ICML 2020 On Layer Normalization in the Transformer Architecture ICML 2020 On the Variance of the Adaptive Learning Rate and Beyond ICLR 2020