conftrace_

learning rate warmup

7 papers

Explore in graph

Co-occurring keywords

adam optimizer (54) learning rate (192) gradient descent (1144) neural network optimization (1293) transformer optimization (24) warmup schedule (2) gpt training (1) layer normalization (65) large batch training (10) adaptive rate scaling (1)

Papers

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence ACL 2025
Why Warmup the Learning Rate? Underlying Mechanisms and Improvements NIPS 2024
Random Function Descent NIPS 2024
Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training NIPS 2024
Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling AAAI 2021
On the Adequacy of Untuned Warmup for Adaptive Optimization AAAI 2021
Improving Transformer Optimization Through Better Initialization ICML 2020