learning rate schedule

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup–Stable–Decay (Extended Abstract) COLT 2026 A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules ICLR 2025 Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs ICLR 2025 The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training ICML 2025 Towards hyperparameter-free optimization with differential privacy ICLR 2025 Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View ICLR 2025 No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths ICML 2024 Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations NeurIPS 2024 The Road Less Scheduled NeurIPS 2024 Accelerating Self-Supervised Learning via Efficient Training Strategies WACV 2023 How I Learned to Stop Worrying and Love Retraining ICLR 2023 Learning Rate Schedules in the Presence of Distribution Shift ICML 2023 Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule JMLR 2023 Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums ICLR 2022 Acceleration via Fractal Learning Rate Schedules ICML 2021 Network Pruning That Matters: A Case Study on Retraining Variants ICLR 2021 Reverse engineering learned optimizers reveals known and novel mechanisms NeurIPS 2021 An Exponential Learning Rate Schedule for Deep Learning ICLR 2020 Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints ICLR 2020 Comparing Rewinding and Fine-tuning in Neural Network Pruning ICLR 2020 Marthe: Scheduling the Learning Rate Via Online Hypergradients IJCAI 2020 On the Generalization Benefit of Noise in Stochastic Gradient Descent ICML 2020 A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation ICLR 2019 Competence-based Curriculum Learning for Neural Machine Translation NAACL 2019 New Convergence Aspects of Stochastic Gradient Algorithms JMLR 2019