learning rate

Parameter-free Clipped Gradient Descent Meets Polyak NeurIPS 2024 Principled Architecture-aware Scaling of Hyperparameters ICLR 2024 Prodigy: An Expeditiously Adaptive Parameter-Free Learner ICML 2024 Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning ICML 2024 Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations NeurIPS 2024 Small-scale proxies for large-scale Transformer training instabilities ICLR 2024 Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates NeurIPS 2024 Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes NeurIPS 2024 Stepping on the Edge: Curvature Aware Learning Rate Tuners NeurIPS 2024 Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features ICML 2024 The Impact of Initialization on LoRA Finetuning Dynamics NeurIPS 2024 The Marginal Value of Momentum for Small Learning Rate SGD ICLR 2024 The Sample Complexity of Gradient Descent in Stochastic Convex Optimization NeurIPS 2024 Universal Rates for Active Learning NeurIPS 2024 Universal Rates for Regression: Separations between Cut-Off and Absolute Loss COLT 2024 Where Do Large Learning Rates Lead Us? NeurIPS 2024 Why Transformers Need Adam: A Hessian Perspective NeurIPS 2024 A Continuous-time Stochastic Gradient Descent Method for Continuous Data JMLR 2023 A new characterization of the edge of stability based on a sharpness measure aware of batch gradient distribution ICLR 2023 Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspective NeurIPS 2023 BiSLS/SPS: Auto-tune Step Sizes for Stable Bi-level Optimization NeurIPS 2023 Closing the gap between the upper bound and lower bound of Adam's iteration complexity NeurIPS 2023 Coin Sampling: Gradient-Based Bayesian Inference without Learning Rates ICML 2023 Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions COLT 2023 FedL2P: Federated Learning to Personalize NeurIPS 2023