learning rate

AutoLR: Layer-wise Pruning and Auto-tuning of Learning Rates in Fine-tuning of Deep Networks AAAI 2021 AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly ICLR 2021 Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization ICML 2021 Continuous Time Analysis of Momentum Methods JMLR 2021 Defending against Backdoors in Federated Learning with Robust Learning Rate AAAI 2021 Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate ICLR 2021 Domain-Independent Dominance of Adaptive Methods CVPR 2021 Dual Adaptivity: A Universal Algorithm for Minimizing the Adaptive Regret of Convex Functions NeurIPS 2021 Fast rates for prediction with limited expert advice NeurIPS 2021 Generalization Performance of Multi-pass Stochastic Gradient Descent with Convex Loss Functions JMLR 2021 High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails NeurIPS 2021 How Data Augmentation affects Optimization for Linear Regression NeurIPS 2021 Label Noise SGD Provably Prefers Flat Global Minimizers NeurIPS 2021 Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling AAAI 2021 Learning by Turning: Neural Architecture Aware Optimisation ICML 2021 MetaGrad: Adaptation using Multiple Learning Rates in Online Learning JMLR 2021 Meta-learning with negative learning rates ICLR 2021 Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent ICML 2021 On the Adequacy of Untuned Warmup for Adaptive Optimization AAAI 2021 On the Origin of Implicit Regularization in Stochastic Gradient Descent ICLR 2021 On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs) NeurIPS 2021 SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation AISTATS 2021 SGD Generalizes Better Than GD (And Regularization Doesn’t Help) COLT 2021 Adaptive multi-fidelity optimization with fast learning rates AISTATS 2020 A General System of Differential Equations to Model First-Order Adaptive Algorithms JMLR 2020