conftrace_

adaptive learning rate

58 papers

Explore in graph

Co-occurring keywords

stochastic gradient descent (1091) non-convex optimization (547) stochastic optimization (1060) regret bound (1926) learning rate (192) variance reduction (523) gradient descent (1144) adam optimizer (54) online learning (1777) convergence guarantee (169)

Papers

MTAdam: Automatic Balancing of Multiple Training Loss Terms EMNLP 2021
A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance ICML 2021
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence AISTATS 2021
Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling AAAI 2021
Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates NIPS 2021
MetaGrad: Adaptation using Multiple Learning Rates in Online Learning JMLR 2021
SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients NIPS 2021
STORM+: Fully Adaptive SGD with Recursive Momentum for Nonconvex Optimization NIPS 2021
Adaptive Gradient Descent without Descent ICML 2020
Training Neural Networks for and by Interpolation ICML 2020
Understanding the Difficulty of Training Transformers EMNLP 2020
Momentum Improves Normalized SGD ICML 2020
AdaScale SGD: A User-Friendly Algorithm for Distributed Training ICML 2020
Better Full-Matrix Regret via Parameter-Free Online Learning NIPS 2020
Meta-Learning with Adaptive Hyperparameters NIPS 2020
Adam with Bandit Sampling for Deep Learning NIPS 2020
Cogra: Concept-Drift-Aware Stochastic Gradient Descent for Time-Series Forecasting AAAI 2019
Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate IJCAI 2019
Lookahead Optimizer: k steps forward, 1 step back NIPS 2019
A Sufficient Condition for Convergences of Adam and RMSProp CVPR 2019
Finite-Time Performance Bounds and Adaptive Learning Rate Selection for Two Time-Scale Reinforcement Learning NIPS 2019
Efficient Learning of Restricted Boltzmann Machines Using Covariance Estimates ACML 2019
Momentum-Based Variance Reduction in Non-Convex SGD NIPS 2019
HyperAdam: A Learnable Task-Adaptive Adam for Network Training AAAI 2019
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost ICML 2018