Skip to content
conftrace_
2022 JMLR JMLR 2022

A Momentumized, Adaptive, Dual Averaged Gradient Method

Abstract

We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems from multiple fields, including classification and image-to-image tasks in vision, and recurrent and bidirectionally-masked models in natural language processing. For each of these tasks, MADGRAD matches or outperforms both SGD and ADAM in test set performance, even on problems for which adaptive methods normally perform poorly. [abs] [ pdf ][ bib ] [ code ] © JMLR 2022. (edit, beta)

🌉 Interdisciplinary Bridge - Artificial Intelligence and Mathematics & Optimization

Authors

Aaron Defazio, Samy Jelassi

Topics

Machine Learning > Optimization & Theory > Optimization Deep Learning > Architectures > Neural Networks Mathematics & Optimization > Optimization > Continuous Optimization Machine Learning > Learning Types > Deep Learning Deep Learning > Optimization & Theory > Neural Network Optimization

Keywords

stochastic gradient descent deep learning gradient descent dual averaging optimization algorithm adam optimizer adaptive gradient method deep learning optimization adaptive gradient
Download PDF

More from JMLR 2022

All JMLR 2022 papers →
The Importance of Being Correlated: Implications of Dependence in Joint Spectral Inference across Multiple Networks
A Perturbation-Based Kernel Approximation Framework
Reverse-mode differentiation in arbitrary tensor network format: with application to supervised learning
A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
Adversarial Robustness Guarantees for Gaussian Processes
On the Robustness to Misspecification of α-posteriors and Their Variational Approximations