Skip to content
conftrace_
2023 NeurIPS NeurIPS 2023

Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise

Abstract

In this work, we study the convergence in high probability of clipped gradient methods when the noise distribution has heavy tails, i.e., with bounded $p$th moments, for some $1

🌉 Interdisciplinary Bridge - Artificial Intelligence and Mathematics & Optimization

Authors

Ta Duy Nguyen, Thien H Nguyen, Alina Ene, Huy Nguyen

Topics

Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Optimization & Theory > Optimization Mathematics & Optimization > Optimization > Stochastic Methods Machine Learning > Optimization & Theory > Stochastic Methods Deep Learning > Optimization & Theory > Neural Network Optimization

Keywords

stochastic gradient descent convergence analysis gradient clipping heavy-tailed noise high probability bound high probability convergence clipped gradient method
Download PDF

More from NeurIPS 2023

All NeurIPS 2023 papers →
Single-Stage Visual Query Localization in Egocentric Videos
Hyper-Skin: A Hyperspectral Dataset for Reconstructing Facial Skin-Spectra from RGB Images
Generalizing Importance Weighting to A Universal Solver for Distribution Shift Problems
Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans
Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion
Conditional score-based diffusion models for Bayesian inference in infinite dimensions