adam optimizer

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions NeurIPS 2024 Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling NeurIPS 2024 The Implicit Bias of Adam on Separable Data NeurIPS 2024 Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise ICML 2024 Why Transformers Need Adam: A Hessian Perspective NeurIPS 2024 Why Warmup the Learning Rate? Underlying Mechanisms and Improvements NeurIPS 2024 AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix NeurIPS 2023 Closing the gap between the upper bound and lower bound of Adam's iteration complexity NeurIPS 2023 Convergence of Adam Under Relaxed Assumptions NeurIPS 2023 Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models NeurIPS 2023 How Does Adaptive Optimization Impact Local Neural Network Geometry? NeurIPS 2023 Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam ICLR 2023 Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be ICLR 2023 STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition ICML 2023 Adam Can Converge Without Any Modification On Update Rules NeurIPS 2022 Adaptive Differential Privacy for Language Model Training ACL 2022 A Momentumized, Adaptive, Dual Averaged Gradient Method JMLR 2022 KOALA: A Kalman Optimization Algorithm with Loss Adaptivity AAAI 2022 The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection AAAI 2022 Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration JMLR 2022 VectorAdam for Rotation Equivariant Geometry Optimization NeurIPS 2022 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed ICML 2021 Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers ICML 2021 Domain-Independent Dominance of Adaptive Methods CVPR 2021 GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training NeurIPS 2021