weight decay

To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets ICLR 2024 Weight decay induces low-rank attention layers NeurIPS 2024 Why Do We Need Weight Decay in Modern Deep Learning? NeurIPS 2024 Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training JMLR 2024 Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses NeurIPS 2023 Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive? ICLR 2023 FedNAR: Federated Optimization with Normalized Annealing Regularization NeurIPS 2023 Globally Optimal Training of Neural Networks with Threshold Activation Functions ICLR 2023 Improving Robustness with Adaptive Weight Decay NeurIPS 2023 Neural Networks Efficiently Learn Low-Dimensional Representations with SGD ICLR 2023 On Minimizing the Impact of Dataset Shifts on Actionable Explanations UAI 2023 On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective NeurIPS 2023 Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization ICLR 2023 Win: Weight-Decay-Integrated Nesterov Acceleration for Adaptive Gradient Algorithms ICLR 2023 Exact Solutions of a Deep Linear Network NeurIPS 2022 Fast Mixing of Stochastic Gradient Descent with Normalization and Weight Decay NeurIPS 2022 Robust Training of Neural Networks Using Scale Invariant Architectures ICML 2022 Surprising Instabilities in Training Deep Networks and a Theoretical Analysis NeurIPS 2022 The Effects of Regularization and Data Augmentation are Class Dependent NeurIPS 2022 Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction NeurIPS 2022 Adaptive Knowledge Driven Regularization for Deep Neural Networks AAAI 2021 Bag of Tricks for Adversarial Training ICLR 2021 Banach Space Representer Theorems for Neural Networks and Ridge Splines JMLR 2021 Convex Regularization behind Neural Reconstruction ICLR 2021 Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs ICML 2021