Neural Network Optimization

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Functions NeurIPS 2022 Stochastic Training is Not Necessary for Generalization ICLR 2022 Subspace Adversarial Training CVPR 2022 Super-Acceleration with Cyclical Step-sizes AISTATS 2022 Superposing many tickets into one: A performance booster for sparse neural network training UAI 2022 Surprising Instabilities in Training Deep Networks and a Theoretical Analysis NeurIPS 2022 Survey on Efficient Training of Large Neural Networks IJCAI 2022 SVT-Net: Super Light-Weight Sparse Voxel Transformer for Large Scale Place Recognition AAAI 2022 Swin Transformer V2: Scaling Up Capacity and Resolution CVPR 2022 Taylor-Lagrange Neural Ordinary Differential Equations: Toward Fast Training and Evaluation of Neural ODEs IJCAI 2022 Temporal Effective Batch Normalization in Spiking Neural Networks NeurIPS 2022 Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting ICLR 2022 The alignment property of SGD noise and how it helps select flat minima: A stability analysis NeurIPS 2022 The Curse of Unrolling: Rate of Differentiating Through Optimization NeurIPS 2022 The Devil in Linear Transformer EMNLP 2022 The Dynamics of Riemannian Robbins-Monro Algorithms COLT 2022 The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains EMNLP 2022 The Importance of Hyperparameter Optimisation for Facial Recognition Applications AAAI 2022 The least-control principle for local learning at equilibrium NeurIPS 2022 Theoretically Provable Spiking Neural Networks NeurIPS 2022 The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance COLT 2022 The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection AAAI 2022 The Role of Baselines in Policy Gradient Optimization NeurIPS 2022 The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks ICLR 2022 The Spectral Bias of Polynomial Neural Networks ICLR 2022