Neural Network Optimization

Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence AISTATS 2021 Stochastic Sign Descent Methods: New Algorithms and Better Theory ICML 2021 STORM+: Fully Adaptive SGD with Recursive Momentum for Nonconvex Optimization NeurIPS 2021 Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation ICML 2021 Strength Through Diversity: Robust Behavior Learning via Mixture Policies CoRL 2021 Structural Credit Assignment in Neural Networks using Reinforcement Learning NeurIPS 2021 Subquadratic Overparameterization for Shallow Neural Networks NeurIPS 2021 SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients NeurIPS 2021 Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization ACL 2021 SWAD: Domain Generalization by Seeking Flat Minima NeurIPS 2021 Symbolic Music Generation with Transformer-GANs AAAI 2021 Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory NeurIPS 2021 Taming Transformers for High-Resolution Image Synthesis CVPR 2021 Tangent Space Backpropagation for 3D Transformation Groups CVPR 2021 Taxonomizing local versus global structure in neural network loss landscapes NeurIPS 2021 Teacher Guided Neural Architecture Search for Face Recognition AAAI 2021 Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performance AAAI 2021 Temporal Convolutional Network with Frequency Dimension Adaptive Attention for Speech Enhancement Interspeech 2021 Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks ICML 2021 The Dynamics of Gradient Descent for Overparametrized Neural Networks L4DC 2021 The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks COLT 2021 The Elastic Lottery Ticket Hypothesis NeurIPS 2021 The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization NeurIPS 2021 The Heavy-Tail Phenomenon in SGD ICML 2021 The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks ICML 2021