Neural Network Optimization

Nonsmooth Implicit Differentiation for Machine-Learning and Optimization NeurIPS 2021 Not All Memories are Created Equal: Learning to Forget by Expiring ICML 2021 Not All Operations Contribute Equally: Hierarchical Operation-Adaptive Predictor for Neural Architecture Search ICCV 2021 Not all parameters are born equal: Attention is mostly what you need EMNLP 2021 NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-end Learning and Control ICLR 2021 OadTR: Online Action Detection With Transformers ICCV 2021 One-pass Stochastic Gradient Descent in overparametrized two-layer neural networks AISTATS 2021 One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking CVPR 2021 On Linear Stability of SGD and Input-Smoothness of Neural Networks NeurIPS 2021 Only Train Once: A One-Shot Neural Network Training And Pruning Framework NeurIPS 2021 On Monotonic Linear Interpolation of Neural Network Parameters ICML 2021 On Orthogonality Constraints for Transformers ACL 2021 On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning NeurIPS 2021 On Proximal Policy Optimization’s Heavy-tailed Gradients ICML 2021 On Randomized Classification Layers and Their Implications in Natural Language Generation NAACL 2021 On the Adequacy of Untuned Warmup for Adaptive Optimization AAAI 2021 On the Convergence of Step Decay Step-Size for Stochastic Optimization NeurIPS 2021 On the distributional properties of adaptive gradients UAI 2021 On the Expected Complexity of Maxout Networks NeurIPS 2021 On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks ICML 2021 On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models ICML 2021 On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent ICML 2021 On the Influence of Masking Policies in Intermediate Pre-training EMNLP 2021 On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization IJCAI 2021 On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay NeurIPS 2021