Neural Network Optimization

On the Provable Generalization of Recurrent Neural Networks NeurIPS 2021 On the Role of Optimization in Double Descent: A Least Squares Study NeurIPS 2021 On the Suboptimality of Negative Momentum for Minimax Optimization AISTATS 2021 On the Transformer Growth for Progressive BERT Training NAACL 2021 On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs) NeurIPS 2021 On Training Implicit Models NeurIPS 2021 OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection CVPR 2021 Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics ICML 2021 Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks ICLR 2021 Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime ICLR 2021 Optimization Planning for 3D ConvNets ICML 2021 Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives JMLR 2021 Optimizing Deeper Transformers on Small Datasets ACL 2021 Optimizing Optimizers: Regret-optimal gradient descent algorithms COLT 2021 Order Regularization on Ordinal Loss for Head Pose, Age and Gaze Estimation AAAI 2021 Orthogonal Over-Parameterized Training CVPR 2021 PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization ICML 2021 Parallelizing Legendre Memory Unit Training ICML 2021 Parameter-Efficient Transfer Learning with Diff Pruning ACL 2021 Partially-Connected Differentiable Architecture Search for Deepfake and Spoofing Detection Interspeech 2021 Particle Dual Averaging: Optimization of Mean Field Neural Network with Global Convergence Rate Analysis NeurIPS 2021 Partition-Based Formulations for Mixed-Integer Optimization of Trained ReLU Neural Networks NeurIPS 2021 Path-BN: Towards effective batch normalization in the Path Space for ReLU networks UAI 2021 Path Length Bounds for Gradient Descent and Flow JMLR 2021 PGT: A Progressive Method for Training Models on Long Videos CVPR 2021