Neural Network Optimization

Online Normalization for Training Neural Networks NeurIPS 2019 On the Continuity of Rotation Representations in Neural Networks CVPR 2019 On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization ICLR 2019 On the Convergence Rate of Training Recurrent Neural Networks NeurIPS 2019 Optimizing a Speaker Embedding Extractor Through Backend-Driven Regularization Interspeech 2019 Ouroboros: On Accelerating Training of Transformer-Based Language Models NeurIPS 2019 Quasi-hyperbolic momentum and Adam for deep learning ICLR 2019 RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications AAAI 2019 Reducing the variance in online optimization by transporting past gradients NeurIPS 2019 Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel NeurIPS 2019 Regularizing Neural Networks via Stochastic Branch Layers ACML 2019 RENAS: Reinforced Evolutionary Neural Architecture Search CVPR 2019 Revisit Batch Normalization: New Understanding and Refinement via Composition Optimization AISTATS 2019 Searching for a Robust Neural Architecture in Four GPU Hours CVPR 2019 SpiderBoost and Momentum: Faster Variance Reduction Algorithms NeurIPS 2019 Spurious Valleys in One-hidden-layer Neural Network Optimization Landscapes JMLR 2019 Stagewise Training Accelerates Convergence of Testing Error Over SGD NeurIPS 2019 Stochastic Gradient Descent Learns State Equations with Nonlinear Activations COLT 2019 Surfing: Iterative Optimization Over Incrementally Trained Deep Networks NeurIPS 2019 Taming the Noisy Gradient: Train Deep Neural Networks with Small Batch Sizes IJCAI 2019 The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes AAAI 2019 The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks NeurIPS 2019 Towards a Better Understanding and Regularization of GAN Training Dynamics UAI 2019 Transductive Auxiliary Task Self-Training for Neural Multi-Task Models EMNLP 2019 Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift CVPR 2019