Neural Network Optimization

Stochastic Variance-Reduced Cubic Regularization for Nonconvex Optimization AISTATS 2019 Surfing: Iterative Optimization Over Incrementally Trained Deep Networks NeurIPS 2019 Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization ICML 2019 Task Refinement Learning for Improved Accuracy and Stability of Unsupervised Domain Adaptation ACL 2019 The Common-directions Method for Regularized Empirical Risk Minimization JMLR 2019 The Comparative Power of ReLU Networks and Polynomial Kernels in the Presence of Sparse Latent Structure ICLR 2019 The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study ICML 2019 The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes AAAI 2019 The Implicit Bias of AdaGrad on Separable Data NeurIPS 2019 The implicit bias of gradient descent on nonseparable data COLT 2019 The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks ICLR 2019 The Lower The Simpler: Simplifying Hierarchical Recurrent Models NAACL 2019 The Role of Memory in Stochastic Optimization UAI 2019 The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares NeurIPS 2019 Three Mechanisms of Weight Decay Regularization ICLR 2019 Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence NeurIPS 2019 Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion Interspeech 2019 Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks NeurIPS 2019 Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies AAAI 2019 Towards Robust ResNet: A Small Step but a Giant Leap IJCAI 2019 Towards Understanding the Importance of Shortcut Connections in Residual Networks NeurIPS 2019 Toward Understanding the Importance of Noise in Training Neural Networks ICML 2019 Traditional and Heavy Tailed Self Regularization in Neural Network Models ICML 2019 Training Neural Networks with Local Error Signals ICML 2019 Transductive Auxiliary Task Self-Training for Neural Multi-Task Models EMNLP 2019