Neural Network Optimization

Understanding the Role of Momentum in Stochastic Gradient Methods NeurIPS 2019 Utilizing Class Information for Deep Network Representation Shaping AAAI 2019 Weighted Channel Dropout for Regularization of Deep Convolutional Neural Network AAAI 2019 What Can ResNet Learn Efficiently, Going Beyond Kernels? NeurIPS 2019 Whether to Pretrain DNN or not?: An Empirical Analysis for Voice Conversion Interspeech 2019 Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model NeurIPS 2019 XNAS: Neural Architecture Search with Expert Advice NeurIPS 2019 Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced NeurIPS 2018 A PID Controller Approach for Stochastic Optimization of Deep Networks CVPR 2018 Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable Programming NeurIPS 2018 Densely Connected Networks for Conversational Speech Recognition Interspeech 2018 Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients ICML 2018 Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks ICML 2018 Efficient Adaptive Online Learning via Frequent Directions IJCAI 2018 Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks NeurIPS 2018 Exploring Hidden Dimensions in Accelerating Convolutional Neural Networks ICML 2018 Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis NeurIPS 2018 Hessian-based Analysis of Large Batch Training and Robustness to Adversaries NeurIPS 2018 How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective NeurIPS 2018 How to Start Training: The Effect of Initialization and Architecture NeurIPS 2018 Implicit Bias of Gradient Descent on Linear Convolutional Networks NeurIPS 2018 Improving Neural Language Models with Weight Norm Initialization and Regularization EMNLP 2018 Improving Optimization for Models With Continuous Symmetry Breaking ICML 2018 Kalman Normalization: Normalizing Internal Representations Across Network Layers NeurIPS 2018 Kronecker-factored Curvature Approximations for Recurrent Neural Networks ICLR 2018