Neural Network Optimization

Semi-flat minima and saddle points by embedding neural networks to overparameterization NeurIPS 2019 SGD Converges to Global Minimum in Deep Learning via Star-convex Path ICLR 2019 SGD: General Analysis and Improved Rates ICML 2019 SGD on Neural Networks Learns Functions of Increasing Complexity NeurIPS 2019 Shadowing Properties of Optimization Algorithms NeurIPS 2019 Shallow-Deep Networks: Understanding and Mitigating Network Overthinking ICML 2019 Sharp Analysis for Nonconvex SGD Escaping from Saddle Points COLT 2019 Show Your Work: Improved Reporting of Experimental Results EMNLP 2019 Simple Stochastic Gradient Methods for Non-Smooth Non-Convex Regularized Optimization ICML 2019 Small nonlinearities in activation functions create bad local minima in neural networks ICLR 2019 Small Steps and Giant Leaps: Minimal Newton Solvers for Deep Learning ICCV 2019 Sorting Out Lipschitz Function Approximation ICML 2019 Spatiotemporal Multi-Graph Convolution Network for Ride-Hailing Demand Forecasting AAAI 2019 SpiderBoost and Momentum: Faster Variance Reduction Algorithms NeurIPS 2019 Spike-Train Level Backpropagation for Training Deep Recurrent Spiking Neural Networks NeurIPS 2019 Splitting Steepest Descent for Growing Neural Architectures NeurIPS 2019 Spurious Valleys in One-hidden-layer Neural Network Optimization Landscapes JMLR 2019 SSRGD: Simple Stochastic Recursive Gradient Descent for Escaping Saddle Points NeurIPS 2019 Stabilized SVRG: Simple Variance Reduction for Nonconvex Optimization COLT 2019 Stagewise Training Accelerates Convergence of Testing Error Over SGD NeurIPS 2019 Static Automatic Batching In TensorFlow ICML 2019 Stochastic Gradient Descent Learns State Equations with Nonlinear Activations COLT 2019 Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate AISTATS 2019 Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations JMLR 2019 Stochastic Nonconvex Optimization with Large Minibatches ALT 2019