conftrace_

2015 NeurIPS NeurIPS 2015

Learning with Incremental Iterative Regularization

Abstract

Within a statistical learning setting, we propose and study an iterative regularization algorithm for least squares defined by an incremental gradient method. In particular, we show that, if all other parameters are fixed a priori, the number of passes over the data (epochs) acts as a regularization parameter, and prove strong universal consistency, i.e. almost sure convergence of the risk, as well as sharp finite sample bounds for the iterates. Our results are a step towards understanding the effect of multiple epochs in stochastic gradient techniques in machine learning and rely on integrating statistical and optimizationresults.

📈 Trend Setter - Regularization

🐣 Hot Topic Early Bird - stochastic gradient

Authors

Lorenzo Rosasco, Silvia Villa

Topics

Machine Learning > Core Methods > Regression Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Optimization & Theory > Optimization Machine Learning > Optimization & Theory > Statistical Learning Machine Learning > Optimization & Theory > Stochastic Methods Deep Learning > Optimization & Theory > Optimization Machine Learning > Learning Types > Regularization Deep Learning > Optimization & Theory > Regularization

Keywords

statistical learning stochastic gradient stochastic gradient descent incremental gradient universal consistency regularization parameter least square finite sample bound iterative regularization

More from NeurIPS 2015

All NeurIPS 2015 papers →

Learning Continuous Control Policies by Stochastic Value Gradients

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Efficient Non-greedy Optimization of Decision Trees

Maximum Likelihood Learning With Arbitrary Treewidth via Fast-Mixing Parameter Sets

Sampling from Probabilistic Submodular Models

A class of network models recoverable by spectral clustering