conftrace_

2021 JMLR JMLR 2021

When Does Gradient Descent with Logistic Loss Find Interpolating Two-Layer Networks?

Abstract

We study the training of finite-width two-layer smoothed ReLU networks for binary classification using the logistic loss. We show that gradient descent drives the training loss to zero if the initial loss is small enough. When the data satisfies certain cluster and separation conditions and the network is wide enough, we show that one step of gradient descent reduces the loss sufficiently that the first result applies. [abs] [ pdf ][ bib ] © JMLR 2021. (edit, beta)

❓ The Questioner

Authors

Niladri S. Chatterji, Philip M. Long, Peter L. Bartlett

Topics

Machine Learning > Core Methods > Representation Learning Machine Learning > Optimization & Theory > Learning Theory Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Learning Types > Supervised Learning Deep Learning > Optimization & Theory > Optimization

Keywords

binary classification neural network optimization gradient descent logistic loss two-layer network interpolating network

More from JMLR 2021

All JMLR 2021 papers →

A Greedy Algorithm for Quantizing Neural Networks

The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks

Information criteria for non-normalized models

Are We Forgetting about Compositional Optimisers in Bayesian Optimisation?

MetaGrad: Adaptation using Multiple Learning Rates in Online Learning

Counterfactual Mean Embeddings