conftrace_

2012 NeurIPS NeurIPS 2012

A Better Way to Pretrain Deep Boltzmann Machines

Abstract

We describe how the pre-training algorithm for Deep Boltzmann Machines (DBMs) is related to the pre-training algorithm for Deep Belief Networks and we show that under certain conditions, the pre-training procedure improves the variational lower bound of a two-hidden-layer DBM. Based on this analysis, we develop a different method of pre-training DBMs that distributes the modelling work more evenly over the hidden layers. Our results on the MNIST and NORB datasets demonstrate that the new pre-training algorithm allows us to learn better generative models.

📈 Trend Setter - Generative Model

🧭 Keyword Pioneer - variational lower bound

🐣 Hot Topic Early Bird - generative model

Authors

Geoffrey E. Hinton, Ruslan Salakhutdinov

Topics

Machine Learning > Core Methods > Representation Learning Deep Learning > Architectures > Neural Networks Deep Learning > Models > Generative Models Deep Learning > Techniques > Pretraining Machine Learning > Bayesian & Probabilistic > Probabilistic Modeling Deep Learning > Learning Types > Generative Model

Keywords

variational inference unsupervised feature learning pretraining greedy layer-wise training hidden layer variational pretraining generative model deep boltzmann machine variational lower bound deep belief network pretraining algorithm

More from NeurIPS 2012

All NeurIPS 2012 papers →

Exact and Stable Recovery of Sequences of Signals with Sparse Increments via Differential _1-Minimization

Tight Bounds on Profile Redundancy and Distinguishability

On Triangular versus Edge Representations --- Towards Scalable Modeling of Networks

Semi-supervised Eigenvectors for Locally-biased Learning

The variational hierarchical EM algorithm for clustering hidden Markov models

Scalable nonconvex inexact proximal splitting