conftrace_

2017 NeurIPS NeurIPS 2017

Bayesian Compression for Deep Learning

Abstract

Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian point of view, where through sparsity inducing priors we prune large parts of the network. We introduce two novelties in this paper: 1) we use hierarchical priors to prune nodes instead of individual weights, and 2) we use the posterior uncertainties to determine the optimal fixed point precision to encode the weights. Both factors significantly contribute to achieving the state of the art in terms of compression rates, while still staying competitive with methods designed to optimize for speed or energy efficiency.

🐣 Hot Topic Early Bird - weight quantization

Authors

Christos Louizos, Karen Ullrich, Max Welling

Topics

Machine Learning > Optimization & Theory > Bayesian Inference Machine Learning > Application Areas > Efficient Computing Deep Learning > Models > Generative Models Machine Learning > Application Areas > Model Compression Artificial Intelligence > Bayesian & Probabilistic > Bayesian Inference Deep Learning > Optimization & Theory > Model Compression

Keywords

model compression neural network pruning network pruning weight quantization posterior uncertainty sparsity-inducing prior bayesian compression

More from NeurIPS 2017

All NeurIPS 2017 papers →

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Log-normality and Skewness of Estimated State/Action Values in Reinforcement Learning

Is Input Sparsity Time Possible for Kernel Low-Rank Approximation?

Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks

Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes