conftrace_

dynamical isometry

8 papers

Explore in graph

Co-occurring keywords

signal propagation (20) free probability theory (4) gradient propagation (34) initialization scheme (8) weight initialization (50) random matrix theory (63) mean field theory (20) infinite width (20) spectral distribution (10) vanishing gradient (43)

Papers

ReZero is all you need: fast convergence at large depth UAI 2021
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry AISTATS 2021
On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization IJCAI 2021
Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function AISTATS 2019
Initialization of ReLUs for Dynamical Isometry NIPS 2019
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks ICML 2018
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks ICML 2018
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice NIPS 2017