conftrace_

dynamical isometry

9 papers

Explore in graph

Co-occurring keywords · papers mentioning both

signal propagation (3) free probability theory (2) gradient propagation (2) initialization scheme (2) weight initialization (2) random matrix theory (2) mean field theory (2) infinite width (1) spectral distribution (1) vanishing gradient (1)

Papers

Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization ICML 2024 On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization IJCAI 2021 ReZero is all you need: fast convergence at large depth UAI 2021 The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry AISTATS 2021 Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function AISTATS 2019 Initialization of ReLUs for Dynamical Isometry NeurIPS 2019 Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks ICML 2018 Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks ICML 2018 Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice NeurIPS 2017