conftrace_

signal propagation

20 papers

Explore in graph

Co-occurring keywords

weight initialization (50) deep neural network (1803) neural network (6616) neural network optimization (1293) residual network (211) dynamical isometry (8) initialization scheme (8) mean field theory (20) random matrix theory (63) neural network initialization (33)

Papers

Transformers need glasses! Information over-squashing in language tasks NIPS 2024
On Feature Learning in Structured State Space Models NIPS 2024
Nonlinear spiked covariance matrices and signal propagation in deep neural networks COLT 2024
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks AAAI 2023
Principled Weight Initialisation for Input-Convex Neural Networks NIPS 2023
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse NIPS 2022
Feature Learning and Signal Propagation in Deep Neural Networks ICML 2022
Sparse linear networks with a fixed butterfly structure: theory and practice UAI 2021
Fractional moment-preserving initialization schemes for training deep neural networks AISTATS 2021
Regularization in ResNet with Stochastic Depth NIPS 2021
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization? ICML 2020
Lamina-specific neuronal properties promote robust, stable signal propagation in feedforward networks NIPS 2020
Characterizing Well-Behaved vs. Pathological Deep Neural Networks ICML 2019
Initialization of ReLUs for Dynamical Isometry NIPS 2019
Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function AISTATS 2019
A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off NIPS 2019
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks ICML 2018
Critical initialisation for deep signal propagation in noisy rectifier neural networks NIPS 2018
Optimal Architectures in a Solvable Model of Deep Networks NIPS 2016
Global seismic monitoring as probabilistic inference NIPS 2010