conftrace_

initialization scheme

10 papers

Explore in graph

Co-occurring keywords · papers mentioning both

mean field theory (4) signal propagation (3) neural network optimization (3) vanishing gradient (2) dynamical isometry (2) gradient flow (2) trainable depth (1) orthogonal initialization (1) scalar regression (1) gradient explosion (1)

Papers

Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models ICLR 2025 Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win AAAI 2022 Fractional moment-preserving initialization schemes for training deep neural networks AISTATS 2021 Optimizing Deeper Transformers on Small Datasets ACL 2021 Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks ICLR 2020 A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks ICLR 2019 A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off NeurIPS 2019 Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks ICML 2018 Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks ICML 2018 Mean Field Residual Networks: On the Edge of Chaos NeurIPS 2017