Jeffrey Pennington

Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks ICML 2025 4+3 Phases of Compute-Optimal Neural Scaling Laws NeurIPS 2024 Scaling Exponents Across Parameterizations and Optimizers ICML 2024 Small-scale proxies for large-scale Transformer training instabilities ICLR 2024 Second-order regression models exhibit progressive sharpening to the edge of stability ICML 2023 Anisotropic Random Feature Regression in High Dimensions ICLR 2022 A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions AISTATS 2022 Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions NeurIPS 2022 Precise Learning Curves and Higher-Order Scalings for Dot-product Kernel Regression NeurIPS 2022 Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm ICML 2022 Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling ICML 2022 Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit ICLR 2021 Overparameterization Improves Robustness to Covariate Shift in High Dimensions NeurIPS 2021 Disentangling Trainability and Generalization in Deep Neural Networks ICML 2020 Finite Versus Infinite Neural Networks: an Empirical Study NeurIPS 2020 Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks ICLR 2020 The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization ICML 2020 The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks NeurIPS 2020 Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition NeurIPS 2020 A Mean Field Theory of Batch Normalization ICLR 2019 Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes ICLR 2019 KAMA-NNs: Low-dimensional Rotation Based Neural Networks AISTATS 2019 Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent NeurIPS 2019 Deep Neural Networks as Gaussian Processes ICLR 2018 Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks ICML 2018 Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks ICML 2018 Sensitivity and Generalization in Neural Networks: an Empirical Study ICLR 2018 The emergence of spectral universality in deep networks AISTATS 2018 The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network NeurIPS 2018 Geometry of Neural Network Loss Surfaces via Random Matrix Theory ICML 2017 Nonlinear random matrix theory for deep learning NeurIPS 2017 Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice NeurIPS 2017 Spherical Random Features for Polynomial Kernels NeurIPS 2015 GloVe: Global Vectors for Word Representation EMNLP 2014 Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions EMNLP 2011