Neural Network Optimization

Symmetries in Overparametrized Neural Networks: A Mean Field View NeurIPS 2024 SynSP: Synergy of Smoothness and Precision in Pose Sequences Refinement CVPR 2024 T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text ACL 2024 Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks NeurIPS 2024 Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergence AISTATS 2024 Tending Towards Stability: Convergence Challenges in Small Language Models EMNLP 2024 TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision ICML 2024 Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks ICLR 2024 TFWT: Tabular Feature Weighting with Transformer IJCAI 2024 The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents ICML 2024 The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks NeurIPS 2024 The Closeness of In-Context Learning and Weight Shifting for Softmax Regression NeurIPS 2024 The Dormant Neuron Phenomenon in Multi-Agent Reinforcement Learning Value Factorization NeurIPS 2024 The effect of Leaky ReLUs on the training and generalization of overparameterized networks AISTATS 2024 The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof NeurIPS 2024 The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks NeurIPS 2024 The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms NeurIPS 2024 The Impact of Depth on Compositional Generalization in Transformer Language Models NAACL 2024 The Impact of Initialization on LoRA Finetuning Dynamics NeurIPS 2024 The Implicit Bias of Adam on Separable Data NeurIPS 2024 The Implicit Bias of Gradient Descent on Separable Multiclass Data NeurIPS 2024 The Implicit Bias of Gradient Descent toward Collaboration between Layers: A Dynamic Analysis of Multilayer Perceptions NeurIPS 2024 The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information NeurIPS 2024 The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space NeurIPS 2024 The Marginal Value of Momentum for Small Learning Rate SGD ICLR 2024