Neural Network Optimization

Learnability of high-dimensional targets by two-parameter models and gradient flow NeurIPS 2024 Learning Discretized Neural Networks under Ricci Flow JMLR 2024 Learning Neural Contracting Dynamics: Extended Linearization and Global Guarantees NeurIPS 2024 Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks AISTATS 2024 Linking Convolutional Kernel Size to Generalization Bias in Face Analysis CNNs WACV 2024 LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning NeurIPS 2024 Local to Global: Learning Dynamics and Effect of Initialization for Transformers NeurIPS 2024 Loki: Low-rank Keys for Efficient Sparse Attention NeurIPS 2024 LoRA-GA: Low-Rank Adaptation with Gradient Approximation NeurIPS 2024 LPZero: Language Model Zero-cost Proxy Search from Zero EMNLP 2024 MaxQ: Multi-Axis Query for N:M Sparsity Network CVPR 2024 Memory-Efficient LLM Training with Online Subspace Descent NeurIPS 2024 MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence NeurIPS 2024 MLP Can Be A Good Transformer Learner CVPR 2024 MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts NeurIPS 2024 Monomial Matrix Group Equivariant Neural Functional Networks NeurIPS 2024 Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers CVPR 2024 Nature-Inspired Local Propagation NeurIPS 2024 Neural Collapse To Multiple Centers For Imbalanced Data NeurIPS 2024 Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit NeurIPS 2024 Neural Redshift: Random Networks are not Random Functions CVPR 2024 Noise-Free Optimization in Early Training Steps for Image Super-resolution AAAI 2024 Non-asymptotic Convergence of Training Transformers for Next-token Prediction NeurIPS 2024 Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers NeurIPS 2024 OmniSDF: Scene Reconstruction using Omnidirectional Signed Distance Functions and Adaptive Binoctrees CVPR 2024