Neural Network Optimization

LoRA Training in the NTK Regime has No Spurious Local Minima ICML 2024 Loss Landscape Characterization of Neural Networks without Over-Parametrization NeurIPS 2024 Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach CVPR 2024 LPGD: A General Framework for Backpropagation through Embedded Optimization Layers ICML 2024 lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning ICLR 2024 LPZero: Language Model Zero-cost Proxy Search from Zero EMNLP 2024 Lyapunov-stable Neural Control for State and Output Feedback: A Novel Formulation ICML 2024 Magnitude Invariant Parametrizations Improve Hypernetwork Learning ICLR 2024 Make Large Language Model a Better Ranker EMNLP 2024 MambaTree: Tree Topology is All You Need in State Space Model NeurIPS 2024 Manipulating dropout reveals an optimal balance of efficiency and robustness in biological and machine visual systems ICLR 2024 Masked Structural Growth for 2x Faster Language Model Pre-training ICLR 2024 Masks, Signs, And Learning Rate Rewinding ICLR 2024 Matching the Statistical Query Lower Bound for k-Sparse Parity Problems with Sign Stochastic Gradient Descent NeurIPS 2024 MD tree: a model-diagnostic tree grown on loss landscape ICML 2024 Mechanics of Next Token Prediction with Self-Attention AISTATS 2024 MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning ACL 2024 Memory-Efficient Fine-Tuning of Transformers via Token Selection EMNLP 2024 Memory-Efficient LLM Training with Online Subspace Descent NeurIPS 2024 MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning AAAI 2024 MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence NeurIPS 2024 Mini-Sequence Transformers: Optimizing Intermediate Memory for Long Sequences Training NeurIPS 2024 Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes NeurIPS 2024 MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters WACV 2024 Mixture of In-Context Experts Enhance LLMs' Long Context Awareness NeurIPS 2024