Neural Network Optimization

Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training NeurIPS 2024 A Neural-Preconditioned Poisson Solver for Mixed Dirichlet and Neumann Boundary Conditions ICML 2024 An Eye Opener Regarding Task-Based Text Gradient Saliency ACL 2024 An Improved Empirical Fisher Approximation for Natural Gradient Descent NeurIPS 2024 A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks JMLR 2024 A Physics-informed Low-rank Deep Neural Network for Blind and Universal Lens Aberration Correction CVPR 2024 Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient NeurIPS 2024 A Precise Characterization of SGD Stability Using Loss Surface Geometry ICLR 2024 A Progressive Training Framework for Spiking Neural Networks with Learnable Multi-hierarchical Model ICLR 2024 A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness NeurIPS 2024 A Quadratic Synchronization Rule for Distributed Deep Learning ICLR 2024 Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? ICLR 2024 A Simple and Effective Pruning Approach for Large Language Models ICLR 2024 A Structure-Aware Framework for Learning Device Placements on Computation Graphs NeurIPS 2024 A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning NeurIPS 2024 A Sublinear Adversarial Training Algorithm ICLR 2024 Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation NAACL 2024 Auto-Evolve: Enhancing Large Language Model’s Performance via Self-Reasoning Framework EMNLP 2024 Autothrottle: A Practical Bi-Level Approach to Resource Management for SLO-Targeted Microservices NSDI 2024 Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch CVPR 2024 BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models NeurIPS 2024 BadEdit: Backdooring Large Language Models by Model Editing ICLR 2024 Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning ACL 2024 Batch Normalization Alleviates the Spectral Bias in Coordinate Networks CVPR 2024 Batch Normalization Is Blind to the First and Second Derivatives of the Loss AAAI 2024