Neural Network Optimization

Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning CVPR 2024 Understanding and Minimising Outlier Features in Transformer Training NeurIPS 2024 Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective NeurIPS 2024 Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling NeurIPS 2024 Unraveling the Gradient Descent Dynamics of Transformers NeurIPS 2024 Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance NeurIPS 2024 Weight decay induces low-rank attention layers NeurIPS 2024 What does automatic differentiation compute for neural networks? ICLR 2024 When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks AAAI 2024 Where Do Large Learning Rates Lead Us? NeurIPS 2024 Why Do We Need Weight Decay in Modern Deep Learning? NeurIPS 2024 Why is parameter averaging beneficial in SGD? An objective smoothing perspective AISTATS 2024 Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training JMLR 2024 Zero-Shot Transfer of Neural ODEs NeurIPS 2024 Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers CVPR 2024 Acceleration of Large Transformer Model Training by Sensitivity-Based Layer Dropping AAAI 2023 Achieving High Accuracy with PINNs via Energy Natural Gradient Descent ICML 2023 A Corrected Expected Improvement Acquisition Function Under Noisy Observations ACML 2023 AdaNorm: Adaptive Gradient Norm Correction Based Optimizer for CNNs WACV 2023 Adaptive Computation with Elastic Input Sequence ICML 2023 Addressing the Length Bias Challenge in Document-Level Neural Machine Translation EMNLP 2023 AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix NeurIPS 2023 A Guide Through the Zoo of Biased SGD NeurIPS 2023 AIO-P: Expanding Neural Performance Predictors beyond Image Classification AAAI 2023 A Length-Extrapolatable Transformer ACL 2023