Neural Network Optimization

Transformer-Based Learned Optimization CVPR 2023 Transformer Language Models Handle Word Frequency in Prediction Head ACL 2023 Transformers learn through gradual rank increase NeurIPS 2023 Transformers learn to implement preconditioned gradient descent for in-context learning NeurIPS 2023 Transformers with Learnable Activation Functions EACL 2023 Turbulence in Focus: Benchmarking Scaling Behavior of 3D Volumetric Super-Resolution with BLASTNet 2.0 Data NeurIPS 2023 Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning ACL 2023 Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods NeurIPS 2023 Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models ACL 2023 Understanding Edge-of-Stability Training Dynamics with a Minimalist Example ICLR 2023 Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias ICML 2023 Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing ICML 2023 Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks NeurIPS 2023 Understanding Plasticity in Neural Networks ICML 2023 Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL NeurIPS 2023 Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization ICLR 2023 Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning ICLR 2023 Understanding weight-magnitude hyperparameters in training binary networks ICLR 2023 Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning EMNLP 2023 Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent NeurIPS 2023 UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction SemEval 2023 When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants ACL 2023 Which Layer is Learning Faster? A Systematic Exploration of Layer-wise Convergence Rate for Deep Neural Networks ICLR 2023 Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers ACL 2023 Why Does Sharpness-Aware Minimization Generalize Better Than SGD? NeurIPS 2023