Neural Network Optimization

Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers NeurIPS 2023 Principled Weight Initialisation for Input-Convex Neural Networks NeurIPS 2023 Regression as Classification: Influence of Task Formulation on Neural Network Features AISTATS 2023 Resilient Binary Neural Network AAAI 2023 Rethinking Gauss-Newton for learning over-parameterized models NeurIPS 2023 Revisiting Reverse Distillation for Anomaly Detection CVPR 2023 Revisiting Token Dropping Strategy in Efficient BERT Pretraining ACL 2023 Self-Evolution Learning for Discriminative Language Model Pretraining ACL 2023 SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics COLT 2023 SGD with Large Step Sizes Learns Sparse Features ICML 2023 Shakes on a Plane: Unsupervised Depth Estimation From Unstabilized Photography CVPR 2023 SLAMB: Accelerated Large Batch Training with Sparse Communication ICML 2023 SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge ICML 2023 SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer CVPR 2023 Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks NeurIPS 2023 Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks EMNLP 2023 Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training EMNLP 2023 Symbolic Discovery of Optimization Algorithms NeurIPS 2023 The Hessian perspective into the Nature of Convolutional Neural Networks ICML 2023 The Implicit Regularization of Momentum Gradient Descent in Overparametrized Models AAAI 2023 TIPI: Test Time Adaptation With Transformation Invariance CVPR 2023 TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models EMNLP 2023 Token-Level Self-Evolution Training for Sequence-to-Sequence Learning ACL 2023 Towards Sustainable Learning: Coresets for Data-efficient Deep Learning ICML 2023 Training Your Image Restoration Network Better with Random Weight Network as Optimization Function NeurIPS 2023