Neural Network Optimization

Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling AAAI 2021 Layerwise Optimization by Gradient Decomposition for Continual Learning CVPR 2021 Learning explanations that are hard to vary ICLR 2021 Learning Graph Neural Networks with Approximate Gradient Descent AAAI 2021 Learning Neural Network Subspaces ICML 2021 Learning with Retrospection AAAI 2021 LeeBERT: Learned Early Exit for BERT with cross-level optimization ACL 2021 Locally Free Weight Sharing for Network Width Search ICLR 2021 Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling ICML 2021 LQF: Linear Quadratic Fine-Tuning CVPR 2021 MLE-Guided Parameter Search for Task Loss Minimization in Neural Sequence Modeling AAAI 2021 MTAdam: Automatic Balancing of Multiple Training Loss Terms EMNLP 2021 Multiplicative Noise and Heavy Tails in Stochastic Optimization ICML 2021 Network Quantization With Element-Wise Gradient Scaling CVPR 2021 Neural Architecture Search With Random Labels CVPR 2021 Neural gradients are near-lognormal: improved quantized and sparse training ICLR 2021 Nondeterminism and Instability in Neural Network Optimization ICML 2021 Numerical influence of ReLU’(0) on backpropagation NeurIPS 2021 On ADMM in Deep Learning: Convergence and Saturation-Avoidance JMLR 2021 On Energy-Based Models with Overparametrized Shallow Neural Networks ICML 2021 On Linear Stability of SGD and Input-Smoothness of Neural Networks NeurIPS 2021 On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory AAAI 2021 On Monotonic Linear Interpolation of Neural Network Parameters ICML 2021 On the Adequacy of Untuned Warmup for Adaptive Optimization AAAI 2021 On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks ICML 2021