Neural Network Optimization

A Mixture-of-Experts Model for Antonym-Synonym Discrimination ACL 2021 A Modular Analysis of Provable Acceleration via Polyak’s Momentum: Training a Wide ReLU Network and a Deep Linear Network ICML 2021 An algorithmic view of L2 regularization and some path-following algorithms JMLR 2021 Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems NeurIPS 2021 Analytic Insights into Structure and Rank of Neural Network Hessian Maps NeurIPS 2021 Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II NeurIPS 2021 An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models ACL 2021 An Improved Single Step Non-Autoregressive Transformer for Automatic Speech Recognition Interspeech 2021 An Optimistic Acceleration of AMSGrad for Nonconvex Optimization ACML 2021 An optimization and generalization analysis for max-pooling networks UAI 2021 A Novel Method to Solve Neural Knapsack Problems ICML 2021 Approximate Newton Methods JMLR 2021 Approximation Theory of Convolutional Architectures for Time Series Modelling ICML 2021 A Recipe for Global Convergence Guarantee in Deep Neural Networks AAAI 2021 Argument Pair Extraction via Attention-guided Multi-Layer Multi-Cross Encoding ACL 2021 A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance ICML 2021 A single gradient step finds adversarial examples on random two-layers neural networks NeurIPS 2021 A Study of Condition Numbers for First-Order Optimization AISTATS 2021 A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix AISTATS 2021 A Trace-restricted Kronecker-Factored Approximation to Natural Gradient AAAI 2021 AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing AAAI 2021 Augmented Shortcuts for Vision Transformers NeurIPS 2021 A Unified Encoding of Structures in Transition Systems EMNLP 2021 A unifying view on implicit bias in training linear neural networks ICLR 2021 A Universal Law of Robustness via Isoperimetry NeurIPS 2021