Optimization & Theory

Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids’ Representations EMNLP 2021 Not all parameters are born equal: Attention is mostly what you need EMNLP 2021 NPAS: A Compiler-Aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration CVPR 2021 Numerical influence of ReLU’(0) on backpropagation NeurIPS 2021 NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization JMLR 2021 Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food CVPR 2021 Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention AAAI 2021 On ADMM in Deep Learning: Convergence and Saturation-Avoidance JMLR 2021 On-Device Streaming Transformer-Based End-to-End Speech Recognition Interspeech 2021 On Energy-Based Models with Overparametrized Shallow Neural Networks ICML 2021 On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective CVPR 2021 On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting ICML 2021 On Linear Stability of SGD and Input-Smoothness of Neural Networks NeurIPS 2021 Online stochastic gradient descent on non-convex losses from high-dimensional inference JMLR 2021 On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory AAAI 2021 Only Train Once: A One-Shot Neural Network Training And Pruning Framework NeurIPS 2021 On Monotonic Linear Interpolation of Neural Network Parameters ICML 2021 On Orthogonality Constraints for Transformers ACL 2021 On Smoother Attributions using Neural Stochastic Differential Equations IJCAI 2021 On the Adequacy of Untuned Warmup for Adaptive Optimization AAAI 2021 On the Approximation Power of Two-Layer Networks of Random ReLUs COLT 2021 On the Convergence of Communication-Efficient Local SGD for Federated Learning AAAI 2021 On the Convergence of Step Decay Step-Size for Stochastic Optimization NeurIPS 2021 On the Difficulty of Membership Inference Attacks CVPR 2021 On the Dynamics of Training Attention Models ICLR 2021