Neural Network Optimization

ISAAC Newton: Input-based Approximate Curvature for Newton's Method ICLR 2023 Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures NeurIPS 2023 LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagation ICML 2023 Learning to Boost Training by Periodic Nowcasting Near Future Weights ICML 2023 MathNAS: If Blocks Have a Role in Mathematical Architecture Design NeurIPS 2023 Maximal Initial Learning Rates in Deep ReLU Networks ICML 2023 MDL-NAS: A Joint Multi-Domain Learning Framework for Vision Transformer CVPR 2023 Mimetic Initialization of Self-Attention Layers ICML 2023 Momentum Ensures Convergence of SIGNSGD under Weaker Assumptions ICML 2023 nanoT5: Fast & Simple Pre-training and Fine-tuning of T5 Models with Limited Resources EMNLP 2023 Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models ACL 2023 Neural Architecture Search for Wide Spectrum Adversarial Robustness AAAI 2023 NIPQ: Noise Proxy-Based Integrated Pseudo-Quantization CVPR 2023 Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be ICLR 2023 Nyström Method for Accurate and Scalable Implicit Differentiation AISTATS 2023 On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Size JMLR 2023 On the Dimensionality of Sentence Embeddings EMNLP 2023 On the Expressivity Role of LayerNorm in Transformers’ Attention ACL 2023 On the Initialization of Graph Neural Networks ICML 2023 Oscillation-free Quantization for Low-bit Vision Transformers ICML 2023 Partition Speeds Up Learning Implicit Neural Representations Based on Exponential-Increase Hypothesis ICCV 2023 Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width NeurIPS 2023 Polarity Is All You Need to Learn and Transfer Faster ICML 2023 Polynomial Implicit Neural Representations for Large Diverse Datasets CVPR 2023 Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima NeurIPS 2023