conftrace_

transformer training

13 papers

Explore in graph

Co-occurring keywords

neural network optimization (1293) transformer model (1996) memory optimization (114) attention mechanism (4001) language model (4599) model parallelism (43) efficient training (51) transformer architecture (1560) residual connection (55) outlier feature (2)

Papers

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers NIPS 2024
Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers NIPS 2024
Global Convergence in Training Large-Scale Transformers NIPS 2024
Understanding and Minimising Outlier Features in Transformer Training NIPS 2024
H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training NIPS 2023
Dynamic Stashing Quantization for Efficient Transformer Training EMNLP 2023
Softmax Output Approximation for Activation Memory-Efficient Training of Attention-based Networks NIPS 2023
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction NIPS 2022
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent EMNLP 2021
Speeding Up Transformer Training By Using Dataset Subsampling - An Exploratory Analysis EMNLP 2021
Understanding the Difficulty of Training Transformers EMNLP 2020
Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change ACL 2020
Making Asynchronous Stochastic Gradient Descent Work for Transformers EMNLP 2019