conftrace_

model parallelism

43 papers

Explore in graph

Co-occurring keywords

distributed training (188) data parallelism (17) distributed computing (98) parallel computing (146) communication efficiency (297) pipeline parallelism (21) memory optimization (114) large language model (13587) neural network optimization (1293) distributed learning (563)

Papers

Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers ICML 2022
Decentralized Training of Foundation Models in Heterogeneous Environments NIPS 2022
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction NIPS 2022
Piper: Multidimensional Planner for DNN Parallelization NIPS 2021
Gradient Forward-Propagation for Large-Scale Temporal Video Modelling CVPR 2021
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models ICML 2021
Memory-Efficient Pipeline-Parallel DNN Training ICML 2021
Efficient Algorithms for Device Placement of DNN Graph Operators NIPS 2020
On the Acceleration of Deep Learning Model Parallelism With Staleness CVPR 2020
Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products NIPS 2019
Ouroboros: On Accelerating Training of Transformer-Based Language Models NIPS 2019
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism NIPS 2019
Scalable Distributed DL Training: Batching Communication and Computation AAAI 2019
Exploring Hidden Dimensions in Accelerating Convolutional Neural Networks ICML 2018
Mesh-TensorFlow: Deep Learning for Supercomputers NIPS 2018
On Model Parallelization and Scheduling Strategies for Distributed Machine Learning NIPS 2014
Deep learning with COTS HPC systems ICML 2013
Large Scale Distributed Deep Networks NIPS 2012