model parallelism

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient ICML 2023 Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI 2022 AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS 2022 Decentralized Training of Foundation Models in Heterogeneous Environments NeurIPS 2022 Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers ICML 2022 Interlocking Backpropagation: Improving depthwise model-parallelism JMLR 2022 Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction NeurIPS 2022 Gradient Forward-Propagation for Large-Scale Temporal Video Modelling CVPR 2021 Memory-Efficient Pipeline-Parallel DNN Training ICML 2021 Piper: Multidimensional Planner for DNN Parallelization NeurIPS 2021 TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models ICML 2021 Efficient Algorithms for Device Placement of DNN Graph Operators NeurIPS 2020 On the Acceleration of Deep Learning Model Parallelism With Staleness CVPR 2020 Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products NeurIPS 2019 GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism NeurIPS 2019 Ouroboros: On Accelerating Training of Transformer-Based Language Models NeurIPS 2019 Scalable Distributed DL Training: Batching Communication and Computation AAAI 2019 Decoupling the Layers in Residual Networks ICLR 2018 Exploring Hidden Dimensions in Accelerating Convolutional Neural Networks ICML 2018 Mesh-TensorFlow: Deep Learning for Supercomputers NeurIPS 2018 On Model Parallelization and Scheduling Strategies for Distributed Machine Learning NeurIPS 2014 Deep learning with COTS HPC systems ICML 2013 Large Scale Distributed Deep Networks NeurIPS 2012