conftrace_

pipeline parallelism

35 papers

Explore in graph

Also known as

PP

Co-occurring keywords · papers mentioning both

distributed training (15) large language model (7) model parallelism (6) memory efficiency (4) throughput optimization (3) mixture of expert (3) model training (3) activation memory (2) data parallelism (2) communication overhead (2)

Papers

BPipe: Memory-Balanced Pipeline Parallelism for Training Large Language Models ICML 2023 SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient ICML 2023 Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees NeurIPS 2022 Group-based Interleaved Pipeline Parallelism for Large-scale DNN Training ICLR 2022 TSPipe: Learn from Teacher Faster with Pipelines ICML 2022 Memory-Efficient Pipeline-Parallel DNN Training ICML 2021 P3: Distributed Deep Graph Learning at Scale OSDI 2021 TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models ICML 2021 GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism NeurIPS 2019 Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learning NeurIPS 2019