distributed training

Towards Domain-Specific Network Transport for Distributed DNN Training NSDI 2024 Zero Bubble (Almost) Pipeline Parallelism ICLR 2024 A^2CiD^2: Accelerating Asynchronous Communication in Decentralized Deep Learning NeurIPS 2023 Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs NSDI 2023 BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing NSDI 2023 Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training NeurIPS 2023 BPipe: Memory-Balanced Pipeline Parallelism for Training Large Language Models ICML 2023 CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks ICML 2023 CoLLiE: Collaborative Training of Large Language Models in an Efficient Way EMNLP 2023 Coordinating Distributed Example Orders for Provably Accelerated Training NeurIPS 2023 Deep Incubation: Training Large Models by Divide-and-Conquering ICCV 2023 DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training CVPR 2023 DropCompute: simple and more robust distributed synchronous training via compute variance reduction NeurIPS 2023 Federated Learning Toolkit with Voice-based User Verification Demo Interspeech 2023 Fedstellar: A Platform for Training Models in a Privacy-preserving and Decentralized Fashion IJCAI 2023 Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-per-Second CVPR 2023 Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data NeurIPS 2023 H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training NeurIPS 2023 LOFT: Finding Lottery Tickets through Filter-wise Training AISTATS 2023 Lowering the Pre-training Tax for Gradient-based Subset Training: A Lightweight Distributed Pre-Training Toolkit ICML 2023 Making Scalable Meta Learning Practical NeurIPS 2023 Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam ICLR 2023 POPGym: Benchmarking Partially Observable Reinforcement Learning ICLR 2023 Quantized Distributed Training of Large Models with Convergence Guarantees ICML 2023 Sancus: Staleness-Aware Communication-Avoiding Full-Graph Decentralized Training in Large-Scale Graph Neural Networks (Extended Abstract) IJCAI 2023