distributed training

Sequence Parallelism: Long Sequence Training from System Perspective ACL 2023 Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning NSDI 2023 SLAMB: Accelerated Large Batch Training with Sparse Communication ICML 2023 SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient ICML 2023 TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches NSDI 2023 TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs NSDI 2023 Toward Large Kernel Models ICML 2023 Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions ICLR 2023 trlX: A Framework for Large Scale Open Source RLHF EMNLP 2023 trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback EMNLP 2023 3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation ICML 2022 A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks NeurIPS 2022 A Deep Learning Dataloader with Shared Data Preparation NeurIPS 2022 Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI 2022 AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS 2022 DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems AAAI 2022 DRAGONN: Distributed Randomized Approximate Gradients of Neural Networks ICML 2022 Federated Learning With Position-Aware Neurons CVPR 2022 FedNST: Federated Noisy Student Training for Automatic Speech Recognition Interspeech 2022 FLAIR: Federated Learning Annotated Image Repository NeurIPS 2022 GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Models NeurIPS 2022 Group-based Interleaved Pipeline Parallelism for Large-scale DNN Training ICLR 2022 Interlocking Backpropagation: Improving depthwise model-parallelism JMLR 2022 Intrinsic Gradient Compression for Scalable and Efficient Federated Learning ACL 2022 On the Convergence of Decentralized Adaptive Gradient Methods ACML 2022