Distributed Learning

DropCompute: simple and more robust distributed synchronous training via compute variance reduction NeurIPS 2023 Learning Kernelized Contextual Bandits in a Distributed and Asynchronous Environment ICLR 2023 MocoSFL: enabling cross-client collaborative self-supervised learning ICLR 2023 Petals: Collaborative Inference and Fine-tuning of Large Models ACL 2023 Why (and When) does Local SGD Generalize Better than SGD? ICLR 2023 Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI 2022 Decentralized Learning for Overparameterized Problems: A Multi-Agent Kernel Approximation Approach ICLR 2022 Decentralized Training of Foundation Models in Heterogeneous Environments NeurIPS 2022 Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences ICLR 2022 PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication ICLR 2022 Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations ICLR 2022 Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads OSDI 2021 EasyASR: A Distributed Machine Learning Platform for End-to-end Automatic Speech Recognition AAAI 2021 Effective Distributed Learning with Random Features: Improved Bounds and Algorithms ICLR 2021 On InstaHide, Phase Retrieval, and Sparse Matrix Factorization ICLR 2021 P3: Distributed Deep Graph Learning at Scale OSDI 2021 Learning Efficient Parameter Server Synchronization Policies for Distributed SGD ICLR 2020 On the Convergence of FedAvg on Non-IID Data ICLR 2020 Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training EMNLP 2019 Local SGD Converges Fast and Communicates Little ICLR 2019 Large scale distributed neural network training through online distillation ICLR 2018 The Effect of Network Width on the Performance of Large-batch Training NeurIPS 2018 Collaborative Deep Learning in Fixed Topology Networks NeurIPS 2017 Large Scale Distributed Deep Networks NeurIPS 2012 Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models NeurIPS 2009