conftrace_

data parallelism

17 papers

Explore in graph

Co-occurring keywords

distributed training (188) model parallelism (43) neural network training (309) memory optimization (114) communication efficiency (297) deep learning (2118) gradient quantization (27) parallel training (37) large language model (13587) distributed deep learning (13)

Papers

BitDP: Ultra-low-bit Communication for Data Parallelism in LLM Training AAAI 2026
Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters AAAI 2025
LegoMT2: Selective Asynchronous Sharded Data Parallel Training for Massive Neural Machine Translation ACL 2025
Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models NIPS 2024
DISTMM: Accelerating Distributed Multimodal Model Training NSDI 2024
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning IJCAI 2023
H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training NIPS 2023
SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training NIPS 2022
Decentralized Training of Foundation Models in Heterogeneous Environments NIPS 2022
Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs CVPR 2022
Ouroboros: On Accelerating Training of Transformer-Based Language Models NIPS 2019
Measuring the Effects of Data Parallelism on Neural Network Training JMLR 2019
Exploring Hidden Dimensions in Accelerating Convolutional Neural Networks ICML 2018
Mesh-TensorFlow: Deep Learning for Supercomputers NIPS 2018
On-the-fly Operation Batching in Dynamic Computation Graphs NIPS 2017
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning NIPS 2017
Large Scale Distributed Deep Networks NIPS 2012