Fast Block Diffusion Training via Multi-Mask Context Sharing
Abstract
AbstractBlock Diffusion Language Models generate multiple blocks of text sequentially, while tokens within each block are denoised in parallel, enabling efficient parallel decoding. However, their training suffers from low computational utilization: each sample is duplicated into a clean context and a masked copy, yet only the masked portion contributes to the training loss, resulting in merely 50% compute utilization. In this paper, we propose Multi-Mask Training (MMT), a simple yet effective training paradigm that significantly improves computational efficiency through context sharing. Our key idea is to generate K different mask patterns for the same clean text and process them together in a single forward pass via a sparse attention mechanism, where each masked segment attends to the shared clean context while remaining isolated from other segments. In pretraining, our proposed approach yields better performance under matched compute, despite being exposed to fewer unique training instances. Under a matched-step protocol, MMT maintains comparable downstream quality while improving training efficiency in AR-to-Diffusion adaptation. This reveals a compelling trade-off between data uniqueness and training efficiency in block diffusion models, demonstrating that the rich training signals from diverse mask samplings can effectively compensate for reduced exposure to unique data.