๐-Diffusion Language Models: Controllable Generalization of Continuous and Discrete Diffusion Dynamics
Abstract
AbstractDiffusion language models (DLMs) have emerged as a promising alternative to autoregressive language models. While discrete DLMs achieve strong performance across language generation tasks, continuous DLMs remain relatively underexplored despite their smooth latent-space dynamics. In this work, we present a unified embedding-space interpretation of continuous and discrete DLMs. Within this framework, we analyze continuous diffusion dynamics as a retention channel that preserves intermediate denoising information during sampling. Motivated by this perspective, we propose ๐-DLM, which smoothly interpolates between continuous and discrete diffusion dynamics through a controllable parameter ๐. By training with randomly sampled ๐, a single model generalizes across continuous, discrete, and intermediate diffusion dynamics, unifying diverse retention behaviors within one framework. Unlike prior methods that rely on fixed or self-confidence-guided diffusion dynamics, ๐-DLM enables controllable exploitationโexploration trade-offs without retraining. Experiments on LM1B and OWT unconditional generation benchmarks show that ๐-DLM consistently outperforms strong baselines. Moreover, varying ๐ enables semantically diverse generation without additional training. Compared with temperature sweeps, varying ๐ shows a more structured relationship between lexical diversity and sentence-embedding-based mode coverage.