Skip to content
conftrace_
2026 EMNLP EMNLP 2026

When i.i.d. Factorization Breaks Language: The Structural Failure of Discrete Diffusion Language Models

AbstractDiscrete diffusion language models (DDLMs) offer potential advantages for language generation, such as parallel decoding and iterative refinement, but they often underperform autoregressive (AR) models and exhibit higher training instability. Recent approaches incorporate AR training signals to stabilize training, yet this introduces strong sequential inductive biases that limit the benefits of non-autoregressive generation. In this work, we analyze the structural causes of degeneration in non-AR DDLMs. We show that standard diffusion formulations rely on an i.i.d. corruption process and factorized sampling structure that implicitly assume token exchangeability, which conflicts with the non-exchangeable dependency structure of natural language. This mismatch leads to degeneration under semantic sparsity, where contextual signals are insufficient. To guarantee stable language generation, we propose the Dependency-Coupling Principle, a structural constraint that preserves necessary contextual dependencies during diffusion generation and provides insight into the performance gap between AR and diffusion language models.

More from EMNLP 2026

All EMNLP 2026 papers →