Skip to content
conftrace_
2026 EMNLP EMNLP 2026

How Much Parallelism Is "Free"? A Principle of Near-Free Parallelism for Parallel Decoding

AbstractParallel decoding improves generation efficiency by processing multiple decode positions within a single decode forward, but reported speedups conflate algorithmic token utilization with the system cost of executing multiple positions. We isolate the system side by introducing Near-Free Parallelism (NFP), the maximum number of positions executable at near-free latency. Analyzing Dense FFNs, MoE FFNs, and Attention against an idle-compute baseline, we find that NFP is shaped not by memory-bound resource slack alone, but also by implementation-induced kernel-granularity slack. Based on these mechanisms, we establish a Near-Free Parallelism principle that predicts the NFP boundary from hardware balance and implementation granularity. Validation on representative Dense and MoE models—spanning both diffusion and autoregressive decoding—shows that the principle accurately predicts practical NFP boundaries, revealing that the standard idle-compute intuition over-predicts them by 2.4–23x in over half of the evaluated deployment settings—offering a system-side budget for parallelism selection and model-system co-design.

❓ The Questioner

More from EMNLP 2026

All EMNLP 2026 papers →