In recent years, large language models (LLMs) have advanced rapidly thanks to pre‑training on massive text corpora, following empirical scaling laws. Most LLMs, however, generate text autoregressively, emitting one token at a time from left to right. This sequential generation leaves GPU parallelism under‑utilized during inference, creating a compute bottleneck, and it irrevocably commits each token, preventing correction of earlier mistakes.
Discrete diffusion models address these issues by starting from a noisy or masked sequence and iteratively denoising it, updating the whole output in parallel. The parallel generative process can reach thousands of tokens per second, opens a new axis for inference‑time compute scaling, and provides built‑in error correction and controllable generation.
Over the past two years, discrete diffusion has been scaled to billions of parameters for both biological sequence modeling and natural language, with open‑weight releases and commercial products from leading labs. This article surveys the key ideas of discrete diffusion and gives a high‑level overview of how modern diffusion LLMs are built, covering foundations, core algorithmic and architectural blocks, applications in biology and language, and implications for future LLM scaling laws.
$$C \propto N^{\alpha}$$
Review