Discrete diffusion, including remasking and uniform‑state samplers, generates a sequence by writing multiple token positions per step. Each token is drawn from a per‑position conditional distribution, and the choice of which positions to write is also governed by those same distributions. For domains of general interest such as pixels, phonemes, or words, inherent dependencies exist between tokens.
We show that a step matches the training distribution only when the positions it writes are conditionally independent given the already fixed tokens. No product of per‑position distributions can represent a dependent group. Moreover, per‑position marginals do not determine dependency: two joint distributions can share identical per‑position marginals while differing in the allowed value combinations.
On the synthetic ScanAndAdd task, whose joint distribution is available in closed form, we verify that every group of two or more undetermined positions selected by a confidence ranking is dependent. The measured generated distribution reaches about $29\times$ the sampling‑noise floor in total variation distance, while per‑sample metrics remain around $1.0$.
Review