Discrete denoising diffusion models (DDMs) have emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are shaped by the construction of the discrete state space: the tokenization scheme, vocabulary topology, and domain-specific structural alphabets.
This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.
Blogger's Review: Discrete diffusion models showcase unique advantages in handling discrete data, especially in terms of generation efficiency and flexibility. Future research could explore their application potential in specific domains, driving innovations in model design.