In this paper, we introduce a new diffusion language model called Token Time Continuous Diffusion (TTCD). The model features:
-
Continuous Space Operation: TTCD operates in continuous space, deterministically mapping Gaussian noise to a final token canvas without further sampling.
-
Per-Token Time Concept: TTCD incorporates a novel notion of per-token times, allowing some tokens to transition from noise to token at a faster rate than others.
This continuous space modeling helps TTCD avoid inaccuracies associated with parallel sampling of multiple tokens, which is critical at high speedups.
The per-token time concept enhances TTCD's ability to model conditional generation, enabling more certain tokens to proceed faster and allowing differentiated influences between tokens during refinement.
TTCD outperforms discrete models at high speedups. We trained a 160M parameter TTCD model on OpenWebText and self-distilled it; we found that at high speedups, our unconditional generation quality is comparable to several existing models of similar size, while we outperform them in conditional generation.
Similar gains were also observed in Sudoku solving.
Blogger's Review: TTCD's innovative approach of integrating continuous space and per-token timing significantly enhances the efficiency and quality of language generation, showcasing a promising new direction for future language modeling research worth exploring further.