Continuous‑state flow and diffusion models offer a promising route to fast, parallel language generation, yet matching the quality of autoregressive models remains challenging. We introduce Rectified Language Flow (ReLF), a language model that generates text by transporting noise into token embeddings via rectified flow. ReLF combines several complementary design choices: fixed token embeddings built with Hellinger‑SVD, self‑conditioning guidance trained with cross‑entropy, and regularized reflow for few‑step generation.
We also propose singular logits, which consist of two components: an analytically computed term that forces commitment to a valid token embedding as noise vanishes, and a bounded neural residual that learns context‑dependent token preferences.
Experiments on OpenWebText with 64 function evaluations show that ReLF achieves lower generative perplexity than autoregressive baselines across all three model sizes while maintaining matched diversity, and it outperforms prior continuous‑flow methods under the same conditions. After two reflow rounds, ReLF still surpasses autoregressive baselines with only 16 function evaluations, generating 1,024‑token sequences 15‑19× faster at batch size 1.
Review