Abstract
Representation alignment (REPA) has been explored to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity.
To address this issue, we propose ReGen, a hierarchical multi-prompt representation generation framework that jointly estimates multiple vector fields for both representations and data within a single diffusion model.
We further introduce generalized flow matching (GFM) to improve the generalization of conditional flow matching (CFM). We validate ReGen on single-stage waveform diffusion models including neural audio codec and Wave-VAE. ReGen significantly improves waveform generation quality from highly compressed latent representations at 12.5 Hz.
We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility (WER) and speaker similarity (SIM) with a small dataset.
Moreover, operating the LDM at 6.25 Hz with rich semantic and acoustic latent representation enables efficient training and sampling, requiring only 1 day of training on 4 GPUs and fast inference with an RTF of 0.08. Audio samples are available at ReGenVoice Demo.
Blogger's Review: The introduction of ReGen marks a significant advancement in waveform generation, enhancing quality and efficiency through the integration of multi-prompt representation generation and generalized flow matching. This research provides new insights into audio processing and generation tasks, particularly in resource-constrained scenarios. Notably, ReGenVoice's impressive performance on small datasets showcases its potential for real-world applications.