In real-time video generation, fast decoding is as crucial as fast denoising. However, current latent video diffusion models rely on 3D convolutional decoders, which are slow and memory-intensive at high resolutions or for long videos.
We introduce FlashDecoder, a fast and memory-efficient pure-Transformer video decoder that decodes latents to pixels frame by frame. At each step, the current frame attends only to a fixed-size window of past frames through a rolling KV cache.
This fixed temporal window ensures fast decoding and memory bounds regardless of video length, enabling constant-latency streaming.
As frames are processed sequentially, temporal causality is enforced without explicit attention masks, allowing training at resolutions up to 1080p while matching the reconstruction quality of convolutional decoders.
On the Wan2.1 and Wan2.2 latent spaces, FlashDecoder matches each convolutional decoder in reconstruction quality (e.g., 41.55dB vs. 41.49dB PSNR at 1080p) while decoding 3.6x-4.7x faster with up to 11x less memory on a single H100 GPU. With architecture-aware inference optimizations, the speedup widens to 12x.
Blogger's Review: The introduction of FlashDecoder marks a significant advancement in video decoding, particularly in handling high resolutions and long videos. Its Transformer-based design not only enhances speed and memory efficiency but also opens new avenues for further research, making it a noteworthy development to follow closely.