Abstract
Video diffusion transformers (vDiTs) generate high-quality video but introduce extremely high compute costs due to long diffusion timesteps and self-attention computation. As diffusion timesteps are reduced, the computation cost of self-attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data.
This paper presents Kaleido, an algorithm-hardware co-design that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channel-wise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (17 dB).
To efficiently support this algorithm, we design a systolic array-like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state-of-the-art accelerators.
Blogger's Review: The introduction of Kaleido offers a fresh perspective in the video generation field, significantly enhancing computational efficiency through precise algorithm and hardware co-design. This technology not only ensures high generative quality but also achieves remarkable optimizations in energy consumption, presenting broad application prospects.