Text-to-video generation has significantly enriched content creation, holding the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, especially when employing Transformers, which incur quadratic complexity in sequence processing, limiting practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. However, its plain design limits its direct applicability to multimodal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a multimodal Mamba framework for efficient text-to-video generation.
Specifically, a MultiModal diffusion Mamba (MM-DiM) block is designed within the framework to enable seamless integration of multimodal information and spatiotemporal modeling. We introduce a novel multimodal token re-composition design, employing a bidirectional scheme for multimodal integration through simple token arrangement, along with visual registers to enhance spatiotemporal consistency. As a result, the MM-DiM blocks in M4V reduce FLOPs by 45% compared with attention-based alternatives when generating videos at 768x1280 resolution. Additionally, several training strategies are explored to better understand training text-to-video models using only publicly available datasets. Extensive experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs.
Project page: M4V Project
Blogger's Review: The M4V framework significantly enhances the efficiency of text-to-video generation by introducing the multimodal diffusion Mamba module, especially in reducing computational costs for high-resolution video processing. Its innovative token re-composition design opens new avenues for future multimodal tasks, warranting further exploration and application.