Abstract
General-purpose embodied agents must understand multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons. Existing approaches typically specialize in visual-language reasoning, video-based world modeling, or action generation, while cascaded pipelines that first synthesize future observations and then infer actions can introduce interface bottlenecks and compound prediction errors.
We present iFLYTEK-Embodied-Omni, a unified multimodal foundation model that jointly models vision (videos and images), language, and action within a single Omni framework. Its modality-specific visual-language, video-generation, and action-generation components communicate through shared multimodal self-attention. This design establishes brain-cerebellum collaboration: the vision-language model and video generation model form a high-level brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, whereas the action generation model serves as a low-level cerebellum that directly converts planned subgoals and shared multimodal context into executable action chunks.
To develop these capabilities, we combine action-annotated and action-free embodied videos from human demonstrations and robot interactions with embodied reasoning, embodied perception, and general-purpose image-text data to construct a comprehensive dataset. We further adopt a four-stage strategy that progressively trains the VLM, VGM, and AGM before jointly fine-tuning the complete model.
Blogger's Review: The design philosophy of iFLYTEK-Embodied-Omni focuses on enhancing the efficiency of embodied intelligence through multimodal collaboration. This brain-cerebellum architecture effectively addresses bottlenecks in traditional methods and holds significant application potential, especially in task execution within complex environments. Its data construction strategy emphasizes the importance of diversity and comprehensiveness, which is commendable.