We define OmniVChat (Omni Video Chat) as a native audio‑visual dialogue task between a user and an omni model. The model receives audio and video from the user simultaneously and returns a text response, with the query embedded in the audio‑video stream, eliminating the need for separate text questions, external captions, or speech recognition. Direct audio‑visual input reduces external latency and computation while preserving perceptual cues.
Research on OmniVChat faces two constraints: scarcity of real recordings and difficulty of evaluation. A good reply often must consider the user’s surroundings, facial expressions, and nearby objects, and such responses can be expressed in many ways, making keyword‑matching unreliable for quality assessment.
Inspired by recent advances in agent systems and video generation, we present OmniVChat-Studio, a multi‑agent data engine that synthesizes single‑ and multi‑turn audio‑visual dialogues. Using these synthetic dialogues we build OmniVChat-Bench, an evaluation benchmark that tests basic dialogue abilities across five categories.
We also introduce OmniVChat-RL, a reinforcement‑learning reward design that jointly targets reply correctness, efficiency, and style. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthetic dialogues improves performance on both OmniVChat-Bench and the human‑recorded OmniVChat-Bench-Human, validating the reward design and demonstrating transfer to real‑world dialogues.
Review