Recent multimodal large language models have made notable strides in spoken interaction and video understanding, yet existing online streaming benchmarks mainly assess causal perception via timestamped QA, overlooking continuous cue detection and the decision of when and how to respond.
To address this gap, we introduce DIVE, a benchmark for evaluating full‑duplex interaction with visual engagement under realistic continuous multimodal streams. The benchmark comprises six sub‑tasks grouped into three interaction scenarios: proactive intervention, continuous engagement, and interaction coordination.
Proactive intervention tests whether a model can spot emerging visual or semantic cues and intervene at the right moment; continuous engagement requires sustained participation through visual tracking and visually grounded query interaction; interaction coordination examines how models balance listening and speaking in overlapping, multi‑party dialogues.
DIVE contains 525 audio‑video samples with fine‑grained human annotations and precise temporal metadata. The evaluation protocol separates response quality, participation decisions, and timing, using metrics such as quality scores, window response rates, and reaction latency.
Experiments on several real‑time models show that the best systems achieve sub‑second average latency, but their window response rates peak at only 53.77%. Response quality varies widely across models and tasks, with adapting ongoing replies to newly arriving information remaining especially challenging.
We hope DIVE will spur progress toward more natural and capable real‑time multimodal interactive systems.
Review: DIVE offers a systematic evaluation framework for full‑duplex real‑time interaction, exposing current models' weaknesses in timing control and sustained engagement, and guiding future algorithmic improvements.