Abstract
Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks. However, long-video scenarios remain challenging due to the tension between limited visual token budgets and the need to capture multiple key events. Existing approaches typically process long videos in two stages: selecting keyframes and performing detailed perception, which exhibit limitations: they lack a modular mechanism for adaptive capacity allocation and self-correction, resulting in unreliable modeling.
To tackle these challenges, we propose MoD-VLLM, a novel Modularized Dynamic-Granularity Video LLM framework for multi-event long video understanding, which unifies temporal grounding and semantic understanding iteratively and self-reflectively. Specifically, we introduce a Positive-Negative Video Segments Grounding module and a Modularized Dynamic-Granularity Reflection module, forming a closed loop to progressively localize the question-related video segments.
The grounding module instructs a Video LLM to distinguish relevant from irrelevant video segments based on the video question. The reflection module employs a modularized scheduler that dynamically selects fine-grained encoding for relevant positive segments to capture detailed perception and coarse-grained encoding for negative segments to maintain global context. Furthermore, we propose a dynamic-granularity reinforcement learning strategy, allowing MoD-VLLM to jointly learn optimal grounding policies and dynamic granularity visual representation. We also introduce MEventBench, a challenging Multi-Event Long Video Benchmark for complex long video reasoning. Extensive experiments on several long video understanding benchmarks and our MEventBench demonstrate that MoD-VLLM significantly outperforms state-of-the-art baselines.
Blogger's Review: The introduction of MoD-VLLM notably enhances the capability of long video understanding. Its modular design and dynamic granularity scheduling effectively address the shortcomings of traditional methods in capturing multiple events, showcasing a promising future for video analysis technology. The introduction of MEventBench also provides a solid evaluation benchmark for subsequent research.