Introduction
Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded. To address this, we introduce LVSum, a human-annotated benchmark for evaluating long-form video summarization with fine-grained temporal alignment.
Dataset
LVSum comprises 72 diverse videos spanning 13 domains with an average duration of 16 minutes, each annotated with up to 10 human-generated summaries containing temporal references.
Methodology and Evaluation
We conduct a comprehensive evaluation of leading proprietary and open-source MLLMs using newly introduced LLM-based metrics for content relevance and modality coherence, alongside standard automatic metrics.
Key Findings
Our experiments reveal three key findings:
- Transcripts contribute substantially more to summarization quality than visual frames alone;
- A significant performance gap persists between model-generated and human-written summaries;
- Current MLLMs exhibit systematic weaknesses in temporal grounding, instruction adherence, and cross-modal coherence.
We release the dataset and code.
Blogger's Review: The introduction of the LVSum benchmark provides a crucial research tool for the long video summarization field, particularly emphasizing temporal consistency. Given the current limitations of models, future research can focus on enhancing temporal and modal coherence to achieve higher quality summary generation.