Abstract
Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios.
MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities. For reproducibility, the benchmark and code are publicly available.
Blogger's Review: The introduction of MMLongEmbed is a significant step forward for evaluating multimodal embedding models in long-context scenarios, particularly highlighting the prevalent issue of superficial feature matching in current architectures. This work not only fills a critical gap in evaluation but also lays a solid foundation for future research. It is hoped that upcoming studies will delve deeper into enhancing model capabilities in long-context understanding.