Abstract
Recent advancements in large visual language models (LVLMs) necessitate robust benchmarks for complex, visually grounded reasoning. A critical limitation in many document understanding benchmarks is that visual content can often be reduced to text, allowing high performance without genuine visual grounding. To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents.
The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories. It is designed to probe a hierarchy of skills, ranging from perception to multi-step visual reasoning.
To quantify benchmark properties, a simple yet effective benchmark-level metric is proposed: the Visual Dependency Index (VDI), defined as the accuracy drop when images are replaced with question-agnostic descriptions. OmniMapBench exhibits a higher VDI than established benchmarks, quantitatively validating its focus on irreducible visual reasoning.
Comprehensive evaluations of 25 leading LVLMs are conducted on OmniMapBench, revealing a significant performance gap, with the top-performing model achieving only 75.03% accuracy. This result underscores the challenges posed by OmniMapBench to current LVLMs. This work aims to catalyze progress in visual-centric reasoning for document understanding of LVLMs. The dataset and code are publicly available at GitHub.
Blogger's Review: OmniMapBench significantly enhances the evaluation standards for visual reasoning capabilities by introducing the Visual Dependency Index. Its challenging dataset will drive deeper research into document understanding, making it a noteworthy contribution to the field.