In recent years, Vision Language Models (VLMs) have made significant strides in visual reasoning. However, most evaluations have relied on simple scenes (such as MS-COCO), failing to showcase complex human interactions or behaviors, and have only used a handful of non-curated human descriptions as benchmarks while lacking an in-depth understanding of model error types. To address this gap, we introduce the Complex Social Behavior (CSB) dataset, which contains 100 images depicting complex social interactions/behaviors.
We analyze the progression of scene descriptions over a decade (2017-2025) of VLMs, covering four pre-Multimodal Large Language Models (MLLMs) and five MLLMs. We evaluate the accuracy of the models and 20 human descriptions relative to a gold standard on the CSB dataset and a sample from MS-COCO. We analyze five visual-cognitive error types: object detection, recognition, hallucination, scene understanding, and spatial dependence.
Our findings reveal that the CSB dataset showed a more pronounced improvement in scene description accuracy compared to MS-COCO, with pre-MLLMs achieving much lower accuracy than the lowest-ranked human descriptions and MLLMs attaining accuracies similar to the highest-ranked human descriptions. We also found that MLLMs have nearly closed the accuracy gap between simpler MS-COCO scenes and those depicting complex behaviors (CSB).
While MLLMs have almost eliminated all error types in the tested datasets, they occasionally rely on different image regions for scene descriptions than humans do (spatial dependence error). Additionally, detection, recognition, and hallucination errors have the highest impact on scene description accuracy. Together, our findings provide a more thorough evaluation of how visual language models have advanced over the last decade.
Blogger's Review: This study introduces the CSB dataset to address the shortcomings of traditional evaluations, clearly illustrating the real performance of visual language models in handling complex scenes. The progress of MLLMs is impressive, yet attention must be paid to the impact of spatial dependence errors to further enhance the practicality and accuracy of the models.