Abstract
Multimodal Large Language Models (MLLMs) have demonstrated strong zero-shot capabilities across diverse inputs such as images, videos, audio, and text. However, a crucial yet underexplored application of these models lies in understanding animal-centric scenarios. As animals are integral to millions of households, benchmarking next-generation AI models on pet-focused tasks, ranging from recognizing distress signals to enabling responsive robotic companions, is essential.
We introduce K9-Bench, a novel benchmark focused on real-world domestic dog videos, specifically targeting canine action and interaction understanding via approximately 5000 question-answer pairs across 907 videos spanning 5 distinct task categories that test long-form, canine-centric multimodal reasoning in MLLMs.
To create this dataset, we propose a scalable, VLM/LLM-powered data generation pipeline that automatically mines canine-centric videos from the web and curates QA pairs requiring fine-grained, multi-hop reasoning over canine actions and temporally extended interaction sequences. We implement bias mitigation strategies designed to eliminate biases introduced by VLMs during dataset curation.
Through extensive experimentation, we find that frontier MLLMs exhibit limited zero-shot performance on canine-centric tasks: although state-of-the-art closed-source models outperform open-source counterparts, they still struggle with compositional reasoning over subtle posture and interaction cues spread over long horizons. We observe that generic chain-of-thought prompting provides only modest performance for such long-horizon reasoning.
Beyond a novel dataset for canine activity analysis, K9-Bench provides a general-purpose dataset construction pipeline that can be adapted to other low-data domains for quantitative analysis. Our project website is available at: K9-Bench.
Blogger's Review: The introduction of K9-Bench provides a significant benchmark for evaluating multimodal large language models in understanding animal behavior, particularly in pet-centric tasks. Despite current models' limited zero-shot performance, this research points towards optimization pathways, especially in complex interactions and long-horizon reasoning. Future developments should focus more on enhancing model generalization and reasoning depth.