This work provides systematic evidence that vision‑language models (VLMs) can exhibit inverse scaling at the subtask level: within the same model family, larger models sometimes perform worse than their smaller counterparts on specific question categories. The study evaluates 19 models from four distinct families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model for a total of 558,334 model‑question evaluations.
A per‑question pairwise analysis is conducted for all model pairs within each family, revealing 375 subtask‑model‑pair combinations that show inverse scaling. These involve 4,342 unique queries where a smaller model answers correctly while a larger model fails, and the patterns replicate across independently designed architectures. Thirteen subtask‑dataset pairs display inverse scaling across multiple families, suggesting that the phenomenon is driven by the tasks rather than by any particular model.
The most affected domains are multi‑image spatial reasoning, visual perception, and university‑level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases. Diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and imply that model selection should be task‑aware.
Review