Multimodal large language models (LLMs) achieve strong results on scientific reasoning benchmarks, yet most prior work treats perception and reasoning as a single measurable process. We introduce a diagnostic suite of five tasks spanning physics and geometry that can pinpoint failures attributable to perception, reasoning, or a combination of both.
Our experiments show that even when a model can solve a problem correctly from text alone, an incorrect diagram interpretation markedly reduces overall accuracy. Accuracy generally improves when moving from raw images to human‑written captions. For several models, performance recovers substantially with corrected captions, indicating that many errors stem from perception bottlenecks rather than genuine reasoning limitations.
A deeper analysis reveals domain‑specific patterns: after a perception error, physics tasks tend to manifest as calculation mistakes, whereas geometry tasks more often exhibit conceptual misapplication. In an extended discussion, InternS1‑mini—despite heavy scientific pre‑training and apparent thinking abilities—underperforms every other model on all tasks, with reasoning traces frequently truncating before completion.
Review: This study offers a fine‑grained diagnostic framework that separates perceptual noise from true reasoning challenges, providing clear guidance for future improvements in multimodal reasoning systems.