Abstract
Multimodal industrial anomaly inspection assistants are critical for next-generation smart factories, enabling interactive vision-language querying. However, multimodal large language models remain impractical for on-site deployment due to high computational demands and privacy risks from cloud inference. Compact multimodal small language models (MSLMs) provide a deployable alternative, yet progress is hindered by a lack of comprehensive robustness analyses and challenging benchmarks reflecting real-world industrial conditions.
To address this gap, we developed RobustMAD, the first deployment-motivated benchmark designed to evaluate model robustness through diverse open-ended queries spanning object understanding, anomaly detection, unanswerable problems, and visual quality degradations. Contrary to conventional assumptions, top-performing MSLMs exhibit promising capabilities, surprisingly outperforming even the larger GPT-5 Nano. However, they still fall short of safety-critical requirements, with RobustMAD revealing critical robustness gaps posing operational risks.
In particular, three recurring failure modes emerge: (i) fragile multimodal grounding under fine-grained distinctions or degraded visual conditions, (ii) insufficiently comprehensive responses, and (iii) weak logical grounding on unanswerable or ill-posed queries, leading to hallucinated outputs. Grounded in these insights, we provide actionable guidance for the design of next-generation multimodal industrial inspection assistants that leverage their promising competence.
Code is available at GitHub.
Blogger's Review: This study showcases the potential of multimodal small language models in industrial applications, despite robustness challenges. The introduction of the RobustMAD benchmark provides a crucial direction for future model improvements, emphasizing the importance of comprehensive evaluations in real-world environments.