We introduce FactoryBench as a benchmark for assessing time‑series models and large language models (LLMs) on machine understanding using industrial robotic telemetry. Question‑answer pairs are arranged along Pearl's ladder of causation—state, intervention, counterfactual, decision—and span five answer formats: four structured types are scored deterministically, while free‑form responses are evaluated via an LLM‑as‑judge voting protocol. A scalable Q&A generation pipeline built around structured question templates automatically creates high‑quality items from dense, multitask, multivariate sensor streams. FactoryWave is the underlying dataset, comprising dense telemetry from a UR3 cobot and a KUKA KR10 industrial arm. Leveraging roughly 15k normalized episodes from FactoryWave, AURSAD, and voraus‑AD, FactoryBench assembles over 70k Q&A items. Zero‑shot evaluation of six state‑of‑the‑art LLMs shows that none exceed 50% on structured levels and the best reaches only 18% on decision‑making, highlighting a substantial gap between current models and operational machine understanding.
Review: FactoryBench offers a systematic, causally layered evaluation framework for industrial robotics, exposing the limitations of existing LLMs in real‑world settings and guiding future research toward more robust machine understanding.