DMAIC-Bench is a benchmark that measures whether large language model (LLM) agents possess self‑restraint in multi‑stage analytical tasks. Such tasks require an agent to decide to continue, redirect, or stop after consulting intermediate evidence. Self‑reflection only critiques output and does not reveal if evidence altered the next action; self‑restraint demands that the agent act appropriately based on that evidence. The benchmark comprises 97 seeded tasks across 15 industrial cases in four sectors, built on Lean Six Sigma’s statistical prerequisites. Each task includes a counterfactual pair that demands opposite actions at the same workflow point, allowing distinction between justified restraint and indiscriminate stopping. Execution traces are scored by a private scorer, and final reports undergo independent expert review. Experiments covered 13 models from six providers, achieving an average routing accuracy of 92% on matched GO decisions but only 19% on matched NO‑GO decisions, counting any forbidden downstream computation as a routing error. Traces reveal agents often performing analyses that the preceding diagnostics explicitly disallow. Blind expert review estimates that reports act on invalid evidence in 47% of reached NO‑GO decisions, with a confidence interval of 31%–61%. Explicit stop permissions help some models but do not consistently restore correct decisions at both GO and NO‑GO forks. The same framework evaluates evidence‑dependent decisions in 16 life‑science tasks (RNA‑seq and GWAS). Project code and data are publicly available.
Review