Vision‑language models (VLMs) are increasingly employed as automatic judges for multimodal and embodied tasks, yet overall accuracy or consistency alone cannot reveal whether a judge changes its verdict for the right reason. We introduce State‑Conditional Invariance (SCI), a reliability principle that requires a judge to stay stable under semantics‑preserving presentation changes while reacting to causal interventions that alter task outcomes. To instantiate SCI we build GridWM‑Judge, a simulator‑grounded diagnostic benchmark derived from deterministic MiniGrid trajectories, offering Full, evidence‑removal, and counterfactual variants together with presentation and representation probes. The benchmark evaluates five diagnostic aspects: local perception, action‑conditioned transitions, presentation stability, representation exchangeability, and Full‑CF causal discrimination. Across five tasks, GridWM‑Judge exposes heterogeneous failure modes in current VLMs, including sensitivity to evidence presentation and encoding, low activation on successful trajectories, and weak paired counterfactual discrimination. Perception‑controlled analyses further indicate that visual grounding and objective recovery are major, model‑dependent bottlenecks. These findings motivate reporting correctness, presentation stability, and causal sensitivity jointly rather than relying on a single score.\n\nReview: The paper provides a comprehensive diagnostic suite that uncovers multi‑dimensional reliability gaps in VLM judges, offering clear metrics for future model development.