Automated agent design increasingly relies on a single score to decide whether a change is an improvement, yet when that score is aggressively optimized it stops measuring its intended target. If the agent also provides a design explanation, the explanation offers a second signal: negate each claimed statement, re‑run, and see if performance drops. We ask whether such a verifier remains reliable when the agent optimizes against it. Our contribution is not a new verifier but a stress‑testing framework: we define a pressure ratio $\rho$ that quantifies how much the verifier steers the agent’s search, and we introduce four falsifiable probes covering four failure modes—no signal, too weak to matter, being gamed by reducing claim count, and being queried before the agent commits. We apply this test to our own Rationale Consistency Score (RCS) verifier embedded in an LLM agent that designs machine‑learning pipelines, using a single generator model, six tasks, and three random seeds. First, in the prior configuration the verifier barely affected the search objective ($\rho \ll 1$), partly due to an accuracy gate, so its robustness had not been fully exercised; nevertheless it still shaped the reported artifact via a final‑selection tie‑break that $\rho$ does not capture. Second, a capability‑control experiment shows that when instructed to shorten its claim list the agent does reduce the number of claims, yet the search itself does not change even when $\rho$ is raised to over a hundred times the previous level; because feedback omits claim volume and horizons are short, this negative result reflects structural constraints rather than evidence of robustness. Third, when the agent can probe the verifier before committing, it retains claims that pass while held‑out faithfulness declines—a drop invisible to in‑loop metrics and partly confounded by RCS’s low scores on small pipelines. We present these findings as preliminary evidence and propose a reporting standard: audit a verifier with $\rho$ and the above probes before trusting it.
Review