Language‑model agents can self‑improve after failures or carry text across episodes without updating the success criterion. This work narrows the attribution problem to criterion revision: when a criterion $K_0$ accepts an outcome that violates a broader commitment $B$, which observations justify that the system has formed and persistently uses a new criterion $K_1$?
We require five non‑compensatory conditions: (1) detection of criterion failure; (2) a model‑emitted revision proposal; (3) transfer of that proposal to a new episode; (4) intervention sensitivity on the claimed carrier; and (5) preservation of the revised behavior. Only when all five hold can we claim genuine criterion revision.
CMB‑0.1 was evaluated on twelve cross‑domain cases across four execution arms: stateless inference, append‑only history, model‑generated but harness‑committed state, and evaluator‑written oracle state. Seven mechanism fixtures yielded 84 deterministic scorer trials; four local quantized artifacts produced 96 calls, resulting in 192 model‑case‑arm trials.
No model trial satisfied all five conditions. While this zero‑success does not prove a universal incapability, it reveals several failure modes: for example, Qwen2.5‑7B answered every transfer and preservation item without providing any revision state, exposing zero‑state reconstruction. Other issues included repeated invalid calls, commitments leaking target distinctions, the harness performing commits, deletions reusing stateless calls, and conflict‑induced multi‑factor changes.
From these diagnostics we derive a prospective CMB‑0.4 protocol. The protocol mandates: (i) concealed transfer, (ii) explicit WRITE/NO‑WRITE/ESCALATE actions, (iii) a separately logged policy‑selected commit, (iv) matched interventions, (v) repeated hidden‑item checks, and (vi) a frozen executable oracle. CMB‑0.4 is intended as a more discriminating test framework rather than a completed confirmatory result.
The paper contributes a full measurement chain, an empirical diagnosis of the first implementation (CMB‑0.1), and a stricter protocol to support future assessments of criterion revision.
Blogger's Review: The study offers a meticulous dissection of blind spots in LLM agents' self‑revision processes. Although the current implementation falls short of meeting all criteria, the proposed CMB‑0.4 framework charts a clear path forward and is a valuable reference for upcoming research.