In modern distributed systems, autonomous agents often execute bounded software tasks under the control of an orchestrator. Orchestrators typically rely on three reliability primitives from service meshes—retry, timeout, and error‑rate circuit breaking—to handle task retries, resumptions, and budgeting.
This paper presents a failure study of a production‑grade agentic software‑delivery platform, covering 147 numbered incidents across 81 runs. Each incident records a cost metric and, in most cases, provides a mutation‑proof reproduction of the failure. The study shows that the three assumptions underlying the primitives are routinely violated in practice, and quantifies the impact:
- A loop of 54 consecutive successful tool calls escaped detection by any error‑rate breaker;
- A progress signal hard‑coded as constant caused a false breaker trip on the third repair round, reducing a run from six functional components to three;
- Twenty‑one events accumulated over six invocations of a single delegation rendered an otherwise idempotent component unwinnable;
- A misrouted failure woke five components for a fault that actually involved only two, causing three by‑standers to regress working code;
- In twelve incidents the enforcement layer blocked correct work, the costliest consuming 107 agent turns with zero accepted writes.
A cross‑cutting root cause and its dual were identified:
Identity Adequacy: In five subsystems, an identity that failed to discriminate produced confident wrong answers; two of these independently derived the same corrective rule.
Evidence Adequacy: A reliability decision must be based on evidence that can move action, is attributable to what it measures, and remains deterministic under identical conditions.
From these findings, the authors derive seven reliability primitives whose enforcement unit shifts from the message to the delegation. The primitives include: identity verification, evidence evaluation, delegation‑level retry, delegation‑level timeout, delegation‑level error‑rate monitoring, delegation‑level idempotence guarantee, and delegation‑level rollback control. The paper also sketches a controlled evaluation framework motivated by the study, though it does not provide a full implementation.
Blogger's Review: The work offers a compelling large‑scale empirical critique of service‑mesh primitives in non‑idempotent agent scenarios. By moving the enforcement boundary to the delegation level, the proposed primitives open a promising direction for more robust orchestration, and merit further experimentation in real‑world systems.