As large language model (LLM) coding agents take on end‑to‑end engineering tasks, system‑level requirements such as schema design, asynchronous orchestration, configuration correctness, and retrieval‑filtering trade‑offs remain empirically under‑studied. This paper presents a single‑agent case study where the agent implements a multi‑component data system against a detailed pre‑existing specification. Storage technology, schema, entity‑resolution algorithm, and retrieval‑filtering strategy were fixed beforehand; the agent’s autonomy lay in the actual implementation, defect diagnosis and repair, and the open interaction‑design choices.
During one uninterrupted session we logged five defects, categorising them by the violated constraint and the detection method employed. We then evaluated, on the public HotpotQA benchmark, the sole retrieval trade‑off in the architecture: restricting candidates to a graph‑identified entity set before ranking versus performing an unfiltered search. Because we could not invoke an LLM for the entity‑identification stage, we substituted the benchmark’s gold evidence labels and reported standard recall instead of the benchmark’s own accuracy metrics.
Across retrieval budgets from 1 to 10, 100 questions, and a pooled corpus of 2,994 paragraphs, filtered recall reached its ceiling at a budget of 3, as candidates were already limited to the gold paragraphs. Unfiltered search, even at a budget of 10, recovered only about 69% of the required evidence, a gap that persisted at every budget and was statistically significant (two‑sided sign test p < 0.0001).
We conclude with a discussion of where the agent’s autonomy succeeded and where manual correction was necessary, noting one instance where a claimed performance fix was never re‑measured on the regression that prompted it.
Review