In LLM agent systems, retries often occur after external validation rejects a candidate, yet the interface between validation and the next model call remains underspecified. To address this, we introduce VeriHarness, a code-controlled agent loop where models generate candidates while external validators control acceptance, budgets, and traces.
We use VeriHarness to compare raw diagnostics with feedback that identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games, feedback containing all three fields raises terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B (+44 percentage points) and from 8/50 to 29/50 for Llama-3.1-8B (+42 percentage points).
Ablations reveal that most gains come from admissible alternatives: feedback containing only location and observed value remains near the raw diagnostic baseline. Presenting complete repair information in prose instead of a keyed JSON record yields nearly the same success, providing no evidence that JSON syntax itself improves repair. This ordering persists across the tested call budgets and one sampled-decoding setting.
Blogger's Review: This study highlights the significance of structured feedback in enhancing the repair capabilities of LLM agents. It validates the critical role of information completeness for success, providing new insights for future model optimizations. By implementing effective feedback mechanisms, LLMs can better adapt to complex environments and improve decision-making quality.