As agentic AI systems increasingly edit, refine, and repair decision policies, evaluating these edits becomes challenging without per-state expert action labels.
We investigate this issue using a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table.
In 5,000 held-out episodes, a multi-restart LLM editor achieves RevPAR 108.47 (95% CI 107.61 - 109.34), close to the benchmark policy's 108.75 (107.81 - 109.68), with a paired gap (LLM minus benchmark) of -0.276 and a 95% CI of [-0.692, 0.146]. A cheap diagnostic projection recovers much of the revenue (107.90), so the LLM editor's distinctive gain is not just raw revenue lift; it also reduces episode composition distance from 1.153 to 0.609. This is the strongest non-benchmark repair result.
This profile is not solely explained by restart search: non-semantic proposers with up to 2,500 evaluations fall short by 8.77 - 14.57 RevPAR points. Furthermore, plausible prompt formats do not account for the results: a shuffled-diagnostic control disrupts region-error correspondence and drops to RevPAR 94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue decreases to 98.91.
These results indicate that agentic policy repair should be evaluated based on whether diagnostic feedback leads to reliable closed-loop outcomes, rather than relying on a single behavioral distance.
Blogger's Review: This paper illustrates how effective policy repair can be achieved using regional diagnostic information in the absence of detailed expert feedback. The method not only enhances revenue but also significantly narrows the gap to the benchmark, highlighting the optimization potential in environments with incomplete information. Notably, the performance of the LLM editor surpasses traditional methods, suggesting promising applications of deep learning in decision support systems.