When a coding agent operates under engineer supervision or a clinical model assists a radiologist, the deployment question becomes: keep the human‑AI workflow, replace it with a human‑only process, or use the agent alone? The workflow is justified only if it outperforms both the human‑only and agent‑only baselines. Once deployed, the alternative outcomes are not observed automatically; obtaining them requires replaying the task under the alternative, and each replay consumes expert time or compute resources.
Given a fixed replay budget, the design problem is to decide which tasks should more likely receive a human‑only replay and which an agent‑only replay. Existing approaches do not target this decision directly. Agent benchmarks often omit which missing baseline to measure; variance‑based sampling ignores which of the two comparisons is closer to failure; Bayesian information methods focus on learning model parameters rather than informing the deployment decision.
We propose TEAM‑Design, a rule that assigns every task two replay probabilities—one for each baseline. The rule raises the probability when the missing baseline outcome is hard to predict from existing task information and when that comparison is harder to establish, and lowers it when replay is expensive. We prove that the rule solves the budgeted design problem optimally, and that drawing replays at random from the recorded probabilities still controls the chance of incorrectly declaring the workflow superior to both baselines.
We re‑analyze six clinical settings (none of which show a human‑AI workflow beating both alternatives) and a coding benchmark (where one does), then evaluate TEAM‑Design on synthetic designs and on a semi‑synthetic design built from a real chest X‑ray reader study. TEAM‑Design performs best when one of the two comparisons is clearly harder to settle than the other, and can be outperformed by variance‑based allocation when the two are similarly difficult.
Review: TEAM‑Design offers a principled way to allocate limited replay budgets, especially in asymmetric baseline scenarios. However, when both baselines are equally challenging, combining it with variance‑based strategies may yield better efficiency.