Language‑model agents can learn from natural language in context, allowing them to refine answers after receiving feedback—a common pattern in modern agentic workflows. Conventional metrics such as Pass@1 or Pass@K only capture one‑shot success and ignore how models exploit feedback from prior attempts. Seq@K is introduced as a natural evaluation protocol that asks whether a model can solve a problem within at most K sequential attempts, receiving feedback after each failure. Unlike long‑horizon multi‑turn benchmarks, Seq@K isolates the core capability of iterative refinement from feedback. We evaluate Seq@K on seven benchmarks using sixteen frontier models and find that it reshapes the model‑selection landscape: rankings often change, Pareto frontiers reveal more efficient configurations, and the advantage of the optimal configuration can widen or shrink compared with Pass@K. Analyzing model trajectories under Seq@K uncovers refinement dynamics such as how quickly models address feedback issues or cling to their original answers. We also study how feedback type influences iterative refinement, highlighting the need for careful design and benchmarking of feedback loops. Together, these findings show that Seq@K provides a more faithful metric for systems that operate in feedback loops and should be considered when evaluating such models.
Review