To move AI assistants beyond simple chat interfaces and enable reliable assistance in real applications such as Gmail, Venmo, and Slack, their tool‑use capabilities must be improved. Existing tool‑use benchmarks are either dependent on human supervision, making them hard to scale, or they are overly domain‑specific, limiting generality. To address these issues we introduce TURTLE (Tool Use Related Tasks with Long‑Horizon Evolution), an Unsupervised Environment Design (UED) approach that evolves tasks using three large language model (LLM) agents: a proposer, a solver, and a feedback generator. The proposer receives rewards based on the solver’s empirical performance, encouraging the creation of tasks that are difficult yet solvable; the feedback generator summarizes the solver’s execution traces, providing exploitable information for the proposer. TURTLE yields self‑tuning test suites adaptable to various environment requirements, from supporting user simulators and dense document retrieval to multi‑app interactions. Empirical evaluation on AppWorld, $\tau$‑Bench, and AutomationBench with Gemma 4 31B‑it shows:
- The evolved tasks target the desired difficulty range, e.g., a 165% increase in the proportion of hard tasks on AppWorld.
- Task diversity rises markedly in terms of reasoning complexity and simultaneous tool interactions.
- A broader spectrum of failure modes emerges compared to the original benchmarks, exposing model weaknesses.
These results demonstrate that TURTLE can automatically generate more challenging and diverse long‑horizon tool‑use benchmarks while preserving solvability, offering a scalable solution for evaluating next‑generation AI assistants.
Review