In the rapidly evolving field of language agents, i.e., LLM agents, there's an increasing deployment in production environments, underscoring the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings, typically relying on pre-packaged tool interfaces, overlooking critical steps, and assuming inputs are clean and fully specified. Consequently, they understate the challenges in real deployments, where uncertainty and noise are ubiquitous, and agents must proactively explore the environment to uncover new tools.
To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' abilities to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information.
Experiments on 15 proprietary and open-source models show that even state-of-the-art systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capabilities of current agents and the demands of real-world applications.
Blogger's Review: The introduction of AgentGym2 is a significant enhancement to current LLM agent evaluation methods, especially concerning the application needs in real environments. It emphasizes not only reasoning and planning capabilities but also the ability to autonomously explore and combine tools in uncertain environments, providing new directions and challenges for future research.