ToolVerse is a comprehensive framework designed to scale up agentic reinforcement learning (RL) environments, allowing agents to perform complex long-horizon reasoning in Tool-Integrated Reasoning (TIR) tasks.
First, ToolVerse automatically builds massive executable agent training environments from nearly 400 real-world Model Context Protocols (MCPs) containing about 4500 tools.
Second, we propose a task design strategy based on a tool dependency graph, utilizing the Dynamic Unlocking Sampling Algorithm to generate long-horizon tasks and produce the GUST (Graph Unlocking Sampling Tasks) dataset.
Finally, to alleviate the credit assignment problem in long-horizon agentic RL, we introduce a fine-grained Turn-Aware Relative Advantage algorithm.
Extensive agentic RL training using ToolVerse has been conducted, and our framework is evaluated on several agentic benchmarks. Experimental results demonstrate that our framework significantly enhances the capabilities of large language models (LLMs) in long-horizon tool use, achieving marked performance boosts and showcasing robust reasoning within dynamic environments.
Blogger's Review: The introduction of ToolVerse provides a novel solution for agents' tool use in complex environments. By constructing vast training environments and innovative task design strategies, it significantly enhances agents’ reasoning capabilities. The effectiveness of this framework suggests that the potential of reinforcement learning in real-world applications will be better harnessed and realized in the future.