Abstract
LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. However, operating MCP at cloud scale becomes difficult.
Challenges
On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tools is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency, potentially reducing task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity.
Solution
We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy and reduces tool selection time by $8.9\times$ and token usage by $23.8\times$, with low per-call overhead, stable under scale-out.
Deployment Lessons
Finally, we share the lessons learned from deploying the gateway system in production.
Blogger's Review: This paper presents an innovative approach to address tool access issues for LLM agents in cloud computing, demonstrating how a gateway system can effectively integrate legacy services and emerging protocols, significantly enhancing scalability and efficiency. The efficiency and stability of its technical implementation provide valuable insights for future similar applications.