Designing an AI Agent Gateway for Tool Discovery and Access

The Challenge: Scaling AI Agent Tool Integration

As AI agents become more sophisticated, their ability to perform specific tasks often relies on interacting with specialized external tools or internal APIs. In an enterprise environment like Uber, the number of such tools can be vast and heterogeneous, using different communication protocols (HTTP, gRPC, TChannel) and data formats (JSON, Protobuf). The core system design challenge is to provide a standardized, scalable, and secure way for AI agents to discover, understand, and utilize these tools without requiring bespoke integrations for every new tool or agent. Uber tackled this with the Model Context Protocol (MCP) Gateway.

MCP Gateway Architecture: Control Plane and Data Plane

The Uber MCP Gateway is architected into two primary components, adhering to a common design pattern of separating management from runtime operations:

• MCP Registry (Control Plane): This acts as the centralized catalog for all MCP servers and their exposed tools. It stores metadata like tool definitions (name, description, input schema), ownership information, and configuration (e.g., enable/disable status). The registry is the source of truth for tool discovery and management, allowing for independent configuration and permission control.
• Proxy Gateway (Data Plane): This component handles the actual runtime requests from AI agents. It consumes configuration from the MCP Registry, identifies the requested tool, applies necessary controls (like permissions), and routes the request to the correct downstream service, translating formats if required. It then returns an MCP-compatible result to the agent.

Automated Tool Discovery with AutoCrawler

A crucial aspect for scalability is automating the process of identifying and cataloging tools. Uber's AutoCrawler, a distributed workflow system powered by Cadence, handles this. AutoCrawler periodically scans for new services, APIs, and schema changes through two main pathways:

1. Existing API Definitions: It parses Internal Definition Languages (IDLs) like Protobuf or OpenAPI specs to extract method names, request/response structures, and documentation. An LLM is then used to refine these into user-friendly tool descriptions, which are stored in the MCP Registry. This represents a "no-code" approach, leveraging existing service contracts.
2. Native MCP Servers: For services that directly implement the MCP protocol and expose their tool definitions, AutoCrawler tracks their heartbeats, calls their `listTools` operation, and registers their tools and schemas in the registry.

Request Flow and Protocol Translation

The Proxy Gateway maintains an in-memory state of tool configurations, refreshed periodically. When an AI agent makes an MCP request to a virtual server endpoint (e.g., `/logging/mcp`), the gateway performs the following steps:

• Identify Tool: Locates the corresponding virtual server and the specific tool requested.
• Route & Transform: Uses in-memory mappings to find the downstream service and operation. It then translates the MCP JSON arguments into the format expected by the backend service (e.g., serializing to Protobuf bytes for a gRPC service).
• Execute & Respond: Sends the transformed request to the downstream service, receives the result, and converts it back into an MCP-compatible format before returning it to the AI agent.