Uber's AI Infrastructure: Building Scalable Developer Tools for Agentic AI
This article provides an inside look into Uber's internal AI infrastructure and developer tools, highlighting how their pre-existing robust tooling enabled a rapid transition to agentic AI. It covers key architectural decisions like building a model-agnostic gateway, in-house solutions for extreme scale, and how core concepts like 'cities as primitives' influenced system design across the platform. The discussion touches upon the evolution of Uber's service architecture and the changing role of engineers in an AI-augmented development environment.
Read original on The Pragmatic EngineerThe Foundation: In-house Tooling for Hypergrowth
Uber's journey to AI agents is rooted in its extensive investment in internal developer tooling, driven by the necessity to handle unprecedented scale and unique operational requirements. Early on, the lack of off-the-shelf solutions forced Uber to build many core services in-house, which later proved invaluable for integrating AI. This approach meant that when generative AI emerged, much of the underlying infrastructure for managing services, metrics, workflows, and experiments was already mature and extensible.
Key Custom-Built Systems
- M3DB: A distributed time-series database built to handle 500 million metrics per second, a scale no vendor solution could manage at the time. This laid the groundwork for their observability platform.
- Cadence: A durable workflow engine for long-running, multi-day processes (e.g., driver onboarding). It provided the reliability needed for complex business logic, now open-sourced and a foundation for Temporal.
- Morpheus: A comprehensive system combining feature flags, experimentation (including switchback tests), and support for Uber's unique two- and three-sided marketplace models.
- uOwn: A service ownership tracking system, essential for managing thousands of microservices.
Design Principle: Build vs. Buy at Scale
Uber's experience exemplifies a critical system design decision: at extreme scale or with highly unique business logic, building custom solutions can be a strategic necessity rather than an indulgence. While costly upfront, these bespoke systems often provide unparalleled control, performance, and extensibility, becoming foundational assets for future innovations like AI integration.
Architecting for AI Agents: The Model Gateway
A pivotal decision for Uber's AI infrastructure was the creation of a single, model-agnostic AI Model Gateway. This architectural pattern centralizes access to various AI models (both vendor-supplied and open-source) and provides a consistent interface, abstracting away the underlying model specifics. This gateway facilitates rapid experimentation and easy switching between different AI providers without requiring changes in downstream applications.
- AI Guard: A security middleware within the gateway that scans requests and responses, measures latencies, and flags anomalies or security concerns.
- Spend Tracking: Centralized cost attribution per user and per model, enabling granular monitoring and optimization of AI token consumption.
- Visibility: Comprehensive audit logs and session traces for debugging, compliance, and performance analysis.
class AIGateway:
def __init__(self, model_configs):
self.models = {name: self._load_model(cfg) for name, cfg in model_configs.items()}
def _load_model(self, config):
# Logic to load different LLM providers (OpenAI, Anthropic, internal models)
pass
def process_request(self, model_name, request_payload, user_context):
# 1. AI Guard: Security and policy checks
# 2. Spend Tracking: Log cost of request
# 3. Model Routing: Select appropriate model backend
# 4. AIFX (Foundation Wrapper): Apply user settings, skills, plugins
# 5. Model Inference: Call model API
# 6. Visibility: Log audit trails, traces
passEvolving Developer Experience with AI
The introduction of AI agents has significantly impacted Uber's developer experience, leading to new tools and methodologies. From enhancing code review processes with AI-generated insights and risk scores to offloading simple CI tasks to local agents, Uber is continuously adapting its infrastructure to support an AI-augmented engineering workflow. This shift emphasizes 'context engineering' – the ability for engineers to provide precise context to agents for reliable output – and highlights the importance of robust internal platforms built over years.