Architecting Multi-Model AI Inference Systems

The Shift from Monolithic AI Provider Dependencies

Traditionally, companies might have committed to a single, powerful AI model provider. However, as open-weight models improve and the cost disparity widens, a new architectural paradigm is emerging. Both Coinbase and Vercel are leading this charge by designing their AI infrastructure to be provider-agnostic, viewing models as interchangeable components within a larger inference pipeline. This strategy allows them to leverage cost-effective options while maintaining the flexibility to use state-of-the-art models when necessary.

Core Architectural Levers for Cost Optimization and Flexibility

Coinbase's success in halving its AI bill while increasing token usage highlights three key architectural components:

1. Internal LLM Gateway: This acts as a control plane, abstracting model selection from engineers. It defaults to lower-cost, self-hosted open-weight models (e.g., Z.ai's GLM 5.2) but allows for overrides to more expensive frontier models when specific tasks demand higher capability. This provides significant cost savings without sacrificing performance for critical tasks.
2. Task-Based Routing: The gateway intelligently routes prompts to different models based on the complexity and nature of the task. Simple execution tasks are handled by cheaper models, while complex planning tasks might be directed to more powerful (and costly) models. This ensures optimal resource utilization.
3. Aggressive Caching: By keeping conversation contexts tied to the same model when cached context is valid, Coinbase achieved a 12x jump in cache hit rates (from 5% to 60%). This dramatically reduces redundant calls to LLMs and associated costs.

Designing the AI Inference Control Plane

The article emphasizes that the competitive advantage is shifting from the models themselves to the infrastructure that orchestrates their use. A robust AI gateway functions as a centralized control plane, intercepting every prompt and making real-time routing decisions based on factors like cache state, task complexity, and current pricing. This requires sophisticated observability to monitor latency, uptime, token consumption, and cost across diverse providers and models to ensure routing decisions are effective.