Designing Resilient AI Systems: Mapping Failure Domains and Enhancing Reliability

Understanding the Four Failure Domains in AI Requests

When interacting with external AI services, even a "free" request is a complex journey across multiple trust boundaries. This article highlights that a typical AI request involves four distinct failure domains, each with its own characteristics and potential points of failure. Engineers must recognize these domains to design robust systems and implement effective error handling strategies.

• Client: Encompasses local network issues, timeouts, and application-level errors originating from the user's system.
• Gateway: Handles authentication, authorization, rate limiting, and initial traffic shaping for the AI service.
• Queue: Manages demand spikes, ensuring requests are processed in an orderly fashion. Latency here often indicates congestion.
• Model: The actual AI inference engine, prone to context limits, stalls, or internal processing errors.

Key Architectural Constraints for Free AI Tiers

Designing systems around free or shared AI tiers introduces specific constraints that necessitate careful architectural considerations. The article outlines critical factors to account for:

• Token Budget: Free tiers often impose strict token limits. Architects must measure real spend per request and shape traffic accordingly, potentially implementing local token counting to prevent unnecessary requests.
• Shared Server Resources: Free servers are typically multi-tenant, meaning other users' traffic can directly impact latency and performance. Designs should anticipate noisy neighbor effects and implement robust retry mechanisms.
• Implicit Queues: While not directly visible, an internal queue always exists. Increased latency often indicates a long queue. Differentiating between a true timeout and a long queue is crucial for appropriate error handling.

Enhancing Resilience and Performance

To mitigate risks and improve the reliability of AI integrations, the article proposes several architectural changes and habits:

• Client-side Budget Checks: Moving token budget validation to the client reduces load on the gateway and prevents unnecessary requests, allowing for faster failure detection.
• Per-domain Circuit Breakers: Implementing circuit breakers for each failure domain prevents a problem in one area from cascading and affecting the entire system. This allows for more granular control and faster recovery.
• Defined Streaming Policy: Deciding early whether to stream responses or buffer them is vital. Streaming is beneficial for chat applications, while buffering might be better for batch jobs, impacting connection duration and queue times.
• Aggressive Caching: Caching repetitive prompts at the client level can significantly reduce token consumption, decrease latency, and alleviate stress on the shared AI service.