DoorDash's High-Availability Proxy Cache with Envoy and Valkey
DoorDash developed Entity Cache, a transparent proxy caching platform leveraging Envoy and Valkey, to significantly reduce redundant service-to-service requests and improve performance within its microservices architecture. This platform handles 1.5M RPS with 99.99999% availability by integrating caching into the service mesh layer, requiring no application code changes. It features advanced optimizations like event-driven invalidation, dual TTLs for resilience, and custom memory management to achieve high cache hit rates and reduce upstream load.
Read original on InfoQ ArchitectureDoorDash's Entity Cache is an infrastructure-level caching solution designed to optimize communication in a microservices environment. It acts as a transparent proxy, intercepting HTTP and gRPC requests using Envoy, and leveraging Valkey (a Redis fork) for cache storage. This architecture offloads caching concerns from individual services, centralizing cache management and invalidation policies within the service mesh.
Key Architectural Components and Flow
- Envoy Proxy: Acts as the transparent proxy, intercepting requests and routing them to either the cache or upstream services.
- Valkey (Redis fork): Serves as the high-performance cache store for responses.
- Kafka-based Event-Driven Invalidation: Ensures cache freshness by propagating updates and comparing timestamps, avoiding expensive distributed deletions.
- Upstream Services: The actual business logic services that the cache protects from redundant requests.
When a request arrives, Entity Cache first checks Valkey. A cache hit returns the response directly. A cache miss forwards the request to the upstream service, stores the response in Valkey according to configured policies, and then returns it to the client. This design eliminates the need for individual teams to implement separate caching logic, streamlining development and ensuring consistency.
Achieving High Availability and Resilience
Resilience Through Stale Data
A notable design decision is the use of dual TTL thresholds. During outages or high load, the system can serve slightly stale but valid cached data, preventing cascading failures. Envoy also plays a role in resilience by removing unhealthy cache instances and directly routing requests to upstream services if the cache itself experiences issues.
Performance Optimizations
- Custom Buffer Pools: Reduces memory allocation overhead, crucial for high-throughput systems.
- Lock-Free Single-Flight Mechanism: Prevents duplicate work for concurrent cache misses on the same key, mitigating thundering herd problems.
- Probabilistic Early Refresh (XFetch): Based on the XFetch algorithm, this technique refreshes frequently accessed cache entries before they expire, minimizing cache stampedes and improving perceived latency. This differs from a simple TTL by proactively refreshing data, ensuring fresh content is often available without a synchronous fetch.
These optimizations resulted in significant improvements: a 50-60% reduction in allocation rates, a five-fold increase in per-pod throughput, and up to an 80% reduction in P99 latency spikes. DoorDash reported cache hit rates over 90% and 60-95% reduction in upstream requests, demonstrating the effectiveness of their comprehensive caching strategy.
Impact during Outages
During a multi-hour upstream service outage, Entity Cache's ability to serve stale but valid data was critical. This prevented a complete system failure, highlighting the importance of resilient caching strategies in distributed systems.