Zalando's In-Process Client-Side Load Balancer for High Fan-Out Traffic
Zalando engineered an in-process, client-side load balancer to optimize a high-throughput API handling millions of requests per second with high fan-out internal traffic. This architectural shift from a shared edge load balancer to a client-side solution resulted in more predictable latency, reduced infrastructure costs, and enhanced observability for their Product Read API. The solution involved reimplementing consistent hashing, implementing occupancy-based load balancing, and a careful rollout strategy.
Read original on InfoQ ArchitectureThe Challenge: Shared Load Balancer Bottlenecks
Zalando's Product Read API, serving millions of requests per second, relied on a shared cluster edge load balancer, Skipper. For internal requests involving high fan-out (up to 100 parallel calls per single batch request), the team experienced unpredictable latency spikes. This was due to the batch waiting on the slowest of 100 hops through infrastructure they didn't own, making it difficult to distinguish Skipper's latency issues from their own application's performance.
Architectural Shift: Client-Side Load Balancing
To address these issues, Zalando decided to move routing for high fan-out internal traffic in-process, within the calling service, while retaining Skipper for edge and single-GET traffic. This meant graduating the internal fan-out path to a client-side load balancer, rather than fully replacing Skipper.
- Consistent Hashing: They reimplemented Skipper's exact consistent hashing algorithm (xxHash64, 100 virtual nodes per endpoint) to ensure identical key-to-pod routing during migration, preventing cache splits. This design minimizes cache churn when endpoints are added or removed.
- Occupancy-Based Load Balancing: The client-side balancer uses occupancy-based load balancing, which considers the current workload of target pods, rather than just round-robin or least-connections.
- Cache-Aware Scaling: To prevent scale-up latency spikes, new pods warm up gradually over 30 seconds using an N-ring fade-in strategy, ensuring they only serve products they are ready for.
- Observability & Reliability: Improvements included replacing control-plane polling with watch-based Kubernetes informers for faster updates, a single jittered retry mechanism, FIFO overload shedding, and richer logging to expose and route around brief node-level freezes.
When to Build Your Own Client-Side Load Balancer
The article emphasizes that building a custom client-side load balancer is typically an extreme edge case. For most scenarios, mature proxies like Envoy or existing load balancers offer robust, maintained, and hardened solutions with consistent hashing out-of-the-box. Zalando's decision was driven by specific high fan-out, high-throughput internal traffic patterns and the inability to control external infrastructure latency.
Results and Trade-offs
The implementation led to more predictable latency, a significant reduction in infrastructure costs (Skipper fleet shrunk from 50+ to 8 pods, daily cost cut from $450 to $110), and better visibility into failure origins. While an attempt at availability-zone (AZ)-aware routing was made to cut inter-AZ costs, it resulted in fragmented caches and DynamoDB read explosions, indicating a trade-off where cost savings were outweighed by performance and complexity issues.