AWS Architecture Blog·July 15, 2026Scaling Live Telemetry with DNS Multi-Value Answer Routing on CloudFront
This article details how Bitdrift scaled their mobile observability platform to handle 121 million concurrent gRPC connections during live sporting events. The core challenge involved a 'thundering herd' effect caused by AWS Route 53's weighted routing policy, which delivered only a single IP per DNS response, overwhelming individual Network Load Balancers (NLBs). The solution was a critical DNS configuration change to Multi-Value Answer routing, enabling even distribution of persistent connections across multiple origins, eliminating outages, and showcasing the critical role of DNS in high-scale distributed systems.
Read original on AWS Architecture BlogBitdrift, a mobile observability platform, faced a significant scaling challenge when supporting real-time telemetry for a major customer during the T20 World Cup cricket series. Their system needed to handle traffic surges from near-zero to 121 million concurrent gRPC connections within seconds. This scenario exposed a critical architectural bottleneck related to DNS resolution for persistent connections behind Amazon CloudFront and Network Load Balancers (NLBs).
The DNS Resolution Imbalance: Thundering Herd with Weighted Routing
Initially, Bitdrift used AWS Route 53's Weighted routing for their origin-facing DNS records. While seemingly reasonable for general load distribution, this policy returns only a single IP address per DNS query. At CloudFront's scale, this meant all CloudFront edge nodes would resolve to the same origin NLB for the entire duration of the DNS TTL (Time-To-Live).
Impact of Persistent Connections
This problem was significantly amplified by gRPC's persistent, long-lived connections. Unlike stateless HTTP requests where new connections might hit different origins, gRPC connections accumulate on the specific origin resolved at connection time. If many CloudFront edge nodes resolve to the same NLB, that NLB becomes a single point of congestion, leading to a 'thundering herd' effect and eventual overload, despite having multiple NLBs provisioned.
The Solution: Multi-Value Answer Routing
The fix involved switching the Route 53 routing policy from Weighted to Multi-Value Answer routing. This policy is designed to return up to 8 IP addresses per DNS query, each with built-in health checks. This change allowed CloudFront edge nodes to immediately spread connections across multiple available NLB origins from the very first DNS resolution, effectively eliminating the single-origin bottleneck.
- Before (Weighted Routing): Returned 1 IP per DNS query. All CloudFront edge nodes converged on a single NLB per TTL window, leading to 79.80% peak 5xx error rates.
- After (Multi-Value Answer Routing): Returned up to 8 IPs per DNS query. Traffic was immediately spread across multiple NLBs, resulting in zero server-side errors and handling 121 million devices and 110K+ peak requests/sec.
This case study highlights that seemingly minor DNS configurations can have a catastrophic impact on the scalability and reliability of systems handling high volumes of persistent connections, especially when integrated with CDN services like CloudFront. Understanding the nuances of DNS routing policies is crucial for robust system design.