Scaling Netflix's Real-Time Service Dependency Map
Netflix redesigned its Service Topology, a real-time service map, to handle production scale. The new architecture features a three-stage streaming pipeline for data processing, applies backpressure to Kafka to prevent data loss, and utilizes Server-Sent Events (SSE) for efficient internal data transfer between stages. This system provides a consistent and up-to-date view of service dependencies for critical operational tasks.
Read original on InfoQ ArchitectureArchitectural Overview of Service Topology
Netflix's Service Topology is a critical system that provides a real-time map of service dependencies. It aggregates data from multiple sources like eBPF network flows, inter-process communication (IPC) metrics, and distributed traces. This unified view is essential for incident investigation, blast-radius analysis, dependency understanding, and production change management. The article focuses on the scaling challenges and solutions for the network-flow ingestion path.
Three-Stage Streaming Pipeline
To address scalability issues and hot instances caused by popular destinations, Netflix redesigned its processing into a three-stage streaming pipeline. This separation of concerns significantly improved load distribution and reduced the burden on individual instances.
- Stage 1: Initial Aggregation: Consumes multi-region Kafka streams, filters invalid records, batches data into five-minute windows, and creates initial aggregators.
- Stage 2: Intermediary Resolution: Resolves network hops (load balancers, NAT gateways, proxies) into direct application-to-application edges and redistributes results.
- Stage 3: Enrichment and Persistence: Enriches nodes with metadata (health, ownership) and persists the final graph to the graph database.
Design Lesson: Decouple Heavy Workloads
Separating CPU-intensive or I/O-heavy operations (like intermediary resolution and graph persistence) into distinct stages or services can prevent bottlenecks and allow for independent scaling and optimization of each component. This reduces the "hot spot" problem where a single node handles disproportionately more work.
Backpressure Management and Internal Transport
The pipeline utilizes Apache Pekko Streams for robust backpressure management. When graph storage cannot keep up with the write rate, backpressure propagates upstream through the stages to the Kafka consumer, pausing consumption. This design prioritizes data integrity and completeness over real-time freshness under extreme load, ensuring no data is dropped and the map remains consistent.
Netflix also switched from gRPC to Server-Sent Events (SSE) for high-volume internal transfers between pipeline stages. They found gRPC's serialization, connection-pool management, and streaming-response memory pressure to be expensive at their scale, opting for SSE's lighter-weight nature and compatibility with reactive backpressure mechanisms.