Building a Real-time Service Topology System at Netflix Scale
This article details the architectural decisions and challenges in building a real-time service dependency mapping system at Netflix. It explores the shift from batch processing to a streaming-first approach, the implementation of reactive streams with backpressure for graceful degradation, and a multi-stage distributed aggregation pipeline to resolve network intermediaries and create an accurate application-level topology.
Read original on Netflix Tech BlogThe Challenge: Real-time Service Dependency Mapping at Scale
Netflix faced the significant challenge of creating a unified, real-time view of service dependencies to aid in troubleshooting, understanding blast radius, and navigating their vast distributed architecture. Traditional batch processing approaches were insufficient as they provided stale data, hindering effective incident response. The core requirement was a system that could continuously ingest, process, and present dependency data with near real-time freshness, even under extreme load.
Streaming-First Architecture with Backpressure
A crucial architectural decision was to adopt a streaming-first approach. Instead of hourly or daily batch jobs, the system continuously ingests flow records from multi-region Kafka streams and IPC metrics via Server-Sent Events (SSE). This enables near real-time topology updates, critical for live incident response and change validation. This contrasts sharply with traditional systems that process historical data, rendering it quickly outdated.
Backpressure for Graceful Degradation
To handle millions of flow records per second without data loss during traffic spikes or slowdowns, Netflix implemented reactive streams with backpressure. This mechanism allows downstream components, when overwhelmed, to signal upstream stages to slow down. For example, if a graph database write becomes slow, it signals the preceding aggregation stage, which in turn signals the Kafka consumer to pause. This prevents memory exhaustion and data loss, allowing the system to degrade gracefully rather than crash. While adding complexity, backpressure is essential for reliable operation at Netflix's scale.
Multi-Layered and Multi-Stage Aggregation Pipeline
The system employs a multi-layer architecture, with physically separate storage for different data sources (e.g., eBPF network flows, IPC metrics) to allow independent optimization. The network layer's ingestion pipeline is a three-stage distributed aggregation process designed to overcome the challenge of network intermediaries (load balancers, NAT gateways). Raw flow logs only show individual hops, but the goal is to derive logical application-to-application dependencies.
- Stage 1: Initial Aggregation (FlowLog Ingestion Service): Consumes from Kafka, filters, batches into 5-minute windows, and creates initial aggregators. Distributes via consistent hashing to Stage 2.
- Stage 2: Network Intermediary Resolution Layer (Intermediate GraphEntity Ingestion Service): This is the core resolution step. It groups flows by intermediary, identifies (Source → Intermediary) and (Intermediary → Destination) pairs, and resolves them into direct application-level edges (Source → Destination). It then re-distributes these resolved edges to Stage 3.
- Stage 3: Final Aggregation and Enrichment (GraphEntity Ingestion Service): Performs final aggregation, enriches graph nodes with external metadata (health, ownership), converts to graph entities, and persists them to the distributed graph database with throttling.
The decision for a three-stage pipeline, rather than two, was critical for production scale. A two-stage approach led to 'hot nodes' in Stage 2 due to data concentration during intermediary resolution and simultaneous I/O for enrichment. Separating resolution (Stage 2) from enrichment and persistence (Stage 3) through graduated redistribution (distribute, resolve, distribute again, persist) effectively spreads the load and isolates compute-heavy tasks from I/O-heavy ones, preventing bottlenecks even when an intermediary experiences 100x typical traffic. Server-Sent Events (SSE) were chosen over gRPC between stages due to gRPC's performance overheads at this scale.