Netflix's Real-Time Distributed Graph Querying with gRPC

Netflix's Real-Time Distributed Graph (RDG) is a critical system for powering real-time insights across various internal services. While previous parts of this series covered data ingestion via Apache Flink and a scalable storage layer for billions of nodes and edges, this article deep dives into the challenges and solutions for efficiently querying this complex graph structure. The core problem addressed is how to turn a dynamic, billion-edge graph into sub-100ms responses for diverse query patterns.

Query Patterns: Balancing Depth and Breadth

Graph queries on the RDG vary significantly, primarily along two axes: how *wide* they fan out at each hop and how *deep* they chain across hops. This diversity necessitates an adaptive query execution strategy.

• Shallow and Wide Queries: These involve high fan-out at a single hop, like

• Deep and Narrow Queries: These involve sequential dependencies across multiple hops, such as "For Account X, show me the Stranger Things viewing history across all profiles." This stresses execution efficiency, as network overhead can quickly accumulate with sequential requests.

Key Architectural Decisions and Trade-offs

To meet stringent latency and scale requirements for varied query types, Netflix made several foundational design choices:

• Breadth-First Traversal: Instead of a traditional depth-first approach that suffers from high latency in distributed environments due to sequential network calls, the RDG uses a breadth-first strategy. This allows for parallel fetching of all nodes at a given level, significantly reducing overall latency by minimizing sequential I/O. The trade-off is increased memory usage to hold each level's state, managed by per-edge-type limits.
• Async-First Execution: The entire execution pipeline is built around asynchronous composition using a small set of dedicated thread pools. This is crucial because latency is dominated by I/O (storage reads, enrichment calls), and threads would otherwise be idle waiting for network responses. Asynchronous I/O prevents threads from blocking, allowing them to handle thousands of concurrent requests efficiently.
• Selective Caching: Not all graph data changes at the same rate. Netflix employs a distributed cache (EVCache) with carefully tuned TTLs only for stable, frequently accessed data (e.g., account plan type, content metadata). This achieves high hit rates (70-80%) while avoiding caching highly volatile or rarely accessed data, which would either lead to stale data or inefficient cache utilization.
• Opt-in Enrichments: External data enrichments are opt-in, meaning clients explicitly request them. This avoids penalizing all queries with unnecessary fetches of metadata (e.g., title artwork) that only specific services require. The enrichment layer is also fail-open, returning graph data even if an enrichment service is unavailable.
• Eventual Consistency: The RDG prioritizes availability and performance over strong consistency, defaulting to eventual consistency. This allows reads from the nearest replica, avoiding coordination overhead. This choice aligns with the typical use cases, which focus on recent member activity rather than strict real-time transactional accuracy.

Three-Layer Architecture

The query serving layer is structured into three main components: The Graph Query Service (entry point, gRPC requests, validation), the Execution Engine (orchestrates breadth-first traversal, applies filters, composes I/O asynchronously), and the Storage Abstraction Layer (interface to KVDAL, handles streaming, manages EVCache). An Enrichment Layer fetches optional metadata from external services in parallel and degrades gracefully. This layered approach enables modularity and efficient processing of complex graph traversals, demonstrating how each design choice contributes to achieving sub-100ms response times for diverse real-time graph queries.