Evolving Netflix's Ads Event Pipeline for Live Streaming with Apache Flink

Introduction to Ads Event Pipeline Evolution

Netflix's Ads Event Pipeline is a critical system for collecting, enriching, and distributing ad telemetry. Initially, it used a centralized system called Ads Event Publisher that relied on a globally replicated Key-Value store, the Ads Metadata Registry, for enriching client ad events with detailed serving context. While effective for Video on Demand (SVOD), this design faced significant challenges when scaling for live streaming events, which exhibit massive, spiky traffic patterns.

Challenges with the Original Design for Live Scale

• Scaling Bottleneck: The original design required synchronous, must-succeed writes to the Metadata Registry on the ad-serving critical path. At live event scale (tens of millions of concurrent viewers), this translated to millions of writes per second, making the registry a severe scaling bottleneck. Pre-scaling was the only safe option, but forecasting live event peaks is inherently difficult.
• Critical Path Dependencies: The synchronous database write meant the Metadata Registry was a hard dependency on the ad-serving critical path. Any slowdown or outage in the registry could halt ad serving entirely, directly impacting revenue and user experience.
• Event Freshness at Risk: High load could lead to slowed writes or lookup timeouts, causing events to be delayed, retried, or routed to dead-letter queues. This was unacceptable for systems like frequency capping, which require strict SLA-compliant event delivery to enforce ad policies effectively.

Transitioning to Apache Flink for Stateful Stream Processing

To address these limitations, Netflix rebuilt the Ads Event Publisher around stateful stream processing using Apache Flink. The core idea was to shift the metadata lookup from a synchronous database write on the critical path to an asynchronous, real-time streaming join. This allows the ad server to simply log its responses to Kafka, and the Flink job then matches these responses with client events downstream, off the critical path.

New Architecture: Flink-based Streaming Join

The new architecture centers on an Apache Flink streaming job with several key stages: normalization, re-routing, joining, deduplication, and publishing. The join stage is the most critical, matching client events (carrying an opaque token) with ad-serving responses (containing full metadata) using a custom Flink KeyedCoProcessFunction. This function holds recent responses in memory for up to 60 minutes, ensuring high match rates while managing state efficiently. Cross-region challenges are handled by encoding the serving region into the ad token and routing client events to the correct region for joining, minimizing global data replication.

• Normalize: Standardize client events from various sources into a common format.
• Re-route: Inspect events and re-route them to the correct processing region for successful joins, especially for cross-region scenarios.
• Join: The core logic where client events are matched with ad-serving responses using a custom `KeyedCoProcessFunction` over a 60-minute window, enriching the event with full serving context.
• Deduplicate: Ensure revenue-bearing events are counted only once.
• Publish: Extract measurement beacons and route enriched events to downstream consumers (frequency capping, billing, reporting).

This re-architecture not only solved the scalability issues for live events but also reduced critical path dependencies, improved event freshness, and provided a more flexible and resilient pipeline for future advertising growth at Netflix.