Netflix's Multimodal AI Video Search Architecture

The Challenge: Unifying Disparate AI Model Outputs at Scale

Netflix faced the significant challenge of enabling editorial teams to search billions of data points generated by multiple specialized AI models analyzing video footage. Each model (e.g., character recognition, scene classification, dialogue transcription) produced different data types (text, vector embeddings) and operated on unaligned, overlapping time intervals. The core system design problem was to merge these diverse, time-sliced outputs into a single, comprehensive, and performant searchable index, managing billions of records while ensuring sub-second query latency.

Three-Stage Decoupled Pipeline Architecture

Netflix's solution is a three-stage, decoupled pipeline designed to handle the scale and complexity of multimodal data processing. Decoupling each stage is a critical architectural decision, preventing bottlenecks and ensuring that heavy computational work does not interfere with real-time data ingestion. This separation allows each stage to focus on a single concern: data persistence, fusion, and indexing.

Stage 1: Transactional Persistence (Apache Cassandra)

Raw annotations from all AI models are ingested and stored in Apache Cassandra. This stage prioritizes data integrity and high-speed write throughput, capturing model outputs without any transformations. Decoupling ingestion from subsequent processing ensures that the system can keep up with the real-time intake of data, regardless of the number of models or data volume. Cassandra's distributed nature makes it suitable for handling the high volume of writes.

Stage 2: Offline Data Fusion (Asynchronous Processing)

This stage is the architectural heart, handling the complex computational work asynchronously, outside the real-time path. The key technique here is temporal bucketing, which normalizes all model outputs by mapping them into fixed one-second intervals. This involves three steps:

1. Bucket Mapping: Continuous detections are segmented into discrete one-second intervals.
2. Annotation Intersection: Annotations from multiple models for the same one-second bucket are fused into a single, comprehensive record.
3. Optimized Persistence: These enriched, fused records are written back to Cassandra using upsert operations, ensuring a single source of truth and gracefully handling new model additions. The one-second bucket size is a trade-off between temporal precision and manageability.

Stage 3: Indexing for Real-Time Search (Elasticsearch)

Once temporal buckets are fused and persisted, they are ingested into Elasticsearch, which serves as the query engine. Each temporal bucket is structured as a nested document, with a parent capturing asset context and child documents housing specific multimodal annotations (character data, scene embeddings, dialogue text). This hierarchical structure is crucial for enabling cross-annotation queries, allowing users to search for concepts like "Joey in the kitchen" by matching different annotations within the same time bucket.

Hybrid Search Capabilities

The system supports hybrid search, combining exact keyword matching (for proper nouns like "Joey") and vector similarity search (for semantic concepts like "kitchen"). This approach leverages the strengths of both, outperforming either method in isolation. Users can control search parameters, including: toggle between exact k-Nearest Neighbor and Approximate Nearest Neighbor algorithms, choose distance metrics (cosine similarity, Euclidean distance), and set confidence thresholds to filter low-probability matches.