Netflix's Multimodal Asset Personalization System Architecture
Netflix's MAPS (Multimodal Asset Personalization at Scale) system tackles the cold-start problem for new content by integrating multimodal embeddings, specifically CLIP, into their personalization models. This allows models to "see" and "hear" assets, transferring taste signals from related content immediately and enabling personalization before sufficient interaction data accumulates. The system also consolidates multiple canvas-specific models into a single unified model, leveraging these embeddings for improved data efficiency and long-term member satisfaction.
Read original on Netflix Tech BlogOvercoming Cold-Start with Multimodal Embeddings
The core problem addressed by MAPS is the cold-start challenge for new or less-interacted-with content assets (e.g., artwork, video previews). Traditionally, personalization models relied heavily on historical interaction data. New assets, lacking this history, would default to popularity heuristics, leading to a suboptimal user experience. The architectural innovation introduces multimodal embeddings, which allow the personalization models to understand the inherent characteristics of an asset (visual themes, talent, color palettes) rather than treating it as an opaque ID. This enables immediate personalization based on a member's preferences for similar attributes observed in other assets, significantly reducing the data needed for effective personalization.
Integrating CLIP Embeddings
Netflix integrates CLIP (Contrastive Language-Image Pre-training) embeddings, a 768-dimensional vector, with the asset's learned ID embedding. This combined representation is then passed through an MLP layer. When a new asset is created, its CLIP embedding is generated instantly. This means the model immediately has a rich, content-aware representation of the asset, allowing it to apply member preferences transferred from similar-looking or similar-themed content, even if the exact asset has never been shown before. This is a crucial architectural decision that transforms how cold-start assets are handled.
asset_representation = MLP(concatenate(e_id(a), e_a))Model Consolidation and Data Efficiency
Historically, Netflix used separate personalization models for different artwork canvases (e.g., billboard, vertical-box, horizontal-panel) due to varying aspect ratios and crops leading to different asset IDs. Since ID-based models couldn't recognize relationships between these cropped versions of the same source image, signal could not flow, and each canvas faced its own cold-start problem. The use of CLIP embeddings, which are largely invariant to crop, resize, and aspect ratio, allows for model consolidation. A single unified model can now pool interaction signals across all canvases. This is particularly beneficial for low-traffic canvases, as they immediately benefit from data learned on high-traffic ones, leading to improved personalization across the entire UI.
Reward-Based Weighting for Training Data
To effectively mix training data from disparate canvases with varying impression volumes and interaction values, Netflix employs reward-based weighting. Instead of simply pooling raw counts, each training example is weighted by the long-term reward score associated with its interaction type. This ensures that canvases contribute to the model's learning proportional to the long-term value of the interactions they drive, rather than just their frequency, optimizing for long-term member satisfaction.
Robust Offline Evaluation with IPS
A critical aspect of system design for personalization models is robust evaluation. Netflix addresses the bias inherent in judging new models on logs from current production policies by using Inverse Propensity Scoring (IPS). A small slice of traffic is served by a randomized policy, ensuring known propensities for showing assets. This allows for unbiased estimates of a candidate policy's reward, ensuring that offline metrics accurately predict online A/B test outcomes. This meticulous evaluation framework is vital for confidently deploying significant architectural changes.