Integrating Foundation Models into Production Personalization Systems at Netflix
This article explores three distinct architectural patterns Netflix employs to integrate a large Foundation Model into its existing personalization applications: embeddings, subgraph integration, and fine-tuning. It highlights the trade-offs, complexities, and motivations behind each approach, considering factors like latency, resource constraints, and the degree of model leverage. The discussion offers valuable insights into practical challenges and solutions for deploying large AI models in real-world production systems.
Read original on Netflix Tech BlogIntegrating large-scale Foundation Models (FMs) into existing production systems presents significant architectural challenges beyond just model training. Netflix's approach to personalization leverages a central Foundation Model to unify member preference learning, distributing its intelligence to various downstream applications. This article details three primary integration patterns, each designed to balance performance, cost, and the specific needs of different application teams.
Integration Approaches
Netflix experimented with three core methods for integrating their Foundation Model, driven by varying application requirements for latency, tech stack compatibility, and the extent to which the full power of the FM needed to be utilized.
1. Embeddings from Foundation Model
This is often the initial and most straightforward integration. The FM generates user profile and item embeddings which are then stored in a specialized Embedding Store. These embeddings serve as features for downstream models, facilitating tasks like candidate generation and recommendations. A critical aspect is maintaining embedding freshness and stability. Netflix addresses this with daily fine-tuning of the FM and a custom embedding stabilization technique to map embeddings into a consistent space despite retraining and drift. A near-realtime embedding generation framework is under development to improve adaptiveness based on user session actions.
System Design Considerations for Embeddings
An efficient and resilient embedding generation pipeline is crucial, often involving batch inference jobs. The Embedding Store itself requires robust capabilities for versioning, timestamping, offline/online access, and managing potential staleness. Consider the trade-off between embedding freshness and the computational cost of frequent updates.
2. Foundation Model as a Subgraph
In this approach, the Foundation Model's decoder stack is incorporated as a subgraph within the application model's larger graph. The FM subgraph processes raw user interaction sequences and feeds its representations directly into the downstream model. This allows applications to fine-tune the FM subgraph as part of their own training, eliminating staleness and potentially leveraging more specific FM layers. However, it increases application model size, inference time, and feature generation complexity.
3. Fine-tuning Foundation Model and Use Directly
Similar to fine-tuning large language models (LLMs), application teams fine-tune the Foundation Model with product-specific data and objectives. This can involve full parameter fine-tuning, freezing layers, or adding custom output heads. Netflix provides a fine-tuning framework to streamline this process. This approach offers optimal adaptation to specific use cases and can serve as a baseline for new applications, but it leads to more models and pipelines to maintain across the organization, requiring careful latency and SLA optimization.