Netflix's In-House LLM Serving Architecture and Design Decisions

Netflix's approach to serving Large Language Models (LLMs) in-house deviates from common practice by running the full stack within their existing production environment rather than relying solely on hosted APIs or a separate ML silo. This strategy demanded careful architectural choices to integrate LLMs seamlessly into their member-scale ML ecosystem, which already handles diverse model types like XGBoost, TensorFlow, and PyTorch.

Unified Serving Architecture for Diverse ML Models

The foundation is a JVM-based serving system that manages end-to-end flows, including routing, A/B testing, feature fetching, inference, post-processing, and logging. This system supports both real-time and cached batch paths. For inference, small CPU models run in-process, while larger, GPU-dependent models delegate to a remote Model Scoring Service (MSS). MSS provides a unified interface for various model types, leveraging NVIDIA Triton Inference Server for GPU management (loading, batching, scheduling) and a Java control plane for deployment, versioning, health checks, and autoscaling.

Key Design Decisions and Trade-offs for LLM Serving

The platform's design was shaped by four interdependent decisions, each with significant implications for performance, maintainability, and developer experience:

• Engine Selection (vLLM): Initially built on TensorRT-LLM, Netflix re-benchmarked against a broader workload mix (embedding generation, prefill-only, autoregressive decoding) and chose vLLM due to its operational fit, including faster iteration on custom architectures, extensibility for custom decoding logic, better debuggability, and familiarity within the ML research community.
• Model Packaging (vLLM Backend): Instead of Triton's Python backend which couples I/O specs to frontend upgrades, Netflix opted for the vLLM backend. This allows models and frontends to evolve independently as I/O tensor specs are dynamically generated at deployment time from a JSON config. Challenges included managing Triton/vLLM version mismatches and providing an escape hatch via the Python backend for models requiring custom pre/post-processing.
• API Surface Design (OpenAI-Compatible HTTP Frontend): To avoid treating LLMs as
• type": "list", "ordered": false, "items": ["Engine Selection (vLLM): Initially built on TensorRT-LLM, Netflix re-benchmarked against a broader workload mix (embedding generation, prefill-only, autoregressive decoding) and chose vLLM due to its operational fit, including faster iteration on custom architectures, extensibility for custom decoding logic, better debuggability, and familiarity within the ML research community.
• "Model Packaging (vLLM Backend): Instead of Triton's Python backend which couples I/O specs to frontend upgrades, Netflix opted for the vLLM backend. This allows models and frontends to evolve independently as I/O tensor specs are dynamically generated at deployment time from a JSON config. Challenges included managing Triton/vLLM version mismatches and providing an escape hatch via the Python backend for models requiring custom pre/post-processing.
• "API Surface Design (OpenAI-Compatible HTTP Frontend): To avoid treating LLMs as \"special snowflakes\", an OpenAI-compatible API was exposed alongside gRPC. This standard interface simplifies the transition from hosted models to fine-tuned self-hosted ones, reusing client libraries and orchestration frameworks. A key operational discovery was the silent dropping of the `response_format` parameter, necessitating a patch to translate it into vLLM's guided decoding parameters.
• "Deployment Strategies (Red-Black vs. Versioned): For GPU-intensive LLMs, new deployments take longer, and I/O schema changes introduce coordination issues. Red-Black deployments (new runs alongside old, traffic shifts, atomic rollback) are preferred for stable interfaces. Versioned deployments (multiple versions serve simultaneously) are used for unavoidable breaking interface changes, decoupling model deployment from consumer updates but incurring temporary increased GPU costs. Embedding variable configurations into the model to make it version-agnostic is recommended for cost-efficiency."]}