Pinterest's Vision-Language Model (VLM) Serving Architecture with NVIDIA Dynamo

The Evolution to VLM Serving at Pinterest

Pinterest, a visual search and discovery platform, is increasingly relying on Vision-Language Models (VLMs) to power next-generation experiences such as Pinterest Assistant and hybrid search. Unlike traditional text-only Large Language Models (LLMs), VLMs interpret both language and visual content, introducing significant architectural and operational complexities. The article outlines how Pinterest is customizing open-source models and building a robust serving stack to meet its product and scale needs, emphasizing the strategic shift towards multimodal AI capabilities.

Key Challenges in VLM Serving

Serving VLMs presents several distinct challenges compared to text-only LLMs due to the inherent complexity of multimodal data and interactions:

• Request Payloads and Preprocessing: VLM requests can include multiple images (URLs, base64, or embeddings), requiring image fetching, preprocessing (resize, normalization), and intricate prompt construction that interweaves text and image content. This can involve thousands of images per request.
• Expensive Prefill: Visual context encoding is computationally intensive, making the "prefill" phase (processing the input prompt) dominant. This leads to larger and more irregular KV (Key-Value) caches, demanding KV-aware routing and cache offloading strategies.
• Multi-turn Workloads: Conversational VLM experiences combine long dialog history with evolving visual context. Managing visual state across turns and ensuring consistent latency under varying prompt complexities is crucial.
• KV Cache Memory Pressure: Large visual contexts and extended conversations lead to significantly larger KV states, necessitating advanced memory management techniques like KV-aware routing, offloading, and disaggregated Encoder/Prefill/Decode designs to prevent frequent evictions.
• Custom Model and Payload Support: VLM serving requires explicit model-specific support for image fields, multimodal content arrays, and projector layers, moving beyond generic text-only API assumptions.

Pinterest's VLM Serving Stack Architecture

Pinterest standardized on NVIDIA Blackwell GPUs for their superior compute throughput, memory bandwidth, and HBM capacity, essential for demanding AI workloads. The core of their Gen AI Serving Solution is NVIDIA Dynamo, an inference orchestration framework providing a flexible and performant Rust-based router. The system uses an OpenAI Chat Completions API facade, model-based Envoy routing, and a custom Model Router gateway for client access. Underneath, vLLM acts as the inference engine, with Weights and Biases for model management.

Key Dynamo Features Utilized:

• P/D Disaggregated Serving: Separates prefill and decode inference phases, allowing for optimized hardware allocation and tailored serving to specific latency requirements (Time-to-First-Token or Inter-Token Latency). This enhances scalability for high-traffic, low-latency product launches.
• KV Cache Offloading with LMCache: Employs multi-tier offloading to CPU memory and disk (NVMe) to manage high GPU memory pressure, especially in high QPS, multi-turn scenarios. LMCache provides sophisticated tiered storage with asynchronous prefetching and compression, reducing TTFT and increasing throughput.
• Multimodal Support in Dynamo Frontend/vLLM: Custom features were developed with NVIDIA to handle complex multimodal payloads, including image decoding, multimodal disaggregated serving, and KV-aware routing for visual content. This reduces recomputation for VLMs and maintains latency in multi-turn interactions.
• Custom Modality Support: Projection Embeddings: To reduce the cost of sending raw images, Pinterest uses precomputed PinCLIP embeddings. Dynamo and the inference engine run a projector layer to map these embeddings into the VLM's native visual token space, leveraging existing visual representations and significantly reducing TTFT (benchmarked at ~85x faster).