Pinterest's Evolution of Embedding Retrieval for Billions of Items
This article details how Pinterest scaled its in-house distributed search platform, Manas, to handle tens of billions of embeddings efficiently. It focuses on architectural changes to optimize cost, scalability, and flexibility, particularly through quantization, SSD-based serving for ANN queries, and the adoption of multi-embedding retrieval models for richer semantic search.
Read original on Pinterest EngineeringThe Challenge of Scaling Embedding Retrieval
Pinterest's discovery engine relies heavily on embedding-based retrieval to surface relevant content. Their existing platform, Manas, serving billions of embeddings, faced significant challenges with increasing corpus size and model complexity. Key issues included the memory-intensive nature of traditional Approximate Nearest Neighbor (ANN) algorithms like HNSW, which require entire indices in RAM, leading to linear cost growth. Additionally, the classic two-tower retrieval paradigm was too restrictive, limiting the expressiveness of similarity scoring.
Architectural Pillars of Evolution
Pinterest addressed these challenges by evolving their Manas embedding retrieval stack across three main fronts:
- Quantization: Reducing the memory footprint of embedding indices by compressing vectors into lower-bit representations. This delivers significant memory reduction and cost savings.
- SSD-based Serving: Shifting from purely in-memory serving to leveraging NVMe SSDs for ANN queries, carefully managing I/O to maintain low latency at a fraction of the memory cost.
- Multi-embedding Retrieval: Moving beyond single-vector representations per candidate to support richer scoring functions that consider multiple embeddings, enabling more expressive ranking at the retrieval stage.
Quantization Strategies and Trade-offs
To reduce memory footprint, Pinterest implemented Scalar Quantization (SQ) and Product Quantization (PQ). SQ applies uniform discretization, offering good compression with minimal recall loss (e.g., 59% HNSW index reduction with over 90% recall). PQ achieves higher compression but with a more significant recall decrease. Benchmarks and A/B testing were crucial to select the optimal quantizer per use case, ensuring negligible impact on user engagement. They also implemented Linear Scaling SQ with SIMD intrinsics to avoid a decoding step and reduce CPU usage during distance computation.
SSD-based Serving with I/O-Aware ANN
Leveraging advancements in NVMe SSDs, Pinterest explored I/O-aware ANN algorithms. They benchmarked DiskANN and SPANN, with SPANN emerging as superior, particularly when combined with PQ quantization for on-disk embedding storage while retaining full-precision centroids. Their SPANN implementation stores the centroid index in memory (using HNSW) and large posting lists on disk, ensuring disk-access efficiency and high recall. This approach significantly reduces CPU time and memory usage compared to in-memory HNSW, even with billions of embeddings.
System Design Takeaway: Hybrid Storage for Cost-Efficiency
This case highlights a critical system design pattern: for large-scale data, strategically combining fast, expensive storage (RAM for centroids/indices) with slower, cheaper storage (SSDs for full data) can achieve significant cost savings and scalability without sacrificing performance, provided I/O patterns are carefully optimized.
Multi-Embedding Retrieval for Expressiveness
To improve the expressiveness of retrieval, Pinterest is adopting "Late Interaction" models like ColBERT, which represent documents and queries as lists of vectors, enabling more nuanced scoring than the traditional two-tower model's single dot product. Integrating this required significant changes to Manas's serving stack to handle multiple query embeddings and perform multiple ANN searches simultaneously, paving the way for more model-based retrieval.