Scaling Pinterest's Manas Platform: From HNSW to Quantized SPANN for Billions of Embeddings
This article details Pinterest's architectural evolution of its Manas distributed search platform to efficiently handle tens of billions of embeddings. It focuses on tackling memory and cost challenges associated with traditional vector search algorithms through the implementation of quantization techniques (Scalar and Product Quantization) and a shift to SSD-based serving using SPANN. The discussed trade-offs between memory footprint, recall, and QPS offer valuable insights into designing large-scale similarity search systems.
Read original on InfoQ ArchitectureThe Challenge of Scaling Vector Search
Pinterest's Manas platform powers critical discovery experiences by performing similarity searches on billions of embeddings. As the data corpus grew, traditional in-memory vector search algorithms like HNSW (Hierarchical Navigable Small Worlds) became prohibitively expensive due to their high memory footprint. This led to significant infrastructure challenges related to cost, hardware allocation, and flexibility, driving the need for more efficient solutions.
Quantization Techniques for Memory Reduction
To address the memory constraints, Pinterest implemented two primary quantization techniques: Scalar Quantization (SQ) and Product Quantization (PQ). These methods compress high-dimensional floating-point vectors into lower-bit representations, dramatically reducing the memory required for storing vector indices.
- Product Quantization (PQ): Compresses vectors into compact byte codes, achieving a substantial 74% reduction for HNSW indices and 93% for IVF indices, with a recall range of 70-80%.
- Scalar Quantization (SQ): Compresses vector components into lower-bit integers, reducing HNSW indices by 59% and IVF indices by 75%, while consistently maintaining over 90% recall.
Quantization Trade-offs
Quantization introduces a trade-off between memory reduction and recall accuracy. While PQ offers greater compression, SQ often maintains higher recall. The choice depends on the specific workload's tolerance for accuracy degradation versus memory savings.
Transitioning to SSD-based Serving with SPANN
To further cut RAM costs, Pinterest evaluated moving index storage to high-throughput SSDs, comparing DiskANN and SPANN. SPANN with PQ emerged as the superior solution, offering significantly better QPS and lower latency than DiskANN with only a minor recall drop. The custom SPANN architecture optimizes I/O by keeping a small, fast centroid index in memory and storing large posting lists on SSDs, ensuring efficient search without excessive RAM.
Late Interaction Models for Enhanced Relevance
Beyond vector search, Pinterest is adopting multi-vector Late Interaction models like ColBERT to improve fine-grained relevance matching for complex queries. This architectural shift requires updating the query parser to decompose multi-token queries into multiple vector embeddings and executing simultaneous Approximate Nearest Neighbor (ANN) searches across various indices. This enhancement allows for richer semantic understanding and more accurate recommendations.