LLM Integration Patterns for Search Systems: DoorDash, Instacart, and Uber Eats
This article explores how DoorDash, Instacart, and Uber Eats integrated Large Language Models (LLMs) into their search systems, showcasing three distinct architectural patterns. It highlights how existing infrastructure and specific problem constraints influenced their design choices, providing a framework for integrating AI into production systems. The core challenge across these platforms was improving user intent understanding for complex search queries.
Read original on ByteByteGoIntegrating LLMs into existing production search systems presents a fundamental architectural question: how deeply should the LLM interact with the runtime? This article analyzes three major food delivery companies—DoorDash, Instacart, and Uber Eats—each of whom answered this question differently, leading to varied system designs for enhancing search capabilities with LLMs. Their approaches provide valuable patterns for system designers considering AI integration.
The Core Problem: Understanding User Intent
Traditional keyword search often fails with complex, natural language queries due to limitations with synonyms, typos, shorthand, language mix, word-sense ambiguity, long-tail queries, and hard constraints (like dietary restrictions). LLMs offer a powerful solution by better interpreting user intent beyond a mere bag of tokens. The architectural challenge lies in effectively embedding this AI capability into a performant and scalable system.
DoorDash: LLMs for Offline Knowledge Graph Enrichment and Query Parsing
DoorDash leveraged its existing knowledge graph. Their strategy focuses on using LLMs primarily *offline* for batch attribute extraction from SKU data, enriching the graph. At runtime, LLMs parse queries into chunks, which are then linked to the enriched knowledge graph. This uses Retrieval-Augmented Generation (RAG) as a guardrail, constraining LLM outputs to existing taxonomy concepts. This approach minimizes runtime LLM reliance, keeping the core retrieval system classical.
DoorDash's RAG as a Guardrail
Instead of generating free-form labels, DoorDash's LLM is prompted to pick from a pre-retrieved list of 100 closest taxonomy concepts from their existing graph for each query segment. This ensures consistency and leverages existing system capabilities effectively.
Instacart: Layered LLM Integration for Query Understanding
Instacart faced a fragmented legacy system for query classification and rewriting. Their solution involves a layered approach for query understanding, splitting along head-versus-tail query distribution. For common (head) queries, an offline RAG-and-cache pipeline is used. For rare (tail) queries, a real-time fine-tuned Llama-3-8B model is employed. This architecture combines: * Context Engineering: RAG pulls Instacart-specific data into the LLM prompt. * Post-processing Guardrails: Semantic similarity filters prevent LLM outputs from drifting. * Fine-tuning: Llama-3-8B is fine-tuned on proprietary data for domain alignment. This design positions LLMs upstream for interpretation, while traditional ML handles downstream retrieval and ranking.
Uber Eats: Two-Tower LLM-Powered Embedding System
Uber Eats aimed for a unified retrieval system across multiple verticals, markets, and languages. They adopted a classic two-tower architecture, where a query encoder and a document encoder produce vectors in a shared embedding space. The innovation lies in using a fine-tuned Qwen LLM as the backbone for both towers. * Query Tower: Runs online, embedding incoming queries in real-time. * Document Tower: Runs offline, pre-embedding billions of documents into an HNSW vector index. Fine-tuning on proprietary query-document interactions aligns the embedding space with Uber Eats' domain, while the base LLM provides world knowledge and cross-lingual capabilities. Optimizations like Matryoshka Representation Learning ensure scalability.
- Offline vs. Online Processing: DoorDash and Uber Eats heavily leverage offline processing for LLM inference (knowledge graph enrichment, document embeddings), reserving online LLM use for lightweight tasks (query parsing, query embedding). Instacart balances with a hybrid cached/real-time approach.
- RAG as Guardrail vs. Context: DoorDash uses RAG to constrain LLM output, while Instacart uses RAG to inject domain-specific context into the prompt.
- Domain Adaptation: All three companies emphasize fine-tuning or prompt engineering with proprietary data to align general-purpose LLMs with their specific user behavior and catalog nuances.
- Legacy Integration: Existing infrastructure (e.g., DoorDash's knowledge graph, Instacart's fragmented system) significantly influenced the chosen LLM integration strategy.