Addressing Cold Start in Natural Language Search with Synthetic Data Generation (CASTLE)
This article details Airbnb's CASTLE framework, an innovative approach to solve the cold-start problem for natural language search by generating high-quality synthetic training data. It leverages de-identified booking sessions to create contrastive listing pairs, which guide an LLM to generate realistic queries and inherently correct relevance labels, ensuring effective model training before real user queries are available.
Read original on Airbnb EngineeringThe Cold Start Problem in Natural Language Search
Introducing a new natural language search system, particularly for a platform like Airbnb, faces a significant cold start problem. This means that to train effective models, a large corpus of user queries and corresponding relevance labels is required. Without existing user traffic, this data must be synthesized from scratch. Traditional methods often generate queries and labels separately, leading to quality issues or a mismatch with real user behavior.
CASTLE's Architectural Approach: Contrastive and Seed-Guided Training
Airbnb's CASTLE (Contrastive And Seed-guided Training for natural Language sEarch) framework addresses the cold-start challenge by jointly generating queries and relevance labels. The core insight is to derive implicit user preferences from historical, de-identified booking sessions. When a user books one listing after viewing several, this action reveals a positive preference for the booked listing over the negative (viewed but not booked) alternatives. These contrastive listing pairs form the structured input for query generation.
Leveraging Contrastive Listing Pairs for Relevance
System Design Principle: Implicit Feedback
CASTLE demonstrates how to extract valuable implicit feedback from user behavior (booking vs. viewing) to create high-quality training data. This is crucial when explicit labels are scarce and avoids the circularity of having an LLM generate both queries and labels independently.
The LLM is prompted to generate a query that is satisfied by the positive listing but not the negative. This ingenious method ensures that the relevance label is correct by construction, eliminating the need for post-hoc human judgment or additional LLM evaluation for label quality. The listing features, including amenities, property details, and review summaries, are used as context for the LLM.
Query Generation Mechanism
- Template-based prompts: Provide structural diversity for common query patterns.
- Few-shot prompts: Use a small set of real user queries as linguistic exemplars to ground generated queries in authentic guest language.
- Attribute-grounded prompts: Focus the query on distinguishing attributes between the positive and negative listings (e.g., 'fireplace' or 'proximity to ski resort').
- Explicit variety mechanism: Encourages varying query length and phrasing to prevent 'query collapse' where the LLM might converge on repetitive, short phrases. This ensures the synthetic data's distribution closely matches real user behavior (e.g., matching the word-count distribution's standard deviation).
Handling Edge Cases and Evolving Systems
CASTLE incorporates a Virtual Judge (VJ), an LLM evaluator, to handle new properties without booking history, ensuring coverage for the 'long tail' of search intents. For systems with some initial traffic, CASTLE-R (real-query seed variant) can be used, showcasing adaptability to different stages of a system's lifecycle.
This hybrid approach significantly improves query distributional alignment (achieving a KL divergence of 1.01, a 9.2x improvement over baselines) and label quality (91-93% human agreement), making the synthetic data highly effective for training robust natural language search models.