Designing a Transformer-Based Sequence Model for Personalized Search at Airbnb

Introduction to Guest Journey Modeling

Airbnb moved from traditional hand-crafted features to a sequence modeling approach to personalize search. The core challenge was to leverage years of guest interactions (views, bookings, cancellations) to predict preferences. This required a system capable of handling vast, sparse, and noisy event data, predominantly listing views, while optimizing for booking conversion rather than just engagement.

Architectural Breakdown: Long-Term and Short-Term Sequences

To manage the computational complexity of modeling long event sequences, Airbnb ingeniously split the guest journey into two distinct sequences, each capped to optimize for relevance and tractability:

• Long-Term Sequence: Captures infrequent but highly informative events over seven years (bookings, reviews, cancellations), capped at 80 events. This provides historical depth.
• Short-Term Sequence: Focuses on recent, potentially high-volume listing views from the past 21 days, capped at 200 events. This provides immediacy to current browsing intent.

Efficiency in Training and Serving

Training a Transformer on hundreds of millions of search-label pairs and serving predictions in real-time presented significant performance challenges. Airbnb implemented several key optimizations:

• Batching of Searches: Instead of re-running the encoder for each individual search within a session, a single forward pass over the full event sequence is performed. Intermediate embeddings are then routed to their corresponding searches, leveraging the causal mask property of Transformers for a ~4x throughput improvement.
• Sequence Length Bucketing: To minimize padding waste during batch processing, sequences are grouped by length.
• Sparse Calculation: Eliminates padded searches from the ranking model, further reducing computational overhead.
• Decoupled Inference: For serving, the sequence encoder runs as a daily batch job, pre-calculating and storing guest embeddings. At query time, the ranking model retrieves these stored embeddings in real-time, combines them with the live query, and generates listing scores. This separation maintains low serving latency while incorporating deep historical context.

Integration with Ranking and Future Directions

The sequence model was integrated incrementally, first with the long-term sequence, then adding the short-term view sequence, and finally co-training with a setwise ranker. A setwise ranker considers a set of candidate listings together, reasoning about their relative differences, which allows for more nuanced personalization than pointwise rankers. This system significantly improved offline NDCG and online uncanceled bookings and views across search and promotional emails, demonstrating the generality of the learned guest representations. Future work includes near-real-time embedding updates and richer event types.