Real-Time Personalization: Extending Airbnb's Recommender with Chronon's Streaming Capabilities
This article details how Airbnb leveraged new features in their open-source feature platform, Chronon (Push Mode and NRT Model Transform), to transition their guest journey sequence recommender from a daily batch process to a near-real-time system. This architectural shift significantly reduced data staleness, improving the freshness and quality of search results by reflecting user activity within seconds, rather than days. The approach highlights how an event-driven, streaming pipeline can integrate model inference to provide up-to-date personalized experiences.
Read original on Airbnb EngineeringThe Challenge of Stale Personalization
Airbnb's original personalization system for search rankings utilized a Transformer-based sequence encoder that ran as a daily batch job. This system would process the previous day's user activity (bookings, reviews, browsing) to generate embeddings, which were then stored for real-time retrieval by the ranking model. While effective for historical data, this batch-oriented architecture introduced significant data staleness, often up to two days, meaning a user's most recent interactions were not immediately reflected in search results. This highlighted a critical gap: the need for a system that could react to guest activity *as it happened* to provide more relevant, up-to-date recommendations.
Architectural Shift: From Batch to Near-Real-Time
To overcome the staleness issue, Airbnb integrated two new general-purpose capabilities into Chronon, their feature platform: Push Mode and Near-real-time (NRT) Model Transform. This allowed them to refactor their data pipeline from a reactive, daily batch job to a proactive, event-driven streaming process, minimizing the delay in updating guest embeddings.
Chronon Push Mode
Push Mode extends Chronon's streaming capabilities by allowing streaming jobs to publish lightweight notifications as soon as a new feature value is committed. Instead of downstream consumers polling for updates, they are immediately notified. In the context of the recommender, this means a guest's new activity (e.g., a page view or search) instantly triggers a notification, kicking off the embedding generation process without waiting for a scheduled batch job.
Chronon NRT Model Transform
NRT Model Transform enables model inference to run directly within the streaming pipeline. Traditionally, models either ran embedded in application code or as part of offline batch jobs. With NRT Model Transform, Chronon can call an already deployed model (like their Transformer sequence encoder) as an integral part of the streaming flow. The output (the fresh embedding) is then written back as a feature value, immediately available for serving. This allows the same production-grade model used offline for training and batch inference to be utilized in real-time inference workflows.
System Design Takeaway: Decoupling and Reusability
By building these capabilities into Chronon as general-purpose platform features, Airbnb achieved greater reusability and scalability. Rather than creating a use-case-specific pipeline, they enhanced their core feature platform, allowing other teams and models to benefit from near-real-time updates. This highlights a key system design principle: building common infrastructure components that can serve multiple applications, reducing duplication and operational overhead.
Impact and Results
The new architecture drastically improved freshness, cutting typical delays from roughly two days to well under a minute (often 10-30 seconds). This real-time reflection of user context led to a significant improvement in quality, with offline evaluations showing a +1.67% NDCG gain and online A/B tests confirming a one-third of a percent increase in uncancelled bookings. These results demonstrate that data freshness is a critical factor in the quality of personalized recommendations.