Model Management and Retraining Strategies in Production Systems
This article from Airbnb Engineering explores the critical decisions involved in managing and retraining machine learning models in production, emphasizing that "retraining" encompasses three distinct actions: refitting, respecifying, and holding. It details the risks and trade-offs associated with each choice, particularly in systems where forecasts directly impact business decisions. The core system design takeaway is the need for a disciplined approach to model updates to maintain accuracy and stability amidst real-world data shifts and structural changes, moving beyond a simple scheduled "retrain."
Read original on Airbnb EngineeringManaging machine learning models in a production environment, especially when their outputs drive significant business decisions, is a complex system design challenge. Airbnb's experience with their forecasting models highlights that the common term "retrain" oversimplifies a crucial decision-making process that can significantly impact a system's reliability and accuracy.
The Three Dimensions of Model Updates
The article deconstructs "retraining" into three distinct strategies, each with its own costs, benefits, and risks for a production system:
- Refitting: Updating an existing model with new data while keeping its structure and features constant. This is the cheapest option but still carries risks, as unusual recent data can degrade performance. It's suitable for ordinary parameter drift.
- Respecifying: Changing the model's structure, features, or underlying assumptions. This is a more significant commitment, akin to deploying a new component, but it's essential for addressing fundamental shifts in the data-generating process that the current model cannot represent. It's where most actual performance improvements live.
- Holding: Deciding to take no action, even if the model is missing its targets. This requires strong conviction and is appropriate when a miss is considered noise or a temporary anomaly that the model is expected to ride out. Incorrectly holding can lead to stale parameters and prolonged inaccuracies.
The "Retrain" Fallacy
A common mistake in production ML systems is treating "retrain" as a single, automatic action, often on a fixed schedule. This default to refitting can lead to systemic issues, such as chasing noise, carrying stale assumptions (ghosts), or overcorrecting for temporary shocks.
Decision Framework for Model Triage
Airbnb developed a triage system based on understanding *what changed* in the underlying data process:
- If the data-generating process is fundamentally the same, but parameters have drifted, refit.
- If the process has structurally changed in a way the model cannot represent, respecify. A persistent, unidirectional miss is a strong indicator of structural problems.
- If the miss is within the model's normal expected range of variation (noise), hold. This is particularly relevant for off-cycle anomalies.
This disciplined approach ensures that architectural decisions around model updates are deliberate and aligned with the observed behavior of the system and the underlying data, preventing costly and counterproductive "fixes" that can destabilize forecasting accuracy over time.