Designing for Disaster Recovery and Resilient AI Systems
This article, an excerpt from Sam Newman's "Building Resilient Distributed Systems," explores critical concepts in disaster recovery (DR) planning, including Recovery Point Objective (RPO) and Recovery Time Objective (RTO), and various DR patterns like backup-and-restore and pilot light. It also touches on unique resiliency challenges for AI-powered software, such as managing model provider reliability and handling inherent uncertainties through multi-model and multi-vendor strategies.
Read original on The Pragmatic EngineerIntroduction to Disaster Recovery (DR)
Disaster Recovery (DR) involves strategies and procedures to recover from catastrophic failures that impact large parts of a system, such as cloud region outages, data center failures, or widespread infrastructure issues. Unlike localized service instance failures, DR addresses "widespread outages" that can render typical robustness measures ineffective. The core challenge in DR is balancing the trade-offs between desired recovery targets (minimal data loss and downtime) and the associated costs and complexities.
Key DR Objectives: RPO and RTO
Defining clear objectives is crucial for any DR strategy. Two fundamental metrics guide this: Recovery Point Objective (RPO) and Recovery Time Objective (RTO).
- Recovery Point Objective (RPO): The maximum acceptable amount of data loss, typically measured as the time elapsed since the last data backup or synchronization. An RPO of one hour means up to one hour of data could be lost in a disaster.
- Recovery Time Objective (RTO): The maximum acceptable duration for a system to be unavailable after an incident. This includes all time required to recover data, provision resources, and redirect traffic to an alternative site.
Trade-offs in RPO and RTO
While a zero RPO and RTO are ideal, they are practically impossible to guarantee and prohibitively expensive to approach. System designers must engage with stakeholders to balance business needs, regulatory requirements, and the cost/complexity of achieving specific RPO/RTO targets. Different workloads within a single system may also have varying RPO/RTO requirements (e.g., critical payment processing vs. new user sign-up).
Common Disaster Recovery Patterns
Various architectural patterns can be employed to achieve different RPO and RTO targets. These patterns often build upon each other in terms of complexity and cost, offering a roadmap for incremental DR improvement. All patterns typically require cross-site data replication and traffic rerouting capabilities.
| Pattern | Description | RPO/RTO Characteristics | Key Requirements |
|---|
Backup and Restore
This is the most basic DR pattern. Data is periodically backed up to an offsite location. In a disaster, new infrastructure is provisioned at a failover site, data is restored from backups, and traffic is rerouted. Automation of infrastructure provisioning is critical for reducing RTO and minimizing errors. Reusing infrastructure-as-code (e.g., Terraform scripts) across primary and failover sites is highly beneficial.