Multi-Region Disaster Recovery for Terraform Enterprise on AWS

The article presents a critical case study born from a real-world regional outage in AWS us-east-1 that left Athenahealth's single-Region Terraform Enterprise (TFE) deployment inaccessible. This highlights a fundamental principle in system design: single points of failure, especially regional ones, pose significant business continuity risks. The proposed solution shifts TFE from an Availability Zone-level resilient setup to a truly multi-Region resilient architecture using an active-passive (pilot light) model.

Multi-Region DR Architecture Overview

The architecture implements an active-passive DR strategy across two AWS Regions (us-east-1 as primary, us-west-2 as DR), aiming for a 12-14 minute Recovery Time Objective (RTO) and less than 1 minute Recovery Point Objective (RPO). Key components and design decisions include:

• DNS Failover: Amazon Route 53 DNS alias records pointing to Network Load Balancers (NLBs) in each Region, with a 60-second TTL. Route 53 health checks probe the TFE endpoint, enabling automatic failover without control plane API calls during an event.
• Compute: Amazon EC2 instances managed by Auto Scaling groups in each Region. The DR Region's Auto Scaling group is scaled to zero during normal operations to minimize cost (pilot light strategy).
• Database Replication: Amazon Aurora PostgreSQL-Compatible global databases provide sub-second replication lag and managed failover. The primary cluster has a writer and readers, while the secondary cluster maintains an inactive writer ready for promotion.
• State File Replication: Amazon S3 Cross-Region Replication for bidirectional replication of Terraform workspace state files, crucial for supporting failback without extensive data resynchronization.
• Secrets Management: AWS Secrets Manager and AWS Key Management Service (AWS KMS) for cross-Region credential and encryption key management, ensuring TFE's encryption password is replicated and available in the DR Region.

Failover Process Explained

The failover sequence is a carefully orchestrated four-step process designed for speed and consistency:

1. Activate DR Auto Scaling group (2-5 minutes): Scale up the DR TFE instances from 0 to 1, validating health checks via the TFE health check endpoint.
2. Promote Aurora PostgreSQL-Compatible global database (~1 minute): Promote the DR writer. This step is critical to perform *before* DNS failover to prevent split-brain scenarios.
3. Confirm Amazon Route 53 DNS failover (~60 seconds): Route 53's pre-configured failover routing policy detects the primary's unhealthiness and redirects traffic to the DR Region's NLB.
4. Validate TFE availability in DR (5-7 minutes): Perform end-to-end validation of TFE functionality in the DR Region.

The article emphasizes the role of AWS Fault Injection Service (FIS) in validating this complex multi-Region DR setup. By injecting real failures into the AWS environment, FIS exposes hidden dependencies and configuration issues that might otherwise remain undetected until a real disaster, providing confidence in the DR plan and demonstrating the value of chaos engineering in distributed systems.

Chaos Engineering with AWS FIS

The solution employs AWS FIS to validate the DR workflow. This includes designing three-phase experiments to expose hidden dependencies in failover automation and validate both failover and failback. This proactive testing is essential for achieving the stated RTO and RPO, preventing extended downtime, and building operational resilience.