AWS Architecture Blog·July 22, 2026Multi-Region Resiliency for AWS CloudFormation Custom Resources
This article details an active-active multi-Region architecture for AWS CloudFormation custom resources, addressing inherent challenges like duplicate execution, lack of distributed locking, and automated failover. It provides a robust solution leveraging DynamoDB Global Tables for idempotency and distributed locking, and Amazon Application Recovery Controller for automated failover, ensuring high availability and consistency across regions. The solution enables critical workloads to withstand regional failures.
Read original on AWS Architecture BlogAWS CloudFormation custom resources extend Infrastructure-as-Code capabilities, allowing custom provisioning logic for resources not natively supported. While powerful, implementing these resources in a multi-Region setup for resilience introduces significant challenges. Natively, CloudFormation doesn't offer multi-Region coordination, leading to potential duplicate executions, lack of distributed locking, and no automated failover mechanisms.
Challenges of Multi-Region Custom Resources
- No native fan-out mechanism: CloudFormation stacks in different Regions independently trigger events, lacking cross-Region coordination.
- Duplicate execution risk: Deploying the same Lambda handler in multiple Regions can lead to multiple processing of the same event.
- No distributed locking: CloudFormation provides no mechanism to ensure only one handler processes an event.
- No automated failover: If a primary Region's handler fails, there's no built-in way to route the event to a secondary Region.
- Idempotency is a developer's burden: Preventing unintended duplicate operations during retries and failovers is solely the developer's responsibility.
Active-Active Multi-Region Solution Architecture
The proposed architecture delivers an active-active multi-Region solution for CloudFormation custom resource processing, built on four core principles:
- Active-Active processing: Both primary and secondary Regions are live and capable of handling events simultaneously.
- No duplicate execution: A DynamoDB Global Table-based distributed locking mechanism ensures only one Region processes any given event.
- Idempotency mechanism: Request state tracking prevents duplicate side effects during retries and failover.
- Fully automated failover: Amazon Application Recovery Controller detects failures and triggers failover without manual intervention.
Idempotency and Distributed Locks
Implementing idempotency is critical in distributed systems, especially in active-active configurations, to prevent inconsistent states from duplicate operations. Distributed locks, often backed by strongly consistent databases like DynamoDB Global Tables, are essential for coordinating operations across multiple active instances and ensuring atomicity.
Event Processing Flow
- Event Initiation: A CloudFormation Stack in any customer Region publishes a lifecycle event (Create, Update, Delete) to a local SNS topic.
- Cross-Region Fan-out: The SNS topic uses cross-Region subscriptions to simultaneously fan out the event to SQS queues in both primary (e.g., us-east-1) and secondary (e.g., us-west-2) infrastructure Regions.
- Primary Handler Processing: The primary SQS queue immediately triggers its Lambda handler. This handler attempts to acquire a lock using a conditional write on a DynamoDB Global Table. If successful, it executes the custom logic and updates the request state.
- Secondary Handler Delayed Processing: The secondary SQS queue is configured with a delay. After the delay, its Lambda handler runs, checks the DynamoDB Global Table replica for an existing lock. It skips processing if the primary has handled the request or acquires the lock and processes it if the primary has not (failover scenario).
- DynamoDB Global Tables: Act as the coordination backbone, providing strong consistency for lock state, idempotency records, and request state tracking across all Regions via bidirectional replication.