Form3's Journey to Active-Active-Active Multi-Cloud Architecture for Payments
Form3 evolved from a single-cloud AWS setup to a triple active multi-cloud architecture driven by regulatory pressure and customer demand. This complex transition involved re-architecting their platform with cloud-agnostic technologies like CockroachDB and NATS JetStream, and custom solutions for cross-cloud networking and DNS resolution, enabling high availability and disaster recovery across distinct financial markets.
Read original on InfoQ CloudForm3, a payment processing platform, transitioned from a tightly coupled AWS single-cloud architecture to an active-active-active multi-cloud setup spanning AWS, Azure, and GCP. This move was primarily driven by UK banking regulators' concerns about concentration risk with single cloud providers and a major customer's requirement for a multi-cloud strategy. The goal was to treat each cloud provider like an availability zone, ensuring continuous operation even if an entire cloud fails.
From v1 (Single Cloud) to v2 (Multi-Cloud)
The initial v1 architecture leveraged AWS services extensively for rapid development, using Java microservices on ECS, SQS for asynchronous workflows, and RDS for PostgreSQL. This allowed Form3 to offload significant operational responsibilities to AWS. However, the tight coupling became a liability when multi-cloud became a necessity.
Key Design Goals for v2 Multi-Cloud
Active-active-active operation: Treat each cloud as an AZ for maximum fault tolerance. Cloud-agnostic technologies: Decouple from proprietary cloud services. Uniform software: Run the same application code across all clouds to avoid maintenance overhead.
v2 Architecture Components
- Kubernetes: Separate Kubernetes clusters in GCP, Azure, and AWS for isolation and fault tolerance. A single logical Kubernetes cluster across clouds was rejected due to single point of failure risks.
- Load Balancing: Client-side load balancing allows customers to direct requests to any cloud, enabling failover at the client level.
- Microservices: Switched from Java to Go for microservices due to Go's lightweight nature, smaller deployment footprint, and readability, which better suits a microservice architecture.
- Data Storage & Messaging: The "secret sauce" relies on two key vendors: NATS JetStream for high-performance messaging and CockroachDB for a globally distributed, Postgres-compatible transactional database. These technologies are critical for maintaining data consistency and message delivery across the active-active-active setup.
Challenges in Building the Multi-Cloud Architecture
Form3 faced significant engineering challenges, particularly in networking and bootstrapping distributed systems across independent Kubernetes clusters. A major hurdle was establishing cross-cloud communication for CockroachDB nodes.
Cross-Cloud DNS Resolution for CockroachDB
To allow CockroachDB pods in one cloud to discover and communicate with pods in other clouds, Form3 implemented a clever DNS solution. Each CoreDNS server in a Kubernetes cluster was configured with special rewrite rules. If a DNS query contained a cloud-specific suffix (e.g., `cockroach-0.azure.svc.cluster.local`), it would forward the request to the CoreDNS server in the target cloud via a network load balancer on a static IP. The target CoreDNS would then strip the cloud suffix and resolve the local Kubernetes service name, returning the internal pod IP. This allowed seamless cross-cloud pod-to-pod communication as if they were in the same cluster.