Form3's Journey to Active-Active-Active Multi-Cloud Architecture
This article details Form3's architectural evolution from a single-cloud AWS setup to a complex active-active-active multi-cloud strategy across AWS, Azure, and GCP. It highlights key engineering challenges and solutions for cross-cloud networking, distributed data management with CockroachDB and NATS JetStream, and custom Kubernetes operators, driven by regulatory demands for cloud exit strategies in the financial sector.
Read original on InfoQ ArchitectureThe Imperative for Multi-Cloud in FinTech
Form3, a payment processing platform, transitioned to a multi-cloud architecture primarily due to regulatory pressure from the UK's banking regulator. The concern was "concentration risk" – the potential for a single cloud provider outage to severely impact the financial sector. This mandated that banks, and by extension their service providers like Form3, must be able to exit a cloud provider, pushing them towards a robust multi-cloud strategy. This case study exemplifies how external compliance can be a strong driver for fundamental architectural shifts.
V1 Architecture: Cloud-Native but Cloud-Locked
Form3's initial architecture was built entirely on AWS to leverage managed services and accelerate development. It utilized AWS ECS for microservices (Java/Docker), SQS for asynchronous workflows, and RDS for PostgreSQL databases. While this approach enabled rapid innovation and offloaded operational burden, it resulted in significant vendor lock-in, which became problematic when multi-cloud became a business imperative.
V2 Architecture: Active-Active-Active Multi-Cloud
The V2 architecture embraces an active-active-active paradigm, treating each cloud provider as an availability zone. The core design goals were:
- Active-Active-Active Deployment: If any single cloud fails, the system continues operating seamlessly.
- Cloud Agnostic Technologies: Migrating from AWS-specific services to portable, self-managed alternatives.
- Single Software Version: Running the identical application codebase across all cloud providers to simplify maintenance.
Key technology choices for V2 include Go for microservices, Kubernetes (independent clusters in each cloud), CockroachDB for distributed SQL data storage, and NATS JetStream for a high-performance, distributed message broker. These choices allowed Form3 to manage their infrastructure directly while achieving cross-cloud resilience. Customer-side load balancing routes requests to any cloud, providing client-side resilience.
The "Three Peaks Challenge": Cross-Cloud Networking and Data
One of the most significant challenges was bootstrapping a single logical CockroachDB cluster across three independent Kubernetes clusters in different clouds. The solution involved privately networking the Kubernetes clusters and implementing a custom DNS resolution strategy.
cockroach-pod-1.svc.cluster.local.azure.internalThis custom DNS scheme allows pods in one cloud to resolve and communicate with pods in another cloud. CoreDNS in each cluster is configured with special rules to forward requests with a cloud-specific suffix (e.g., `.azure.internal`) to a network load balancer pointing to the CoreDNS server in the target cloud. On the receiving end, rewrite rules strip the cloud suffix to allow the local CoreDNS to resolve the internal service name.
Architectural Insight: Embracing Infrastructure Ownership
The shift from V1 to V2 demonstrates a critical trade-off: convenience vs. control. While AWS managed services offered speed, they introduced vendor lock-in. Moving to self-managed Kubernetes, Go, CockroachDB, and NATS JetStream provided the necessary portability and control for multi-cloud, but also increased the operational overhead, requiring a strong platform engineering team. This is a common decision point in scaling architectures.