American Express's Cell-Based Architecture for High-Scale Payment Processing
This article explores American Express's cell-based architecture, a highly resilient approach to processing payments at scale with low latency. It details how self-sufficient cells, deterministic routing, and a simple Global Transaction Router ensure fault isolation and continuous operation despite component failures, drawing lessons from decades of experience in distributed systems.
Read original on ByteByteGoThe Challenge: Reliable Real-Time Payment Processing
Processing millions of credit card transactions instantly and reliably is a significant system design challenge. American Express operates as the crucial middle layer between acquiring banks and card issuers, demanding extremely low latency and high availability. The transition to cloud-native infrastructure, with its inherent ephemeral nature of servers, further emphasized the need for a design resilient to frequent, unpredictable failures.
Cell-Based Architecture: Fault Isolation at Scale
American Express employs a cell-based architecture, where a 'cell' is a complete, self-sufficient copy of the payment processing stack. This includes all microservices, databases, DNS, and supporting infrastructure required to process a transaction. Key properties of a cell include:
- Independent Deployment: Each cell can be deployed and operate autonomously.
- Self-Contained: Owns all its necessary components (microservices, databases).
- Single Failure Domain: Problems within a cell are localized and do not propagate.
- Maintenance & Incident Isolation: Cells can be taken out of rotation without affecting the rest of the platform.
- No Synchronous Cross-Cell Dependencies (Critical Path): Crucial for preventing cascading failures.
Cells vs. Microservices
While microservices divide a system by function, cells divide it by failure domain. One cell typically contains many microservices, providing a higher level of fault isolation.
Data Locality and Routing Strategies
Maintaining cell self-sufficiency requires effective data management. American Express categorizes data as immutable, semi-static, or dynamic. For immutable and semi-static data, a push and distribute model is used, where reference data is replicated to every cell proactively. This avoids synchronous lookups to a central system during transaction processing. For dynamic data, which changes frequently, the strategy is reversed: the transaction is moved to the data via deterministic routing based on transaction content (e.g., partner, market, payment type). This ensures consistency without sacrificing latency for critical transactions.
The Global Transaction Router: A Critical Chokepoint Made Resilient
The Global Transaction Router is a central component that routes incoming and outgoing traffic, enforcing cell boundaries. It acts as the sole communication path between cells and handles all external communication. To prevent this critical component from becoming a single point of failure, it's designed with extreme simplicity and minimal dependencies. It contains no complex business logic, parses only necessary values for routing, and operates with a near-stateless design (non-persistent storage for state). Its configuration is loaded into memory and updated asynchronously, and multiple instances run in parallel across regions, favoring an active-active deployment model over active-standby for maximum resilience.