Cloudflare's Meerkat: Global Consensus with QuePaxa for Control Plane State
Cloudflare introduces Meerkat, an experimental distributed consensus service designed to manage control-plane state across its global network of 330+ data centers. Unlike traditional consensus algorithms like Raft, Meerkat leverages QuePaxa, which allows all replicas to perform writes and avoids timeouts, making it robust in unpredictable wide-area network conditions. This architecture aims to provide linearizable consistency and strong fault tolerance for critical services.
Read original on Cloudflare BlogCloudflare faces significant challenges in maintaining strong consistency and high availability for its internal control-plane services across its vast global network. Traditional consensus algorithms, like Raft, struggle in these wide-area network (WAN) environments due to their reliance on a single leader and fixed timeouts, which can lead to unavailability during network partitions or leader failures. To address this, Cloudflare's Research team developed Meerkat, a new consensus service powered by the QuePaxa algorithm.
The Need for Global Strong Consistency and Fault Tolerance
Control-plane data, such as resource placement or database leadership information, is critical for Cloudflare's operations. This data demands linearizable consistency, meaning operations appear to execute instantaneously in a single, global order, simplifying reasoning for developers. Additionally, the system must exhibit strong fault tolerance, remaining available for reads and writes from any data center as long as a majority of machines are alive and can communicate, and individual machine or link failures do not impact availability. This latter point is a key differentiator from Raft-based systems.
Limitations of Raft in WAN Environments
Raft's leader-based architecture is a bottleneck in WANs. If the leader fails or becomes unreachable due to network issues, the system becomes unavailable until a new leader is elected. This election process relies on timeouts, which are difficult to tune for unpredictable network latencies, often leading to incidents and service disruptions.
Meerkat and the QuePaxa Algorithm
Meerkat's core innovation lies in its use of the QuePaxa consensus algorithm. Unlike Raft, QuePaxa allows all replicas to perform writes concurrently and progress is never halted due to a timeout. This design makes it significantly more resilient to the high-latency and unreliable links inherent in a global network. Meerkat exposes a consensus log upon which applications, such as a transactional key-value store and a leasing system, can be built. Each replica translates application requests into log events, which are then distributed to all other replicas via QuePaxa, ensuring all replicas maintain the exact same sequence of events. This consistent log enables linearizability for applications.
- Multi-leader writes: All replicas can accept write requests, improving availability and throughput in a globally distributed setup.
- Timeout-free progress: Eliminates the dependency on timeouts for leader election, crucial for unpredictable WAN latency.
- Linearizability: Guarantees that operations appear to occur in a single, atomic global order, simplifying application development.
- Fault tolerance: Tolerates `f` faults in `2f + 1` machines, remaining available as long as a majority of replicas can communicate.
Linearizability vs. Serializability
While the article focuses on linearizability for its key-value store, it also notes that Meerkat's key-value store provides serializability. Linearizability ensures that all operations appear to execute instantaneously at some point between their invocation and response. Serializability, on the other hand, ensures that concurrent transactions result in a system state that is equivalent to one where the transactions were executed serially. Both are strong consistency models, but they apply at different levels (single operations vs. transactions).