Circuit Breakers: Mitigating Cascading Failures in Distributed Systems
This article explains the critical role of circuit breakers in distributed systems, highlighting how a 'slow' dependency can be more detrimental than a 'down' one due to resource exhaustion. It details the three essential components for robust resilience: timeouts, retries with jitter, and the circuit breaker pattern itself, emphasizing their distinct roles in preventing cascading failures and ensuring system stability.
Read original on Dev.to #systemdesignThe Peril of Slow Dependencies
In distributed systems, a dependency that is slow can be far more damaging than one that is completely down. When a service is down, requests fail quickly, freeing up resources. However, a slow service holds onto critical resources like threads, database connections, and memory for extended periods. This resource exhaustion can lead to the calling service, and potentially the entire system, becoming unresponsive, even if the slow dependency eventually returns a successful response.
The Three Links of Resilience
Self-Protection Mechanism
The primary goal of these patterns is to protect *your* service from a problematic dependency, not to fix the dependency itself. They enable graceful degradation and prevent cascading failures.
Link One: Timeouts
- Connection Timeout: Limits the time spent establishing a connection.
- Request Timeout: Limits the time spent waiting for a response after the connection is established.
- Setting Timeouts: Use latency percentiles (e.g., P99.9) of the dependency's healthy operation to determine appropriate timeouts, allowing for a small, acceptable rate of false timeouts.
- Slow Call Threshold: Beyond simple timeouts, sophisticated circuit breaker implementations (like Resilience4j) include thresholds for calls that are *slow but eventually succeed*, preventing them from silently degrading system performance.
Link Two: Retries with Jitter
Retries can be a "powerful medicine" but also a "selfish" one. While useful for transient failures, naive retries against an overloaded dependency can exacerbate the problem, leading to metastable failures. Implementing exponential backoff with jitter (random delay) is crucial to prevent all clients from retrying simultaneously, which can create a thundering herd effect and prolong outages.
Link Three: The Circuit Breaker Pattern
The circuit breaker is the ultimate defense, explicitly stopping calls to a failing dependency. It operates in three main states:
- Closed: Normal operation; calls pass through, failures are counted.
- Open: Circuit is tripped; calls are immediately rejected with an exception, preventing attempts to the unhealthy service. This achieves "fail fast."
- Half-Open: After a configurable wait duration, a limited number of probe requests are allowed through to check if the dependency has recovered. If they succeed, the circuit closes; if they fail, it returns to the Open state.
stateDiagram-v2 [*] --> Closed
Closed --> Open: failure rate crosses the threshold
over a real number of calls
Open --> HalfOpen: wait duration has elapsed
AND a call actually arrives
HalfOpen --> Closed: the permitted probe calls succeed
HalfOpen --> Open: ONE probe fails, timer restarts
Closed: Closed
calls pass through, failures counted
Open: Open
call not attempted, fails in microseconds
HalfOpen: Half-Open
a few real calls allowed through as probesUnlike retries (which assume eventual success), circuit breakers assume failure and prevent further attempts, safeguarding the calling service. Modern implementations often trip based on a failure rate over a sliding window (count-based or time-based) rather than simple consecutive failure counts, providing more robust detection of degradation.