Preventing Progressive Collapse in Distributed Systems
This presentation explores the concept of progressive collapse, originating from civil engineering, and applies it to distributed systems. It highlights how small, localized failures can cascade into widespread outages through real-world examples like the Ronan Point tower collapse, AWS us-east-1 outage, and a problematic e-commerce site. The article emphasizes the importance of resilience engineering strategies such as strengthening components, isolating failures, and reducing interconnections to build more robust and fault-tolerant software architectures.
Read original on InfoQ ArchitectureUnderstanding Progressive Collapse
Progressive collapse describes a scenario where an initial, seemingly minor failure triggers a chain reaction, leading to a much larger, catastrophic system failure. This concept, borrowed from civil engineering (like the Ronan Point tower collapse due to a gas explosion), is highly relevant to distributed digital systems where interdependencies can easily lead to cascading failures. Understanding the root causes and propagation mechanisms is crucial for designing resilient systems.
Real-World Examples of Cascading Failures
The article details two prominent software system failures:
- AWS us-east-1 Outage (October 2021): A race condition in a DynamoDB subsystem responsible for updating DNS routes led to the deletion of critical DNS entries in Route 53. This, in turn, caused network load balancers, compute, queues, and EKS to fail, impacting numerous AWS services and customer applications (e.g., Alexa, Slack, Zoom). The interdependency of AWS internal services meant a failure in one foundational component cascaded across the region.
- Used Car Website Incident: A legacy downstream service started hanging connections indefinitely, causing the upstream application's connection pool to exhaust. This led to blocked threads, CPU saturation, and a complete system collapse. The issue was exacerbated by lenient timeouts and users repeatedly refreshing the page, creating a thundering herd effect on an already struggling system.
Key Takeaway: Interdependencies are Critical
Both examples highlight that distributed systems are inherently complex due to numerous interdependencies. A failure in a low-level or seemingly minor component can have far-reaching effects if proper isolation and resilience mechanisms are not in place. Focusing solely on individual component reliability is insufficient; the interactions between components are equally vital.
Strategies to Prevent Progressive Collapse
To build more resilient systems and prevent cascading failures, architects should consider:
- Timeouts and Retries: Implement strict, well-tuned timeouts for all external calls (downstream services, databases, external APIs). Combine this with circuit breakers to prevent repeatedly hitting failing services and graceful degradation strategies.
- Bulkheads and Isolation: Design services with independent resource pools (e.g., thread pools, connection pools) to isolate failures. If one part of the system experiences an issue, it shouldn't exhaust resources needed by other, healthy parts.
- Reducing Interconnections: Minimize direct dependencies where possible. Asynchronous communication patterns (e.g., message queues) can decouple services, allowing a failed service to recover without blocking its callers.
- Redundancy and Availability Zones: Distribute services across multiple availability zones or regions to mitigate the impact of localized infrastructure failures. However, be mindful of shared dependencies across these zones, as seen in the AWS outage.
- Graceful Degradation: Design systems to operate in a degraded but functional state when certain components fail. This might involve returning cached data, displaying placeholder content, or disabling non-critical features.
Analyzing Failure Modes
It's essential to analyze potential failure modes and their propagation paths during system design. This includes thinking about resource exhaustion (CPU, memory, connections, threads), network partitions, data corruption, and dependency failures. Regular chaos engineering experiments can help uncover unforeseen weaknesses.