Three-Tier Escalating Self-Healing Architecture with AI
This article outlines a three-tier self-healing system designed to address different types of production failures efficiently. It introduces an architecture that escalates recovery actions from simple reflexes to AI-driven diagnostics and code patching, ensuring the cost of recovery matches the severity and complexity of the incident. The core idea is to prevent over-reacting to transient issues while providing sophisticated responses to more persistent or complex problems, using strict boundaries to control AI's scope.
Read original on Dev.to #systemdesignThe article proposes a tiered approach to self-healing in distributed systems, moving beyond the common "restart and hope" strategy for service failures. The problem highlighted is that not all failures are equal; some are transient and easily resolved, while others require deeper analysis and more complex interventions. Over-restarting for minor blips wastes resources and masks underlying issues. The solution is a system that triages failures and applies recovery actions with escalating complexity and cost.
Tier 1: Reflexes (Autonomic Recovery)
This tier handles the majority of incidents (around 95%) with immediate, low-cost responses. It operates with sub-second latency and involves no external API calls or complex logic. Key components include:
- Component Supervisor: Monitors service health.
- Exponential Backoff: Retries operations with increasing delays.
- Per-Component Circuit Breaker: Prevents cascading failures by stopping requests to unhealthy components after a certain threshold, allowing them to stabilize without constant restarts. This is crucial for preventing "noisy neighbor" issues or brief upstream blips from destabilizing services.
Tier 2: The Immune System (AI-Driven Diagnostics)
When Tier 1 mechanisms fail to resolve an issue, the incident escalates to Tier 2. This tier focuses on diagnosing the type of failure using AI, but with strict constraints on its output. A "watcher" collects an evidence bundle (logs, tracebacks, on-disk state, environment context) and feeds it to a small, specialized LLM.
Constraining AI for Safety
A critical design decision in Tier 2 is to limit the LLM's role to classification only. Instead of generating free-form solutions, the LLM selects a failure category from a predefined enum (e.g., `CORRUPT_STATE`, `STALE_LOCK`, `RESOURCE_EXHAUSTION`). This approach ensures that the decision is auditable and prevents the LLM from suggesting arbitrary or unsafe actions. Regular code then maps the classified category to a specific, pre-approved repair action (e.g., `reset_state_file`, `remove_stale_lock`).
Tier 3: Surgery (AI-Assisted Code Patching)
Tier 3 is engaged for more complex issues, specifically `CODE_BUG` classifications from Tier 2. A larger LLM is used to read the failing component's source code and generate a potential patch. Crucially, this tier includes stringent safety gates:
- Scope Guard: The patch is only auto-deployed if all changed files are strictly within the failing component's module subtree. This prevents the LLM from making widespread, potentially destabilizing changes.
- Test Suite Validation: The generated patch must pass the existing test suite. If tests break, the patch is rejected, and human intervention is required.
The article emphasizes that the effectiveness of this system relies heavily on maintaining strict boundaries between tiers. Leaky boundaries can lead to silent absorption of bugs (Tier 1 misconfigured), unsafe actions (Tier 2 generating arbitrary repairs), or system instability (Tier 3 making out-of-scope changes). The true innovation lies in the careful composition and escalation logic of existing recovery mechanisms with controlled AI integration.