AI-Driven Testing for Distributed Systems: A Claim-Centric Approach

Testing distributed systems reliably is a critical challenge in software architecture due to inherent complexities like concurrency, network partitions, and state management. Traditional testing often falls short, missing subtle bugs that manifest in production. This article presents an innovative approach using AI agents to automate and enhance the testing process for these complex systems.

Limitations of Traditional Distributed Testing

• Partial Network Partitions: Unpredictable network failures that affect only a subset of nodes.
• Non-deterministic Concurrency: Race conditions and interleavings that are hard to reproduce.
• Crash-Recovery: Ensuring data consistency and system availability after node failures.
• Upgrade/Rollback: Validating smooth transitions between system versions.
• Idempotency Under Replay: Handling retries without unintended side effects.
• Timing-Sensitive Ordering: Correct processing of events in a specific order despite network delays.

Claim-Driven Testing Methodology

The core of this framework is a claim-driven approach, which shifts the focus from test-driven development to verifying product promises. Each scenario is designed to falsify a specific product claim under a given fault, making tests more robust and less susceptible to being weakened over time. This approach ensures explicit coverage adequacy as a deliverable.

Model-Based Consistency Verification

For consistency-critical aspects (safety, durability, idempotency, isolation, ordering, membership), each scenario binds an abstract model (e.g., register, queue, log) to an operation-history schema and a named checker. This moves beyond mere chaos engineering by combining fault injection with formal verification through models and checkers (e.g., linearizability, serializability). Every test verdict is a 9-state classification, preventing silent passes and pinpointing blame (SUT, harness, checker, environment).

### Scenario S3: linearizable_append_under_partition
- Falsifies if it FAILs: C1 (every acknowledged append is durable and linearisable), C5 (leader election completes within 5s)
- Workload: 8 clients, 70% append / 30% read, 5min, key-skew zipf
- Faults: asymmetric partition isolating current leader at T+60s for 30s
- Oracle: linearizability via Porcupine over per-key histories

§7.M (model / history / checker discipline)
- Model under test: log
- Operation history: default 11-field schema (...)
- Checker: linearizability (Porcupine) per-key, then no-lost-ack against final state
- Nemesis + landing: asymmetric-partition (iptables drop one direction). Landing evidence = iptables drop counter goes 0 → 14,712 over the 30s window AND raft log emits "leader-lost; starting election" within 2s of injection.
- Ambiguous outcomes: timeouts → timeout_marker=true, complete_ts =null, treated as could-have-succeeded; retries are separate ops sharing input
- Reduction plan: if FAIL, bisect fault window + fix seed, then classify SUT / harness / checker / environment per references/test-case-reduction.md

The AI Agent Workflow

• Design Skill: An AI agent analyzes the codebase, extracts product claims, generates failure hypotheses, selects appropriate testing techniques from a catalog (e.g., Jepsen, deterministic simulation, chaos engineering, formal methods), and produces a structured Markdown test plan. This plan includes an architectural summary, scope, SUT model, existing test inventory, coverage matrix, and detailed scenarios.
• Execute Skill: Another AI agent reads the test plan, discovers the System Under Test (SUT)'s existing tools, probes the environment, and runs the defined scenarios. It captures fault landing evidence, performs green-but-broken audits, assigns one of nine verdict states, and classifies any failures to the SUT, test harness, checker, or environment. This results in a comprehensive findings report with an adequacy-vs-plan assessment.