Scaling Self-Organizing Multi-Agent Clusters with Shared State

Traditional multi-agent systems often rely on a central supervisor to break down tasks, distribute them to subagents, and aggregate results. While this provides predictable behavior and central control, it introduces a single point of failure and a bottleneck limited by the supervisor's context window. The article proposes a decentralized alternative where agents coordinate by converging through shared state.

Shared State Coordination Pattern

The core of this pattern is that coordination logic resides within a shared environment, typically a simple store like an S3 bucket, rather than a dedicated orchestrator or message bus. Each independent agent reads a 'direction' (the goal), observes a bounded set of peer logs, makes a contribution, and appends to its own log. This log entry serves as the entire coordination message, allowing other agents to independently react and adjust their behavior. This design significantly improves resilience, as agents can join, fail, and leave without impacting the overall cluster operation.

• Agents: Independent processes that read and write to a shared store, never directly connecting to each other.
• Shared Environment: A single store containing a common 'direction' file, append-only logs for each agent, and a working area for artifacts.
• Direction: A high-level goal defined in a markdown file, allowing agents to determine their own path to convergence.

Architectural Considerations and Trade-offs

This self-organizing pattern is particularly well-suited for workloads that benefit from many quasi-independent contributions, emergent decomposition, and diversity of approaches, such as reviewing large codebases or generating design alternatives. It offers high parallelism and fault tolerance, as the failure of one agent does not halt the entire system. However, it trades off strict ordering and central verification, which are strengths of supervisor-led systems, meaning errors might propagate more widely before final verification.

The Kiro-flock reference implementation uses AWS EC2 instances for agents and Amazon S3 as the shared environment. A control plane built with Amazon API Gateway and AWS Lambda manages cluster lifecycle, while Amazon CloudWatch collects metrics and Amazon Bedrock assists with post-run analysis. Each agent runs a fresh Kiro CLI session per iteration to prevent state drift, ensuring decisions are based on the latest shared information.

Coordination Algorithms

The mechanism by which agents decide "whose work do I read?" defines the coordination algorithm, with three main types demonstrated by kiro-flock:

• Amorphous (Ring): Agents read a fixed window of neighbors. This scales well (constant per-agent work) but signals spread slowly, fostering diversity. Ideal for parallel work or as an opening phase before consensus.
• Mesh (Full Visibility): Every agent reads all other agents' latest entries. Fast alignment for small groups, but context grows linearly, limiting scalability (around 30-50 agents), and diversity collapses due to immediate consensus.
• Swarm (Recency): Agents read the K most recently active peers. The cluster reorganizes around active areas, good for ideation and runs well past 100 agents by focusing on emergent activity.

{"ts":"2026-07-21T14:12:42Z","iteration":0,"action":"wrote discussion-failure-modes.md","result":"covered 5 failure modes (premature convergence, echo chambers, hot-spot collapse, idle cascades, stale reads) with mitigations","next_intent":"read neighbours next iteration, look for topics to challenge or extend"}