Scaling Agentic AI Workloads: Concurrency and Resource Management

The Shift from Request to Workload: Understanding Agentic Sessions

Traditional web services often treat operations as discrete, bounded requests. However, with agentic AI systems like the GitHub Copilot SDK, an "agent session" is a long-lived *workload*. Each session can persist for minutes, manage its own transcript, interact with a shell, modify a codebase, create subprocesses, and consume significant upstream resources. This fundamental difference means that scaling isn't just about handling more HTTP requests; it's about orchestrating a larger number of semi-autonomous computational units that operate concurrently, each with its own state and resource demands.

Key Distributed Systems Challenges with Agentic Concurrency

Introducing concurrency to agentic workloads immediately surfaces classic distributed systems problems. The article highlights several critical questions that arise:

• Isolation: How are individual agent jobs and their resources (sessions, workspaces, credentials) isolated from one another to prevent interference?
• Resource Ownership: Who owns each resource (e.g., file system, network port, upstream API capacity), and how are conflicts resolved?
• Failure Handling: What happens when a worker processing an agent session unexpectedly fails? How are resources cleaned up and jobs retried safely?
• Idempotency: Can a job run multiple times without causing unintended side effects on external systems?
• Concurrency Limits: Which resource is the primary bottleneck for safe concurrency (e.g., CPU, memory, disk I/O, upstream API rate limits)?
• State Consistency: How do operations affecting external systems (like creating pull requests) maintain consistency in the face of timeouts or retries?

Architectural Principles for Concurrent Agentic Systems

1. Isolation: Each agent session requires its own dedicated and ideally disposable sandbox, which includes a unique workspace and isolated runtime. This prevents one agent's actions (e.g., modifying files, installing dependencies) from affecting another. The initial sequential approach unintentionally masked shared state issues, such as agents writing to the same `/tmp/agent-workdir`. The solution involves dynamically allocating unique working directories per job, for example, using UUIDs.

import { randomUUID } from "node:crypto";
const safeName = `${repository}-${branch}`.replace(/[^a-zA-Z0-9_-]+/g, "-");
const workdir = `/tmp/agent-${safeName}-${randomUUID()}`;

2. Concurrency: Instead of unbounded parallelism (e.g., `Promise.all`), concurrency must be explicitly bounded by the resource most likely to be exhausted. This requires careful monitoring and telemetry to identify bottlenecks, which could be CPU, memory, disk, or external API rate limits, not just the number of available CPUs. A worker pool helps manage this.

3. Persistence: While agent sessions themselves can be ephemeral, the outcomes of a job (its identity, attempts, and effects on external systems like GitHub pull requests) must be durable. Cleanup operations, like disconnecting sessions and removing workspaces, must be robustly handled in `finally` blocks to ensure resources are released even if the agent encounters an error, preventing resource leaks and maintaining system stability.

async function runCopilotRemediation(job: Job, workdir: string) {
  const client = new CopilotClient({ workingDirectory: workdir });
  let session: CopilotSession | undefined;
  try {
    await client.start();
    session = await client.createSession({ /* ... */ });
    return await session.sendAndWait({ /* ... */ });
  } finally {
    await session?.disconnect();
    await client.stop();
    await removeWorkspace(workdir);
  }
}