Implementing the Bulkhead Pattern for System Isolation in Node.js

The Challenge: Cascading Failures from Slow Dependencies

In distributed systems, a single slow external dependency can inadvertently bring down unrelated, critical services. The article illustrates this with a scenario where a slow LLM provider consumes all available concurrency slots or connections, preventing even a simple login request from being processed. This isn't due to the Node.js event loop blocking, but rather a depletion of finite resources shared across different service paths, such as per-instance concurrency limits (e.g., Cloud Run), reverse proxy connection caps, or shared database connection pools.

The Solution: The Bulkhead Pattern

The bulkhead pattern is an architectural design principle derived from shipbuilding, where compartments prevent a breach from flooding the entire vessel. In software, it involves isolating elements of an application into separate pools of resources. This ensures that if one component or external dependency fails or degrades, it only impacts its designated pool, leaving other services operational.

• Resource Isolation: Each critical dependency or service path is assigned its own dedicated set of resources (e.g., threads, connection pools, concurrent request limits).
• Failure Containment: A slowdown or failure in one isolated component cannot consume resources allocated to other components.
• Graceful Degradation: When a bulkhead is full, new requests for that specific dependency can be immediately rejected or queued for a short duration, preventing prolonged waits and allowing the system to shed load intelligently.

Implementing a Simple Bulkhead in TypeScript

The article provides a concise TypeScript implementation of a semaphore-based bulkhead. This implementation manages concurrent calls and waiting requests for a given operation. It uses three key parameters for each dependent service:

• `limit`: The maximum number of concurrent calls allowed to the dependency.
• `maxQueue`: The maximum number of calls allowed to wait if the limit is reached.
• `maxWaitMs`: The maximum duration a queued call will wait before being rejected.

export class Bulkhead {
  private active = 0;
  private queue: Waiter[] = [];
  private opts: Options;

  constructor(opts: Options) {
    this.opts = opts;
  }

  async run<T>(fn: () => Promise<T>): Promise<T> {
    await this.acquire();
    try {
      return await fn();
    } finally {
      this.release();
    }
  }

  private acquire(): Promise<void> {
    const { name, limit, maxQueue, maxWaitMs } = this.opts;

    if (this.active < limit) {
      this.active++;
      return Promise.resolve();
    }

    if (this.queue.length >= maxQueue) {
      return Promise.reject(new BulkheadFullError(name));
    }

    return new Promise((resolve, reject) => {
      const w: Waiter = { resolve };
      if (Number.isFinite(maxWaitMs)) {
        w.timer = setTimeout(() => {
          // ... rejection logic
        }, maxWaitMs);
      }
      this.queue.push(w);
    });
  }

  private release(): void {
    this.active--;
    if (this.queue.length > 0) {
      const next = this.queue.shift();
      if (next?.timer) clearTimeout(next.timer);
      next?.resolve();
    }
  }
}