Achieving Resilience: REST vs. Kafka for Database Failures

The Challenge of Database Downtime

Database failures are an inevitable reality in complex systems. Traditional synchronous REST APIs often suffer from tight coupling, meaning a database outage directly translates to API failures and degraded user experience. This article explores a practical comparison of how two architectural styles - synchronous REST and asynchronous event-driven with Kafka - handle such failures, highlighting the critical role of decoupling for system resilience.

Experiment Design: Simulating Uber's Location Tracking

To quantify resilience, a chaos engineering experiment was set up, mimicking Uber's real-time driver location tracking system. Both architectures were subjected to a sustained load of 50 requests/second and a simulated PostgreSQL crash for 120 seconds. Key metrics included total requests, successful requests, error rate, and latency. This approach provides empirical evidence of architectural robustness under stress.

Architecture A: Synchronous REST

• Direct HTTP POST to REST API
• Immediate database INSERT
• Response after database confirmation
• Result: 50% error rate, immediate client errors, no request buffering.

Architecture B: Asynchronous Kafka

• HTTP POST to Producer API
• Event published to Kafka topic
• Consumer processes events with circuit breaker pattern
• Asynchronous database INSERT
• Result: 0% error rate, all requests buffered and processed, graceful recovery.

The Role of the Circuit Breaker Pattern

A crucial component in the Kafka architecture's success was the implementation of the Circuit Breaker pattern. When the consumer detected database failures, the circuit breaker transitioned to an OPEN state, preventing further attempts to write to the unavailable database and allowing events to queue up in Kafka. Upon database recovery, it transitioned to HALF_OPEN, cautiously testing connectivity before resuming full event processing. This pattern is essential for preventing cascading failures in distributed systems. A simplified JavaScript implementation was provided:

class CircuitBreaker {
  constructor(threshold = 5, timeout = 30000, resetTimeout = 60000) {
    this.state = 'CLOSED';
    this.failureThreshold = threshold;
    this.resetTimeout = resetTimeout;
    this.failureCount = 0;
    this.nextAttempt = Date.now();
  }

  async execute(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() < this.nextAttempt) {
        throw new Error('Circuit breaker OPEN - database unavailable');
      }
      this.state = 'HALF_OPEN';
    }
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  onSuccess() {
    this.failureCount = 0;
    if (this.state === 'HALF_OPEN') {
      this.successCount++;
      if (this.successCount >= 3) {
        this.state = 'CLOSED';
        this.successCount = 0;
      }
    }
  }

  onFailure() {
    this.failureCount++;
    if (this.failureCount >= this.failureThreshold) {
      this.state = 'OPEN';
      this.nextAttempt = Date.now() + this.resetTimeout;
    }
  }
}