Designing Resilient Event-Driven Webhook Queues for High-Volume Systems
This article discusses a common pitfall in webhook handling where receiving and processing events are conflated, leading to data loss under load. It proposes an 'Ingest-and-Acknowledge' pattern, separating these concerns using asynchronous queues to build resilient, scalable, and idempotent webhook processing systems. The core idea is to quickly acknowledge the sender, persist the raw event, and then process it asynchronously.
Read original on Dev.to #systemdesignThe Problem: Conflating Ingestion and Processing
A frequent architectural anti-pattern in event-driven systems, particularly with webhooks, is performing heavy processing synchronously within the initial request handler. This approach, while simple for development, introduces significant risks in production. When downstream services (e.g., LLMs, CRMs, messaging APIs) experience high latency or failures, the webhook sender might time out and retry or, worse, drop the event entirely. This leads to silent data loss and poor system resilience under traffic spikes.
The Solution: Ingest-and-Acknowledge Pattern
The recommended architectural shift is to decouple event reception from event processing. The 'Ingest-and-Acknowledge' pattern ensures that the initial webhook endpoint performs minimal, fast operations: generating an idempotency key, persisting the raw event to a durable store, and immediately sending a 200 OK response to the sender. The actual, potentially long-running processing is then handed off to an asynchronous queue.
app.post('/webhook/order', async (req, res) => {
const eventId = generateIdempotencyKey(req.body);
// Step 1: persist raw event immediately before any processing
await rawEventStore.insert({ id: eventId, payload: req.body, receivedAt: Date.now(), status: 'pending' });
// Step 2: acknowledge the sender instantly
res.status(200).send('ok');
// Step 3: hand off to the queue this does NOT block the response
await queue.enqueue('process-order', { eventId });
});Key Benefits of Asynchronous Processing
- Idempotency at the door: By generating a deterministic ID and checking against a raw event store, duplicate webhook deliveries are handled gracefully, preventing duplicate processing.
- Zero data loss on downstream failure: Since the raw event is durably stored before any complex processing, failures in downstream services do not result in lost data. Events can be replayed from the queue.
- Decoupled scaling: The lightweight ingestion endpoint can handle extremely high throughput, while the heavier, asynchronous processing can scale independently, based on its own resource requirements and latency tolerance.
Queue-Level Retry and Backoff Policies
Instead of implementing retry logic within business code, leverage the queue's built-in retry mechanisms with exponential backoff. This centralizes failure handling. Also, configure queues to *not* remove failed jobs (e.g., `removeOnFail: false`) to create a queryable dead-letter queue for manual inspection and debugging, preventing silent failures.
It is crucial to apply rate limits at the queue consumer level, not on the webhook ingestion endpoint. The ingestion endpoint should be designed to always accept and store events as quickly as possible. Rate limits on consumers prevent overwhelming downstream APIs (LLMs, CRMs, etc.) while maintaining high availability for incoming webhooks.