Designing for Idempotency and Data Consistency in Distributed Systems
This article explores practical approaches to handling duplicate requests and ensuring data consistency across various systems. It covers strategies like request deduplication at the API gateway, implementing idempotency keys for offline-first applications, and centralizing access control logic to prevent discrepancies, highlighting common failure modes and robust solutions.
Read original on Dev.to #architectureIn distributed systems, operations can often be retried, leading to duplicate requests. This article delves into real-world scenarios where duplicate requests cause undesirable side effects and presents architectural patterns to mitigate these issues, focusing on idempotency and data integrity.
Request Deduplication at the Point of Capture
When integrating multiple data intake channels, preventing duplicate submissions is crucial. The article describes a CRM scenario where a headless JSON endpoint receives lead submissions. To handle repeat submissions, a sophisticated merge strategy is employed:
- Matching Logic: Deduplication relies on strict matching rules (email, then phone) to avoid erroneously merging distinct individuals.
- Master Record Priority: Existing data in the master record is prioritized, with new submissions only filling blank fields and never overwriting populated ones.
- State Preservation: Critical fields like 'stage' or 'owner' are never reset by a repeat submission to avoid disrupting ongoing processes.
- Additive Updates: Tags are appended, not replaced, and trigger associated events. Inquiry messages are captured as inbound notes.
- Controlled State Changes: The only allowed state change for a repeat submission is promoting a 'never-in' contact to 'in-pipeline', ensuring safe, one-way progression.
Idempotency for Offline-First Applications
Offline-first applications often queue writes locally and retry them when connectivity is restored. This naturally leads to the potential for duplicate requests. To counter this, the article proposes an idempotency mechanism using a custom header, similar to `Idempotency-Key`.
public function handle(Request $request, Closure $next): Response {
$key = trim((string) $request->header('Idempotency-Key'));
if ($key === '') {
return $next($request);
}
$cacheKey = $this->cacheKey($request, $key);
if (is_array($stored = Cache::get($cacheKey))) {
return response()->json(
array_merge($stored['body'], ['idempotent' => true]),
$stored['status'],
);
}
$response = $next($request);
if ($response->isSuccessful() && $response instanceof JsonResponse) {
Cache::put(
$cacheKey,
[
'status' => $response->getStatusCode(),
'body' => $response->getData(true),
],
self::TTL_SECONDS
);
}
return $response;
}Key Takeaways for Idempotency Implementation
Optional Header: An optional idempotency key ensures flexibility for clients. Cache Only Successes: Only successful responses are cached; failed requests (e.g., 4xx errors) remain retryable with the same key, allowing clients to fix payloads. Scoped Keys: Idempotency keys should be scoped (e.g., user + method + path) to prevent conflicts and misuse across different operations or users. TTL: A time-to-live (TTL) for cached responses prevents unbounded cache growth while covering retry windows.
Centralizing Access Control Rules
Duplication isn't limited to requests; business logic can also be duplicated across services, leading to inconsistencies. The article highlights the problem of access rules (e.g., who can view paid content) being implemented independently in different components. Consolidating this logic into a single, reusable resolver (`ContentAccessResolver`) ensures a single source of truth and prevents discrepancies when new access points are added (e.g., a new API).