Correcting Event Sourcing: Achieving Consistency in AWS Architectures
This article details a team's journey in refining their event-sourced architecture on AWS, addressing critical consistency issues. It highlights common pitfalls in event publishing and state management, offering solutions through the Outbox Pattern with DynamoDB transactions and more granular, business-oriented domain events. The discussion also covers strategies for handling event reordering and ensuring eventual consistency in distributed systems.
Read original on Dev.to #systemdesignInitial Event Sourcing Flaws
The article begins by outlining common architectural missteps observed in an existing distributed event-sourced system. The primary issues revolved around eventual consistency guarantees and the nature of the events themselves. Understanding these flaws is crucial for designing robust event-driven systems.
- Lack of Transactional Consistency: Events were published to a notification service, and application state was written to a database separately. This created a race condition where one operation could succeed while the other failed, leading to an inconsistent system state.
- Coarse-Grained Events: Events like `AccountChanged` or `PersonalDetailsChanged` contained the full state rather than just the delta of what changed. This made it difficult to understand the actual business event without comparing previous states, hindering effective event consumption and replay.
Implementing the Outbox Pattern with DynamoDB Transactions
To address transactional consistency, the team adopted the Outbox Pattern, leveraging DynamoDB transactions. This pattern ensures that the application state change and the event publication are treated as a single, atomic operation, preventing inconsistencies.
- A dedicated table stores domain events.
- Another table stores the current application state.
- A separate table, potentially with a TTL, is used for sending out integration events. This separation allows for including additional metadata with events without increasing read costs on the primary state table, and DynamoDB Streams are then used to publish these events to SNS.
Outbox Pattern in DynamoDB
By using DynamoDB transactions, the write to the application state table and the write to the outbox table (for the event) can be performed atomically. A separate process (e.g., a Lambda triggered by DynamoDB Streams on the outbox table) then publishes the event, ensuring that the event is only published if the state change was successfully committed.
Refining Event Granularity and Resequencing
The article emphasizes a shift towards more business process-oriented events (e.g., `AccountCreated`, `EmailAddressReplaced`) that contain only the delta of changes. Each event and the application state also include a revision number, critical for optimistic concurrency control and enabling consumers to reorder events.
Recognizing that strict message order guarantees are challenging across multiple distributed systems (e.g., DynamoDB Streams guarantee order per item, not across all events), the solution explicitly delegates resequencing responsibilities to the consumers. The revision number in events and state provides the necessary mechanism for consumers to reconstruct the correct sequence of events, aligning with the Resequencer Pattern.