AT Protocol: Decentralizing High-Scale Backends with Stream Processing
This article explores the architectural evolution from traditional monolithic web backends to high-scale, eventually consistent stream-processing systems, then extends these concepts to a decentralized context using the AT Protocol. It details how the protocol facilitates shared state and user data repositories across multiple independent services, ensuring data authenticity and reliable event-driven propagation for open application networks.
Read original on Hacker NewsFrom Monolithic to Stream-Processing Architectures
The article begins by tracing the scaling challenges of a traditional web backend, moving from a single SQL database to sharded and replicated SQL, then to NoSQL for better horizontal scalability. It highlights the trade-off between strong consistency and performance, advocating for eventual consistency to achieve higher scale. The transition to NoSQL often sacrifices complex querying capabilities, which are then reintroduced through precomputed views served by dedicated 'View servers'.
To maintain data consistency between the canonical NoSQL data and these view servers, an event log (like Kafka) is introduced. This log captures all changes to the NoSQL cluster, allowing view servers to reliably consume and replay events, ensuring their state remains up-to-date even after failures. This architecture, often termed stream processing, effectively custom-builds a database by separating storage from querying.
Key Stream Processing Components
A stream processing architecture typically involves: 1) A primary, scalable data store (often NoSQL) for canonical data. 2) An event log/message queue to capture and broadcast data changes. 3) View servers that consume events, build and maintain materialized views for efficient querying, often relaxing consistency for higher throughput.
Decentralizing with AT Protocol
The AT Protocol extends this high-scale backend pattern into a decentralized system. Instead of isolated internal services, each component (NoSQL cluster, event log, view server) becomes an external, public service with APIs that anyone can consume or even host. The core idea is to enable multiple instances of these services to work together, forming an open network where applications share state, user accounts, and content.
- Shared Data Model (User Data Repository): Defines a standardized structure for user data, comprising JSON 'records' organized into 'collections' within each user's repository. These repositories are addressable by URLs and cryptographically signed for authenticity.
- Circular Data Flow: Writes are committed to user data repositories, which then emit events into a network of event logs. View services listen to these logs, build local materialized views, and provide data to applications. This ensures that all participants can observe and react to changes across the decentralized network.
- P2P Meets High-Scale: AT Protocol aims to merge peer-to-peer technologies with proven high-scale backend practices, prioritizing user experience comparable to centralized systems while maintaining an open, federated architecture.
This approach addresses the limitations of traditional federation and blockchain architectures for large-scale social applications, aiming to provide a global, convenient experience without sacrificing decentralization and openness.