Canva's S3-Based Session Revocation Architecture
Canva redesigned its session revocation infrastructure, moving from MySQL lookups to an S3-based approach. This architecture stores revocation data as compact, immutable records in S3, which are then distributed to application gateways as in-memory indexes. This shift significantly improved deployment speed, reduced database load, and decreased memory footprint while supporting hundreds of millions of active sessions.
Read original on InfoQ ArchitectureThe Challenge of Scalable Session Revocation
As platforms scale, managing session revocation across a vast number of active sessions becomes a significant architectural challenge. Traditional methods, often relying on centralized databases for every authentication request, can lead to substantial database load, especially during deployments or peak usage. Canva faced this issue with its prior MySQL-based revocation system, where hundreds of gateway instances could simultaneously query for millions of revocations, creating a "coordinated database load" bottleneck and impacting deployment speed.
Canva's S3-Based Solution Overview
Canva's innovative solution leverages Amazon S3 as a durable, highly available store for revocation data, sidestepping the need for another operational datastore like Redis. The core idea is to treat revocation data as immutable objects stored in S3, which are then fetched and indexed in-memory by application gateways. This design pattern shifts the burden of frequent database lookups to an efficient, distributed object storage system, enabling gateways to authenticate requests without contacting a networked datastore for most cases.
Key Architectural Principles
Immutability: Storing revocation records as immutable objects simplifies data consistency and recovery. Cache-aside with asynchronous updates: Gateways maintain an in-memory cache, updated asynchronously from S3. Leveraging object storage beyond files: Using S3 not just for large files but as a durable, high-throughput coordination primitive for small, structured data.
Data Structure and Distribution
- Revocation data is divided into 30-minute S3 objects.
- Each revocation is a compact 16-byte binary record (principal + timestamp).
- Gateways download relevant S3 objects and build in-memory, sorted arrays for efficient searches.
- Conditional GETs are used to download only changed chunks, minimizing bandwidth.
- Data older than 12 hours is discarded from gateway memory.
Worker Processes and Concurrency
Asynchronous workers are responsible for scanning for new revocations from the primary database, merging them into the latest 30-minute S3 object, and uploading the updated object. To manage concurrent updates to the same S3 object by multiple workers, conditional PUTs are employed for optimistic concurrency control. While ZooKeeper leader election can help reduce conflicts, it's not strictly required for the correctness of the system, indicating a robust design capable of handling eventual consistency or retries.