Scaling Strategies at Slack and GitHub: Addressing Thundering Herds, Event Floods, and Database Sharding
This article discusses Maude Lemaire's experiences scaling large-scale systems at Slack and GitHub, highlighting practical challenges and solutions related to performance, load testing, and database architecture. It covers specific issues like thundering herds, event floods, and the implications of ORM usage on database load and sharding strategies.
Read original on The Pragmatic EngineerIntroduction to Scaling Challenges
Scaling popular products like Slack and GitHub involves overcoming various performance bottlenecks and architectural limitations. This article presents insights from Maude Lemaire, a principal engineer who led backend performance at Slack and tackled database scaling at GitHub, offering concrete examples of how these companies addressed critical system design issues.
Mitigating Thundering Herds and Event Floods at Slack
Slack encountered classic distributed system problems such as thundering herds and event floods. A thundering herd occurs when many users simultaneously react to an event (e.g., mass emoji reactions), causing a massive fan-out to all connected clients. Slack addressed this by implementing a seemingly arbitrary but effective limit of 23 custom emojis, significantly reducing network load. Event floods, a related issue, overwhelm individual power users with a high volume of un-deduplicated events via WebSocket connections.
Thundering Herd Mitigation
Instead of allowing unbounded user actions that trigger cascading effects, consider implementing sensible limits or aggregation mechanisms. Sometimes, a small constraint on user behavior can lead to significant system stability gains without impacting user experience negatively.
Evolving Load Testing at Scale
To prepare for a large enterprise customer, Slack needed to load test 500,000 concurrent WebSocket connections. The initial approach of using headless Chrome instances proved prohibitively expensive. This led to the development of Koi Pond, a custom-built, lightweight Go application designed to simulate WebSocket connections cost-effectively. This demonstrates a common trade-off: investing in custom tooling for critical infrastructure needs can yield substantial savings and better control compared to off-the-shelf solutions.
GitHub's Database Scaling Challenges with Active Record
GitHub's Ruby on Rails monolith extensively used Active Record, an Object-Relational Mapping (ORM). While convenient for developers, it often led to unintended database load, with a single homepage refresh triggering 100+ SQL queries across multiple database clusters. A critical consequence was the prevalence of scatter queries (queries without a sharding key) that needed to be executed across all database shards. This limitation severely constrained GitHub's ability to scale horizontally, preventing them from adding more than 16 database shards without significant performance degradation for these scatter queries. This highlights a fundamental tension between developer productivity and infrastructure scalability, where abstractions can inadvertently hide performance costs.
ORM Abstraction Leakage
ORMs can abstract away database interactions, making development faster. However, this abstraction can 'leak', leading to developers being unaware of the underlying SQL queries and their impact on database performance, especially in high-scale, sharded environments. Auditing queries and enforcing sharding key usage is crucial.