Real-time notifications at scale: fan-out on write vs fan-out on read
Andrei Sato
·2 views
building a notification system for a social platform with millions of users always brings up the fan-out on write vs. fan-out on read dilemma. we have users with millions of followers, so pure fan-out on write means an insane amount of writes for a single action, which can overwhelm our system. imagine a popular user posting, and suddenly we're writing to 10M separate notification inboxes. that's a lot of i/o.
on the other hand, pure fan-out on read means every time a user opens their notifications, we have to aggregate and filter potentially a massive amount of data in real-time. this can lead to unacceptable latency, especially for users with many subscriptions. it's a classic performance vs. cost vs. architectural complexity tradeoff.
we've been thinking about a hybrid approach: fan-out on write for users with fewer than, say, 10,000 followers, and fan-out on read with heavy caching for celebrity-tier users. but even that has complexities around cache invalidation and ensuring consistent visibility. for those managing systems with 10M+ users and power-law distribution of followers, what strategies have you found effective? any specific database choices or caching layers that really helped?
3 comments