Designing a Scalable News Feed: Fan-Out Strategies and the Celebrity Problem

The Challenge of News Feed Design

Designing a news feed is a classic system design problem that highlights the importance of understanding user distribution and read-to-write ratios. The "celebrity problem"—where a user with millions of followers posts—exposes the limitations of naive fan-out strategies. A successful design must efficiently deliver content to a wide range of users, from those with few followers to those with millions, while maintaining low latency and high availability.

Fan-Out on Write (Push Model)

In a push model, when a post is created, it is immediately written into the feed list of all followers. This approach ensures fast reads, as user feeds are precomputed and often cached. It works well for users with a typical number of followers, where the write cost is manageable.

def on_post_created(post_id, author_id):
    followers = get_followers(author_id)
    pipe = redis.pipeline()
    for follower_id in followers:
        key = f"feed:{follower_id}"
        pipe.lpush(key, post_id)
        pipe.ltrim(key, 0, 799) # keep newest 800 
    pipe.execute()

Fan-Out on Read (Pull Model)

The pull model defers feed computation until a user requests it. When a user opens their feed, the system fetches recent posts from all accounts they follow, merges them, and sorts them. This approach makes writes cheap but shifts the complexity and cost to reads, potentially leading to performance issues if a user follows many accounts or if read volume is high.

def get_feed_pull(user_id, count=20):
    followees = get_followees(user_id)
    posts = []
    for author_id in followees:
        posts.extend(
            get_recent_posts(author_id, limit=count)
        )
    posts.sort(key=lambda p: p.created_at, reverse=True)
    return posts[:count]

The Hybrid Fan-Out Strategy

The most robust solution combines both push and pull models. A configurable threshold (e.g., 100,000 followers) determines the strategy:

• Below Threshold: Accounts with fewer followers use fan-out on write. Posts are pushed to followers' feeds for fast reads.
• Above Threshold (Celebrities): Accounts with many followers use fan-out on read. Posts remain on the author's timeline and are pulled on demand when a follower loads their feed.

This hybrid approach ensures that both write amplification for popular accounts and read latency for regular users are managed effectively, providing a scalable and responsive news feed system.