Scaling WebSockets: From 100k to 1M Users and Tackling Backpressure

The article recounts a real-world incident where a WebSocket-based live commentary platform, designed with a standard fan-out pattern (Redis Pub/Sub to Go services broadcasting to clients), encountered severe memory issues and node crashes when scaling from 100,000 to 1 million users. The core problem was identified as backpressure failure caused by slow consumers rather than a general traffic spike or CPU bottleneck, leading to gigabytes of stale data accumulating in memory.

The Initial Architecture: A Deceptive Fan-Out

The initial design was straightforward: a writer service published score updates to Redis Pub/Sub, and a fleet of Go services subscribed to Redis, then fanned out these updates to connected WebSockets. Each client was given a buffered channel (256 messages) to absorb transient network lags. This design, while simple and effective at lower scales, became a "RAM hoarder" as the user base grew.

type Client struct {
    hub *Hub
    conn *websocket.Conn
    send chan []byte // Buffered channel: 256 slots
}

func (c *Client) writePump() {
    for {
        select {
        case message, ok := <-c.send:
            if !ok { c.conn.WriteMessage(websocket.CloseMessage, []byte{}); return }
            // This Write() call to the WebSocket connection can block if the client is slow.
            // Meanwhile, the 'send' channel continues to fill up.
            w, err := c.conn.NextWriter(websocket.TextMessage)
            if err != nil { return }
            w.Write(message)
        }
    }
}

The Slow Consumer Problem

The math was stark: 200,000 slow consumers, each with a 256-message buffer of ~1KB payloads, amounted to over 51GB of queued messages across the cluster, leading to OOM kills on 16GB nodes. The system was trying to save every packet for every user, ultimately crashing the platform for everyone.

Solutions: Discipline, Brutality, and Jitter

1. Ruthless Send (Drop-If-Full): The `send` logic was changed from blocking to non-blocking. If a client's buffer was full, the message was dropped, and crucially, the client connection was closed. This was based on the principle that real-time data has an expiration date; a client significantly behind is consuming resources for stale, valueless data. This capped memory usage per client.
2. Message Coalescing: For clients who were only slightly slow, a coalescing buffer was introduced. Instead of queuing every individual update, the client struct held a pointer to the *latest* state. The write pump would then send only one aggregated message representing the most recent state, decoupling the server's update rate from the client's consumption rate.
3. Random Jitter for Reconnects: Closing connections for many slow users simultaneously led to a "thundering herd" problem upon reconnect. Thousands of clients attempting to reconnect at the same time overwhelmed load balancers and authentication services. This was mitigated by adding random jitter (0-5 seconds) to client-side reconnect logic, spreading the load over time.

These changes resulted in handling 1.2 million concurrent users with stable memory usage and improved latency, demonstrating that effective WebSocket scaling is fundamentally about managing queues and backpressure, not just opening more sockets.