Uber RAMEN: Real-time Notification System for Ride-Hailing Apps

The Challenge: Real-time, Reliable Notifications at Scale

Delivering instant ride offers and real-time location updates to millions of active users in a ride-hailing application presents significant system design challenges. The core requirements include extremely low latency (under 100ms), guaranteed delivery even on weak mobile networks, and efficient resource utilization to avoid overwhelming backend systems. Traditional polling mechanisms are highly inefficient for this scale, leading to excessive requests, wasted server resources, high latency, and significant battery drain on client devices.

RAMEN's Three-Tier Architecture

Uber's RAMEN (Real-time Asynchronous Messaging Network) is a custom-built push messaging infrastructure. It employs a three-tier architecture to manage the complexity of decision-making, payload construction, and message delivery:

1. Fireball Service (Decision Engine): Consumes Kafka events, evaluates business rules, and determines *when* a message needs to be pushed, handling priority and localization.
2. API Gateway (Payload Builder): Aggregates data from various microservices, builds the actual message payloads, and serializes them (e.g., using Protobuf). This layer determines *what* to push.
3. RAMEN Server (Delivery Layer): Manages millions of concurrent persistent connections, routes messages to the correct device, and guarantees at-least-once delivery. This layer handles *how* to push.

Protocol Evolution: From SSE to gRPC over QUIC/HTTP/3

The choice of transport protocol is critical for real-time systems. Initially, RAMEN utilized Server-Sent Events (SSE) over HTTP/1.1. While simple, SSE is unidirectional (server-to-client only), requiring separate HTTP POST requests for client acknowledgments (ACKs). It also suffered from head-of-line blocking and heavy JSON payloads. Uber transitioned to gRPC bidirectional streams over QUIC/HTTP/3 for significant improvements:

• Full-duplex Communication: Allows both server and client to send data simultaneously on a single connection.
• Binary Framing (Protobuf): Enables smaller, more efficient payloads and lower CPU usage.
• Multiplexing: Multiple logical streams can share one physical connection, eliminating head-of-line blocking.
• QUIC (UDP-based): Offers improved resilience on unstable mobile networks, faster connection establishment, and connection migration (e.g., switching from Wi-Fi to 4G without dropping the stream).
• In-stream ACKs: Acknowledgments are sent directly on the gRPC stream, streamlining communication.

Scalability and Reliability for Stateful Connections

RAMEN servers are stateful, meaning each server holds specific TCP/gRPC sockets for particular users. This poses unique challenges for scalability and high availability. To manage millions of connections across a cluster of hundreds of servers, Uber employs Apache Helix and ZooKeeper for sharding and automatic rebalancing. ZooKeeper stores cluster topology, while Helix detects server failures and redistributes shards (groups of user connections) to healthy servers, allowing clients to seamlessly reconnect.

• At-Least-Once Delivery: Achieved using a persistence layer (Cassandra for durable storage, Redis for fast caching). Messages are written to Cassandra, cached in Redis, then pushed. If no ACK is received from the client within a timeout, the message is retried from Cassandra.
• Sequence Numbers: Each message carries an incrementing sequence number. Upon client reconnection, the client informs the server of the last received sequence number, enabling the server to resend only the missed messages and prevent duplicates.
• Graceful Connection Draining: To prevent a "thundering herd" problem during deployments or scaling down, RAMEN servers do not abruptly terminate connections. Instead, they stop accepting new connections and send a "Graceful Disconnect" message to existing clients with a randomized backoff hint, staggering client reconnects over a period.