MetaRoCE: A New RDMA Transport Protocol for AI-Scale Ethernet Networks
MetaRoCE is a new RDMA transport protocol designed by Meta for AI workloads on commodity Ethernet. It addresses the challenges of scaling AI infrastructure to millions of GPUs by moving network intelligence to the endpoints, enabling native out-of-order delivery, multipathing, and loss tolerance, which are critical for high throughput and low tail latency in large-scale distributed AI training and inference. The protocol is being open-sourced through the OCP to foster industry adoption and collaboration.
Read original on Meta EngineeringMetaRoCE is a novel RDMA (Remote Direct Memory Access) transport protocol developed by Meta to optimize network performance for large-scale AI workloads. Unlike traditional RoCE (RDMA over Converged Ethernet) which expects lossless, in-order packet delivery, MetaRoCE is designed to operate efficiently over commodity Ethernet, which is inherently lossy and can deliver packets out-of-order. This shift in design philosophy is crucial for supporting AI clusters scaling to hundreds of thousands or even millions of GPUs distributed across multiple data centers and regions, where traditional network assumptions can lead to significant compute capacity waste due to network friction.
Core Design Principles and Architectural Innovations
The core innovation of MetaRoCE lies in shifting network intelligence from the fabric (switches) to the endpoints (NICs). This allows for a more adaptive and resilient transport layer. Key design principles include:
- Native Out-of-Order Delivery: MetaRoCE treats out-of-order packet arrival as the norm. Packets carry their own destination information, allowing data to be written directly to memory without reorder buffers or head-of-line blocking, which are common bottlenecks in traditional RDMA.
- Native Multipathing: Connections are spread across multiple logical paths, each with its own state (RTT, ECN, utilization). The NIC dynamically selects paths, rebalancing traffic to avoid congested or failed links. This prevents a single slow path from stalling the entire connection.
- Loss Tolerance by Design: The protocol anticipates and tolerates packet loss, unlike traditional RoCE which relies on PFC (Priority Flow Control) for losslessness. It uses per-path sequence numbering and selective acknowledgments to retransmit only missing packets on the specific path where loss occurred.
- Congestion Control from Both Sides: MetaRoCE combines sender-driven AIMD (Additive Increase, Multiplicative Decrease) congestion control with receiver-driven fair-share rate hints. This dual-sided approach helps in faster incast resolution, better fairness, and lower tail latency.
- Topology Independence: It operates effectively over various network topologies (fat-tree, multiplane, deep/shallow-buffer fabrics) by only requiring basic switch features like ECN marking and ECMP, avoiding proprietary network functionalities.
Separation of Concerns: Streams and Paths
A significant architectural decision in MetaRoCE is the separation of ordered message streams from network paths and bandwidth. A single connection can carry multiple independent ordered streams (e.g., for different communicators or collectives) above, while utilizing many paths below, all managed by a unified congestion controller. This design ensures that the connection state does not grow linearly with workload parallelism, enhancing scalability without modifying existing RDMA Verbs APIs for most applications.
System Design Takeaway
When designing high-performance distributed systems, especially for AI/ML workloads, consider that network fabrics may not always guarantee in-order, lossless delivery. Architecting resilience and adaptivity at the transport layer or application layer, through techniques like out-of-order processing, multipathing, and explicit loss handling, can lead to more robust and scalable solutions than relying solely on network guarantees.