Meta's Prometheus Supercluster: Architecting a Gigawatt-Scale AI Training Infrastructure
This article details Meta's Prometheus supercluster, a massive AI infrastructure designed for training foundational models, emphasizing the architectural challenges and solutions in hardware, networking, and reliability. It highlights the deployment of next-generation NVIDIA GPUs, a specialized multi-layer network fabric, and robust fault tolerance mechanisms required to achieve high 'goodput' and minimize interruptions in large-scale AI training.
Read original on Meta EngineeringIntroduction to Prometheus: A Gigawatt-Scale AI Supercluster
Meta's Prometheus supercluster is an ambitious AI infrastructure project, designed to train next-generation foundational models like Muse. The system is engineered to scale to approximately 1 million GPUs, functioning as a single, tightly interconnected machine. This scale demands significant advancements in hardware, networking, and overall system design to ensure high performance and reliability for demanding AI workloads.
Hardware Architecture and GPU Platforms
Prometheus leverages multiple generations of NVIDIA GPU platforms, including the GB200 (Catalina) and GB300 (Clemente) NVL72 architectures. These platforms integrate 72 GPUs, often with companion CPUs and high-bandwidth NICs (e.g., 800G I/O per GPU), into a single logical domain. The NVL72 architecture significantly increases the scale-up domain compared to previous generations, offering massive compute, memory, and NVLink bandwidth within a single rack or dual-rack configuration.
Key Hardware Features
- Catalina (NVIDIA GB200 NVL72): Two-rack configuration with 36 GPUs per rack, 1:1 CPU-to-GPU ratio, 800G I/O per GPU via ConnectX-7 NICs.
- Clemente (NVIDIA GB300 NVL72): Single-rack deployment with 2:1 GPU-to-CPU ratio, 800G per GPU natively via ConnectX-8 SuperNICs, increased HBM capacity (288GB per GPU), and improved FP4 throughput.
Network Design for AI Training Workloads
Unlike traditional web services, AI training requires extremely high bandwidth, low latency, and tight coordination across thousands of GPUs. Meta addresses this with a dedicated, Ethernet-based Back-End (BE) network for direct GPU-to-GPU communication. This network features a hierarchical design to match training parallelism, with low-latency, high-bandwidth fabrics within buildings and an aggregation network connecting these building fabrics across distances up to 100 km.
- Building-Level Fabrics: Multi-layer topologies interconnecting up to 20,736 GPUs. Initially used Disaggregated Scheduled Fabric (DSF) to mitigate congestion, later evolving to Non-Scheduled Fabric (NSF) with advanced Ethernet switches and adaptive routing.
- Backend Aggregation (BAG) Network: A multi-layer hub-and-spoke network using deep-buffered chassis switches (e.g., Arista 7800-series with Broadcom Jericho3) to connect building fabrics, delivering multiple Pbps of bandwidth across regions.
Network Optimization for AI Training
The network design is critical to AI training 'goodput.' Meta's approach minimizes exposure to network communication bottlenecks by optimizing parallelism design, traffic patterns, and communication algorithms in close collaboration with their AI research teams, ensuring GPUs spend minimal time waiting on network operations.
Engineering Reliability and Fault Tolerance
At Prometheus' scale, reliability is paramount, as a single fault can halt an entire training job. Meta employs a co-engineering model with NVIDIA, including shared telemetry and joint triage, to rapidly identify and fix root causes of system halts. A key reliability strategy is managing the 72-GPU failure domain: although a larger domain increases the 'blast radius' of a fault, Meta uses in-rack spare GPUs (64 active, 8 spares) within each NVL72 rack. This allows failed GPUs to be replaced immediately, enabling training jobs to restart from checkpoints rather than waiting for physical hardware repair, significantly improving Mean Time Between Interruptions (MTBI) and overall job goodput.