Meta's AI Storage Architecture for Maximizing GPU Utilization and Research Velocity

The Challenge: AI Workloads and Storage Bottlenecks

AI model training and development are increasingly bottlenecked by storage and interconnect performance, despite rapid advancements in compute power (GPUs). Traditional storage architectures, often optimized for cost-per-byte and global replication, introduce significant latencies and inefficiencies for data-hungry AI workloads. These bottlenecks lead to GPU stalls, directly impacting computational cost and time-to-market for AI innovations, and hinder research velocity by requiring extensive data movement for geo-distributed GPUs.

Legacy BLOB Storage Architecture Limitations

• Layered Metadata Lookups: The previous `getObject` API involved multiple stateful metadata lookups across several layers (namelayer, volumeslayer, containerlayer). These lookups could cross regions, resulting in hundreds of milliseconds of latency, which is unacceptable for AI workloads requiring millisecond access.
• Dataplane Proxy: All data requests were proxied through API servers, adding latency and consuming power, which is a critical constraint in GPU-heavy data centers.
• Global Replication: Data and metadata were globally replicated by default, optimizing for high durability and availability against region outages, but not suitable for AI's regional data locality needs.
• HDD Optimization: Built for HDDs and optimized for cost per byte, the architecture was ill-suited for the IOPS demands of flash-based AI workloads, where computational cost of storage is negligible compared to GPUs.

Rebuilding the Foundation: Key Design Choices

Meta rebuilt its BLOB storage foundation to directly address the AI workload requirements, making several crucial design choices:

• Unified Metadata Schema: Consolidated disparate metadata stores into a single, flat schema backed by ZippyDB. This enables O(1) lookup complexity for resolving paths to storage addresses, significantly reducing metadata access latency.
• Elimination of Dataplane Proxy: A "fat client" SDK was developed to stream bytes directly from Tectonic storage servers to clients. This reduces latency, increases throughput, and improves power efficiency by offloading proxy servers.
• Regional Deployment: The BLOB-storage stack is now deployed regionally and co-located with GPUs. This aligns with the data locality requirements of AI training jobs, minimizing cross-region data transfers and improving performance and research velocity.

Optimizations for Spikes, Hot Spots, and Tail Latencies

• Distributed Data Cache: Leverages spare memory on GPU hosts as a distributed cache for frequently accessed data, integrating with Meta's Owl subsystem. This absorbs traffic spikes, reduces I/O load on storage, and improves p50/p99 latencies.
• Readplan Metadata Cache: Caches the mapping from paths to storage addresses in a distributed memory store (similar to memcache). This provides 1-2ms access to metadata and mitigates metadata hot shards.
• Hedged Reads: Implements hedged reads on the client side to combat tail latencies caused by slow storage nodes.
• Dynamic Concurrency Control: The client SDK dynamically tunes parallelism based on application-level congestion signals to prevent egress spikes during checkpointing, which can lead to congestion, timeouts, and GPU stalls.