Leveraging Opportunistic Capacity for AI Inference in Ads Recommendations at Scale
This article from Meta Engineering details the system design behind using idle, opportunistic inference capacity to improve ads recommendations. It explores how two elasticity levers, Rank More Ads (RMA) and Serve Better Models (SBM), dynamically adjust workload based on available resources, while ensuring baseline performance is protected through isolation, feedback loops, and value-aware load shedding. The system addresses the paradox of globally constrained but locally idle capacity in large-scale AI inference fleets.
Read original on Meta EngineeringThe Inference Capacity Paradox
Large-scale AI inference fleets often face a paradox: they can be globally constrained in terms of overall capacity, yet simultaneously have significant pockets of idle resources locally. This idleness stems from several factors, including provisioning for peak demand (leading to idle capacity during off-peak times), mismatches between model development and capacity delivery timelines, regional resource ratio imbalances (e.g., more CPUs than accelerators or vice-versa), and hardware heterogeneity across different accelerator types. Meta designed a system to aggregate these fragmented, temporary idle resources into a shared *opportunistic capacity pool*.
Key Properties for Opportunistic Workloads
Successfully utilizing volatile, preemptible opportunistic capacity requires workloads with specific characteristics. The workload must exhibit a high degree of elasticity, able to quickly expand or contract its optional work without impacting baseline performance. It needs to maintain sustained quality at scale, meaning additional capacity continues to meaningfully improve output. Finally, limited cross-service dependencies are crucial to minimize coordination overhead and improve resilience to capacity fluctuations.
Elasticity Levers: RMA and SBM
Meta identified two complementary elasticity levers to consume this idle capacity in their ads recommendation system:
- Rank More Ads (RMA): Increases the number of candidate ads evaluated by the ranking stages. This improves the chance of finding more relevant ads but requires both CPU (for upstream processing like retrieval and feature processing) and accelerator capacity.
- Serve Better Models (SBM): Adjusts the share of requests routed to more powerful, computationally expensive models. SBM primarily consumes accelerator capacity without significantly increasing upstream CPU demand, as the number of ads entering ranking remains constant. It uses request-value signals to prioritize which requests benefit most from better models.
These levers are complementary as they consume different resource combinations (RMA: CPU+Accelerator, SBM: Accelerator-only), allowing the system to utilize a broader range of opportunistic capacity. The system dynamically adjusts the allocation between RMA and SBM based on their measured performance and resource requirements.
Safeguarding Baseline Performance
Design Principle for Opportunistic Capacity
A core design principle for systems utilizing opportunistic capacity is that losing this capacity must never overload or compromise the performance of the baseline workload. The system must degrade gracefully.
To achieve this, the system employs several critical mechanisms: physical and logical isolation (SBM uses separate routing targets for baseline vs. powerful models, RMA logically limits incremental work on shared replicas), closed feedback loops (PID controller for RMA based on tail latency, value-aware model selection with calibration for SBM), and multi-layer value-aware load shedding (server-side and client-side mechanisms that immediately shed lower-value opportunistic work if capacity is suddenly reclaimed, operating faster than central control loops). These measures ensure that optional work expands and contracts safely, protecting the core latency and reliability targets of the baseline ads serving.