Meta Andromeda: Model-System Co-Design for High-Throughput Ads Retrieval
This article details Meta Andromeda's evolution, focusing on a co-design approach involving ML models, system architecture, and GPU hardware to achieve unprecedented scale and efficiency in personalized ads retrieval. It highlights how fusing model and system workloads into a single GPU pipeline, combined with a Mixture-of-Experts (MoE) architecture, enables near ranking-level performance at retrieval scale, processing over a billion ads per second per GPU.
Read original on Meta EngineeringIntroduction to Ads Retrieval at Scale
Meta Andromeda is Meta's machine learning system responsible for personalized ads retrieval, the initial phase in a multi-stage ads recommendation pipeline. Its primary function is to efficiently narrow a vast pool of eligible ads into a smaller, highly relevant set within milliseconds. This stage is crucial because while subsequent ranking stages offer more precise scoring, their computational cost prohibits application at the retrieval's massive scale. The core challenge lies in achieving near ranking-level model performance at an extremely high throughput (billions of ads per second) with a sustainable compute cost, especially as the ad candidate pool rapidly expands due to generative AI creative tools.
Challenges and Bottlenecks in Scaling Retrieval
- Exponential Growth in Candidate Pool: Generative AI has led to hundreds of millions of new ads, increasing the scale of the candidate pool to unprecedented levels.
- Higher Model Performance Demands: Small improvements in retrieval model quality yield significant ROI, necessitating continuous architectural evolution without proportional cost increases.
- Memory Bandwidth Bottleneck: At multi-billion ads per second throughput, intermediate tensors between GPU kernels generate terabytes per second of memory traffic, saturating High Bandwidth Memory (HBM).
- Network Saturation: Shipping ad tensors between local CPU hosts (for per-user logic like index matching) and remote GPU accelerators (for model evaluation) saturates network links, becoming a bottleneck.
Model-System Co-Design Innovations
To overcome these challenges, Meta Andromeda adopted a novel model-system co-design approach, pushing the boundaries of efficiency and performance through synergistic innovations across ML model architecture, system design, and hardware utilization. This collaborative effort with NVIDIA targeted the NVIDIA GH200 Grace Hopper Superchip.
Retrieval Mixture-of-Experts (MoE) Architecture
Inspired by Large Language Models (LLMs), a Mixture-of-Experts (MoE) architecture was introduced for retrieval. This highly expressive yet inference-efficient design establishes a new Pareto frontier for performance-efficiency, achieving near ranking-level model performance with approximately 300x fewer inference-time FLOPs compared to previous iterations. Key aspects include:
- Ensemble of Specialized User Towers: Replaced a single user tower with multiple specialized user experts to capture diverse, time-evolving user interests through high-fidelity signals (sequential ad engagement, user-cohort, LLM-derived features).
- Joint Encoding in Candidate Ads Tower: Incorporates real-time serving context for context-aware ad representations.
- MoE Attention-Based Gating Network: The interaction layer dynamically routes candidates across the most relevant user experts, delivering high performance at a fraction of the computational cost.
Single GPU Pipeline: Fused Kernel and Co-Location
System Design Principle: Locality of Reference
By co-locating related computation and data, and minimizing data movement across memory hierarchies and network boundaries, significant performance gains can be achieved. Fusing GPU kernels reduces HBM traffic, while co-locating per-user logic with GPU scoring minimizes network latency and bandwidth consumption.
The 2026 Meta Andromeda system pushes model-system co-design by unifying fragmented workloads onto a single GPU pipeline with two major innovations:
- Fused GPU Kernel: The retrieval MoE's 300x FLOPs reduction created headroom to fuse both model execution and non-model system workloads (embedding gathering, interaction models, scoring) into a single GPU kernel. This eliminates intermediate tensor materialization to HBM, keeping results in GPU registers and shared memory, thereby overcoming the memory bandwidth bottleneck. This achieves over 10 billion ads per second on a single GH200.
- Co-Location Strategy: Per-user request operations (index matching, top-K selection), traditionally run on separate CPU hosts, are now co-located with the fused GPU kernel on the same GPU server. This drastically reduces network traffic by approximately 100x, as only a compact request payload (user features, targeting query) is sent, and only final top-K results are returned. This shifts the bottleneck back to GPU compute rather than network bandwidth. Redesigning the ad index structure for GPU-parallel execution was crucial for this.
The combined single GPU pipeline processes over one billion ads per second on a single GPU server while scoring every eligible ad, setting a new throughput benchmark for ads retrieval-stage models and demonstrating significant headroom for future growth.