Meta's Multi-Stage Architecture for Scalable Ads Ranking with LLM-Style Scaling Laws

The Challenge of Scaling Ads Ranking

Meta's ads recommendation platforms must process billions of user interactions daily, retrieving and ranking thousands of ads within milliseconds. Traditional hybrid sequence modeling approaches, which use separate models for user event sequences and sparse feature interactions, face significant challenges. These include lossy knowledge transfer, reliance on manual feature engineering, and scaling ceilings due to interference between ranking and sequence model components, particularly as sequence lengths and transformer models grow.

Multi-Stage Architecture for Efficiency

To address the tension between model complexity and serving efficiency, Meta developed a multi-stage sequence model. This architecture fundamentally decouples the computationally intensive user modeling from the latency-sensitive online ranking. This separation allows for significant scaling of model capacity and complexity in the offline stage without proportionally increasing online serving costs.

• Offline User Model (First Stage): Processes long user histories asynchronously using deep transformer models. It generates and caches user embeddings that capture deep behavioral patterns, independent of specific ad candidates. This stage can scale to thousands of sequence lengths and multiple transformer layers.
• Online Ranking Model (Second Stage): Combines the precomputed offline user embeddings with real-time user signals and ad candidate information for final ranking. This stage is optimized for speed and strict latency budgets, leveraging the rich representations generated offline.

Sequence Model Innovations

• Dense Tokenization: Integrates sparse features with sequential behavioral data into a single dense vocabulary. This allows attention mechanisms to directly learn complex feature interactions from data, moving away from manual feature engineering.
• Target-Aware Multi-Head Attention: A memory-efficient attention mechanism that processes tokenized sparse features and ad candidate information fused with user behavior sequences. It enables each layer to weigh past user behaviors against the specific ad being scored, progressively distilling long sequences into compact representations.

LLM-Style Scaling Laws and Levers

The multi-stage model demonstrates predictable LLM-style scaling laws where performance improvements show a log-linear relationship with compute. This indicates a strong architectural fit for sequence learning, despite the differences from LLMs (e.g., integrating sparse ID features). Key levers for continued scaling include:

1. Balanced Model Shape: Scaling model depth, width, and sequence lengths proportionally to avoid bottlenecks.
2. Multi-Stage Tunability: The ability to scale the offline model (for gradual, unhindered improvements) or the online model (for steeper, but latency-bounded, improvements) independently.
3. Sequence Composition: Emphasizing diverse action types within user sequences for richer behavioral representations.
4. Semantic Feature Representation: Incorporating semantic content features from foundation models to address cold-start scenarios and improve signal coverage.