Scaling Meta's Generative Ads Model (GEM) for Ads Recommendations

Overview of Meta's Generative Ads Model (GEM)

Meta's Generative Ads Model (GEM) represents a significant shift in recommendation system architecture, moving from isolated, per-domain optimizations to a single, central foundation model. This approach aims to learn from data across all of Meta's ad platforms, enabling full-funnel optimization across various ad surfaces like Facebook, Instagram, and Reels. The core challenge addressed is how to scale this centralized intelligence while maintaining performance, efficiency, and real-time responsiveness for diverse advertiser objectives.

Key Architectural Innovations and Challenges

• Bending the Scaling Curve with Kunlun: Traditional recommender systems (RecSys) don't scale with transformer architectures like LLMs due to heterogeneous features and shifting distributions. Meta developed Kunlun, an LLM-style architecture co-designed with new GPU hardware, to make larger, deeper models more cost-efficient and enable long-context learning over user engagement history. This resulted in LLM-like scaling laws and 2.5x model complexity growth.
• Efficient Intelligence Transfer with LoopFM: Serving a massive foundation model in real-time is prohibitively expensive and introduces high latency. Meta addressed this with LoopFM, a structured representation post-training technique. LoopFM transfers GEM's internal representations and sequential historical memory to downstream user-facing models, effectively doubling intelligence transfer without additional compute cost or latency. Solaris further harvests gains through elastic pre-computation.
• Accelerating Central Compute Engine: Previously, each ad surface (Facebook, Instagram, etc.) had independent prediction systems. GEM unifies this into a single compute engine that jointly predicts clicks and conversions across all surfaces and advertiser objectives. This allows for generalizable representations learned on data-rich tasks to transfer to data-sparse conversion tasks, using cost-free data mixing and a domain expert framework.
• Scaling Training Efficiency with Custom Kernels and Parallelism: Training such a large RecSys foundation model across thousands of GPUs requires intricate orchestration. Unlike LLM workloads, recommendation workloads have unique data characteristics that make high GPU utilization challenging. Meta developed custom kernels with low precision, new 5D parallelism strategies co-designed with GPU hardware, and optimized model architecture to achieve 20-25% model FLOPS utilization (MFU) and 4x total training FLOPs.

Kunlun Architecture Details

The Kunlun architecture is central to GEM's ability to handle ultra-long user sequences and massive user-item interaction features. It standardizes an LLM-style framework to better capture RecSys-specific traits (heterogeneous features, data dynamics) and introduces several innovations:

• Standardized Unified Framework: Each layer incorporates customized attention, feed-forward networks, a simplified summarization component, and Mixture-of-Experts (MoE) style cross-interaction for modeling complex interactions.
• Scalable Architecture Co-design: Computation complexity is reduced via Block Attention (quadratic to linear complexity for sequences), and components are unified (e.g., Per-Example Feed-Forward Networks reformulated into Generalized Dot-Product Attention) for up to 2x forward speedup. Expert Parallelism enables scalable cross-interaction.
• Efficient Sequence Scaling: Innovations like Component Skip Connection (CompSkip) reduce FLOPS by selectively activating sequence modules (43% reduction with negligible quality loss). Sequence Unification (OneSeq) allows a single, unified sequence layer to process long, heterogeneous user sequences, maximizing GPU efficiency.