Hierarchical Interest Representation for Meta Ads Deep Funnel Optimization: System Architecture
This article from Meta Engineering introduces Hierarchical Interest Representation (HIR), an upstream representation layer designed to enhance deep funnel ad optimization. HIR leverages a transformer-based graph learning architecture with bias-aware attention and self-supervised cross-view distillation to generate unified embeddings and Bag-of-Meaning tokens for users and ad entities. The system addresses challenges of sparse connections and long-range relationships in large-scale ad networks by blending real-world knowledge from LLMs with user engagement signals, enabling improved personalization and ranking across Meta's platforms.
Read original on Meta EngineeringMeta's Hierarchical Interest Representation (HIR) is an innovative system designed to tackle the complexities of deep funnel advertising optimization at an enormous scale. It aims to connect users with highly relevant ads by understanding latent interests, even from sparse engagement signals. This involves building a sophisticated knowledge representation layer over a vast, dynamic graph of users, advertisers, and products.
Core System Design Principles
- Dimension Reduction: Projects a raw, sparse graph into a more stable 'super-graph' of latent interest primitives, making sparse user-ad edges denser and more stationary.
- Knowledge Enrichment: Integrates multimodal content (text, images, video) from advertisers and products, processed by LLMs, to understand the semantics of entities beyond just engagement data. This helps generalize to unseen entities.
- Unified Relational Representation: Learns joint embeddings for users, entities, and their interest primitives in a single metric space, enabling inference of relationships and affinities (e.g., user proximity to interests, ad alignment with interests).
- Multi-Hierarchical Granularities: Learns super-graphs that cascade through multiple layers to capture both high-level, stable interest anchors and fine-grained, sparse deep funnel intent, offering flexibility for various downstream applications like ranking and personalization.
Architectural Components and Challenges
The HIR architecture is built around an Enriched Engagement Graph and a Hierarchical Encoder. The graph unifies various entity types (users, ads, advertisers) with typed, weighted, time-decayed edges, capturing diverse interactions. Scaling this graph to billions of nodes and interactions monthly is a significant challenge, requiring efficient storage and processing of rich initial embeddings and learnable ID embeddings via deep hash embeddings.
The Hierarchical Encoder, a transformer-based model, is central to learning representations from this massive graph. Key design considerations include:
- World Knowledge Integration: Utilizing custom LLM inference engines to process multimodal advertiser and product content, providing semantic context.
- Structure Encoders: Fusing node-type, deep-hash-controlled node-ID, world knowledge features, and metadata (Node Encoder) along with positional encoding (Position Encoder) and edge type/weight/temporal signals (Edge Encoder) to provide rich input to the transformer.
- Bias Composition: Incorporating graph-structural signals (node-type transitions, shortest-path distance) as attention biases. This makes the transformer topology-aware, crucial for capturing long-range relationships without over-smoothing, which is a common issue in traditional message-passing graph neural networks.
- Memory-Efficient Attention (FlexAttention): Addresses the computational cost of graph-structural bias in attention by computing bias terms on the fly, avoiding materialization of full pairwise bias matrices. This allows for full graph-structural awareness while maintaining memory efficiency and supporting variable-length subgraphs.
Scaling Graph-Based ML Models
Designing machine learning systems over extremely large, sparse graphs like Meta's ad network presents challenges in data representation, computational efficiency, and capturing meaningful relationships. Strategies like hierarchical representations, knowledge distillation, and specialized attention mechanisms (e.g., FlexAttention) are crucial for making these systems tractable and performant. The use of deep hash embeddings helps manage the vocabulary scale for dynamic entities.
Training the Hierarchical Encoder employs Cross-View Distillation, a self-supervised scheme where a 'teacher' view (broad) and 'student' view (narrow) are sampled for each anchor node and passed through the encoder. This approach helps the model learn robust representations by reconciling different perspectives of the graph data.