Designing Recommendation Systems to Combat Clickbait with Semantic Retrieval
This article explores architectural approaches used by Meta, LinkedIn, and YouTube to evolve their recommendation systems beyond simple engagement signals, which often promote clickbait. It details the shift towards semantic retrieval using embeddings and language models to understand content meaning, thereby improving relevance and user experience. The piece highlights different architectural patterns for implementing this shift, from unified models to multi-stage funnels and generative retrieval.
Read original on ByteByteGoSocial media feeds face a fundamental challenge: optimizing for engagement, a common proxy for relevance, often inadvertently promotes low-value clickbait content. This occurs because simple behavioral signals (clicks, likes, replies) are easy to manipulate. The core problem lies in the retrieval stage of recommendation pipelines, which traditionally relies on these behavioral signals to quickly filter a massive corpus of content down to a manageable set for ranking.
The Shift to Semantic Retrieval
To combat clickbait, platforms are moving from behavior-based retrieval to semantic retrieval. This approach focuses on understanding the actual *meaning* of content and its relevance to a user's interests, rather than just their past interactions. The key enabler for this shift is the use of embeddings and dual-encoder (two-tower) models.
How Dual-Encoder Models Work
A dual-encoder model uses two independent encoders: one for the user (profile, activity) and one for each content item (post text, metadata). Both convert their inputs into points in a shared high-dimensional space. Related items and interested users land close together in this space. Content embeddings can be precomputed and indexed, allowing for fast nearest-neighbor searches to find relevant posts for a given user embedding at request time.
Architectural Divergence in Practice
While adopting semantic retrieval, LinkedIn, Meta, and YouTube have implemented distinct architectural patterns:
- LinkedIn's Unified Retriever: Consolidated five separate retrieval systems into a single, fine-tuned LLaMA-3 based dual-encoder model. This simplifies maintenance but requires converting structured features (e.g., view counts) into templated text sequences for the language model. Converting raw counts to ranked buckets significantly improved accuracy.
- Meta's Ranking Funnel: Maintains a multi-stage funnel with an ecosystem of over a thousand specialized models. Candidates pass through increasingly expensive models. Early stages use lightweight models, while late stages apply heavier models to predict multiple user actions (saves, hidden posts) and combine them into a single value score, optimizing for a broader set of objectives beyond raw engagement. This approach prioritizes specialization and fine-grained control over various objectives.
- YouTube's Generative Retrieval: Introduces a novel approach where retrieval is framed as a generation task. Videos are assigned Semantic IDs. A pre-trained language model, adapted with these IDs, generates the Semantic IDs of likely next videos based on user history, effectively bypassing a traditional search index. This innovative method aims to directly generate highly relevant items.
Each approach presents trade-offs between operational complexity, model specialization, and the ability to finely tune for multiple objectives (engagement, diversity, integrity). The common thread is moving beyond simplistic behavioral signals to a deeper, semantic understanding of content and user intent.