F3 Lightning: Scaling Ads ML Feature Delivery at Meta
This article discusses F3 Lightning, an infrastructure layer built on Meta's Facebook Feature Framework (F3) to improve the release process of machine learning features for ads ranking models. F3 Lightning decouples feature releases from shared execution, allowing independent deployment while preserving F3's efficiency gains by optimizing computation sharing at runtime through stable identifiers, a single execution plan, shared memory, and dense index mapping.
Read original on Meta EngineeringThe Challenge: Efficient ML Feature Delivery at Scale
Meta's F3 (Facebook Feature Framework) handles over 20,000 features for ads ranking models, compiling them into a single merged Directed Acyclic Graph (DAG) to optimize computation by sharing common operations. While this approach was highly efficient, it created a significant bottleneck: the merged graph was also the release unit. A single error in any feature definition could block the entire release pipeline, leading to frequent failures and slow deployments as the number of features grew.
F3's Shared Execution Architecture
F3 represents feature extraction as a DAG where nodes perform work (e.g., reading input, filtering, aggregation) and edges define dependencies. Engineers author features independently, and the F3 compiler merges these into one graph, collapsing duplicate operations. This ensures shared work runs only once per request, saving significant CPU. However, this monolithic release unit meant all features shipped together or not at all.
F3 Lightning: Decoupling Release from Execution
Core Problem & Solution
Problem: Monolithic release of ML features due to shared execution graph led to frequent blockages. Solution: F3 Lightning separates release granularity (per feature) from execution efficiency (shared runtime).
F3 Lightning was designed to allow each feature to be independently compiled, validated, deployed, and rolled back. The main challenge was to achieve this without sacrificing the computational efficiency gained by F3's shared execution. This was addressed through a series of key optimizations:
- Deployment Granularity: Each feature is compiled into its own isolated configuration, making it independently releasable. This ensures that a problematic feature does not block others.
- Efficiency Parity: To maintain performance, F3 Lightning re-establishes shared computation at runtime without merging features at compile-time. This includes: Stable Computation Identifiers, One Execution Plan Across Every Feature, a Shared Memory Buffer, and Dense Index Mapping.
Key Optimizations for Efficiency Parity
- Stable Computation Identifiers: The compiler assigns identifiers based on the computation's logic (operation, parameters, input identifiers), not its position in a graph. This allows the runtime to recognize and deduplicate identical computations across independently compiled features.
- One Execution Plan: At service startup, F3 Lightning builds a single, topologically sorted execution plan from all active feature configurations. Duplicate computations (identified by stable IDs) are collapsed into a single node, ensuring they run only once. The plan is flattened into an ordered list for efficient forward-pass execution.
- Shared Memory Buffer: Instead of scattered per-feature data containers, F3 Lightning uses a single memory buffer per request. This improves data locality, reduces cache misses, and eliminates per-feature memory allocations. Intermediate results are stored once, with consumers receiving references.
- Dense Index Mapping: To avoid costly hash lookups at runtime, F3 Lightning assigns a consecutive integer slot to every unique computation identifier at startup. All references are rewritten to use these dense indices, enabling direct array access for maximum speed.
These optimizations allow F3 Lightning to deploy ML features 3.6x faster than traditional F3, with no increase in infrastructure compute cost, significantly improving iteration speed for model development.