Optimizing Ads Service Latency with Custom Kernel Scheduling via sched_ext at Meta

The Challenge: Latency in High-Scale Ad Serving

At Meta's scale, the ads serving fleet handles over 5 million requests per second, where even small latency degradations can severely impact ad relevance and advertiser ROI. A Linux kernel upgrade (v6.9 with EEVDF scheduler) introduced a latency regression, reducing the number of ads ranked and forcing a subset of hosts to remain on an older kernel, creating significant technical debt. This highlighted the need for a more controlled and workload-aware scheduling mechanism.

Introducing sched_ext: A BPF-based Extensible Scheduler

To address the latency regression and gain finer control over CPU scheduling, Meta leveraged sched_ext, an open-source, BPF-based scheduler framework that entered kernel v6.12. This framework allows developers to implement custom scheduling policies as BPF programs, which are then called by the kernel through event-driven callbacks (e.g., thread wake-up, enqueue, dispatch, idle transitions).

Custom Policy Design for Ads Workload

Meta's custom scheduling policy for the ads workload soft-partitions CPUs into two pools: one for latency-critical request path threads and another for less latency-sensitive work. This domain-specific knowledge, encoded directly into the BPF program, prioritizes work that improves p99 request latency. The policy dynamically adjusts pool sizes using load-based heuristics, aiming to keep related work on the same CPUs to improve L3 cache locality and reduce costly DRAM access.

The policy is packaged as a user-space binary, enabling rapid experimentation and deployment. Changes can be rolled out by simply restarting the scheduler process to load a new BPF program, drastically reducing the iteration time from months (for kernel patches) to days.

Impact and Strategic Advantages

• Performance Gains: Initial launch delivered a 28% reduction in p99 latency on the ads retrieval path, a 1.1% increase in weighted-ads-ranked, and 3.28 MW of power savings across the fleet.
• Continuous Optimization: Follow-on policy updates (purely user-space changes) further reduced p99 latency by an additional 60% and decreased timeout errors by 18%. This rapid iteration capability transformed sched_ext from a kernel upgrade unblocker into a continuous optimization platform.
• Decoupled Development: sched_ext provides a parallel and decoupled optimization path, allowing Meta to refine custom schedulers independently of upstream Linux kernel evolution, ensuring critical services remain optimized despite core scheduler changes.
• Industry Asset: sched_ext was upstreamed to Linux v6.12, making this powerful mechanism available to any operator needing workload-specific scheduling policies without kernel forking.