Automated Resource Right-Sizing for Serverless Media Processing at Netflix
This article details Netflix's Stratum Resource Tuner (SRT), an automated system for right-sizing container resources within their serverless media processing platform, Stratum. It addresses the challenges of manual over-provisioning and evolving infrastructure by continuously monitoring resource usage, generating recommendations, and safely applying them through a verification process. The system aims to optimize operational costs and reduce cognitive load for engineers managing distributed media workloads.
Read original on Netflix Tech BlogThe Challenge of Resource Over-Provisioning in Distributed Systems
In large-scale distributed systems, particularly serverless environments like Netflix's Stratum media processing platform, managing container resource allocations (CPU, memory, disk, network) is a significant challenge. Engineers often over-provision resources for new functions or copy existing configurations, leading to substantial waste as infrastructure evolves, codecs improve, and workloads dynamically change. Manual tuning is high-friction, risky, and does not scale, resulting in increased operational costs and cognitive load.
Introducing the Stratum Resource Tuner (SRT)
Netflix developed the Stratum Resource Tuner (SRT) to automate the right-sizing of container resources. SRT operates without human intervention once functions opt-in, following a five-step workflow: metrics collection, recommendation generation, experimental verification, application, and post-apply monitoring. This automated approach aims to ensure optimal resource utilization and cost efficiency across hundreds of thousands of daily container executions.
- Metrics Collection: Stratum containers report granular resource utilization (CPU, memory, disk, network) at sub-minute resolution. These metrics flow through Netflix's Keystone event streaming platform into both an Elasticsearch store for short-term lookups and an Iceberg table for long-term trend analysis.
- Recommendation Generation: A daily scheduled workflow processes a month of usage data from Iceberg. It generates recommendations by targeting p99 usage for compressible resources (CPU, network) and p99.9 usage with a safety buffer for critical resources (memory, disk) to prevent hard failures. Safeguards include discarding minor reductions and clamping maximum reductions (e.g., 15% for memory/disk) per cycle.
- Verification and Application: Recommendations undergo experimental verification, typically by running a percentage of traffic on the new configuration. If experiments pass, the recommendation is applied. If they fail, it is rejected, and the original configuration remains. This robust safety architecture is crucial for automated production changes.
- Monitoring: After applying a recommendation, SRT enters a soak period, continuously monitoring post-apply metrics for anomalies to ensure stability.
Safety Architecture and Trade-offs
Safeguarding Automated Changes
The SRT incorporates several critical safety mechanisms: configurable operational thresholds, a gradual rollout of changes, and strict limits on the magnitude of resource reductions. The distinction between 'compressible' (CPU, network) and 'incompressible' (memory, disk) resources dictates the aggressiveness of tuning, reflecting a clear trade-off between cost savings and service stability. This highlights the importance of designing for graceful degradation and preventing cascading failures in automated systems.
The system design prioritizes safety, acknowledging that automated changes in a production environment carry risks. By introducing a state machine for recommendations (CREATED, EVALUATING, APPLIED, REJECTED, MONITORED), immutable recommendation values post-creation, and configurable safety limits (e.g., 15% max memory reduction), Netflix builds trust in the automation. This iterative process of generating, verifying, and monitoring changes is fundamental to operating high-scale, self-optimizing distributed systems.