Netflix's Evolution to Open Source Flink Autoscaler for 30,000+ Streaming Jobs
Netflix migrated from a custom cluster-level Apache Flink autoscaler to the open-source Flink Autoscaler to optimize resource utilization for its complex, stateful streaming pipelines. This shift allowed for fine-grained, operator-level parallelism adjustments based on true processing rates, significantly reducing compute expenditure and improving efficiency for over 30,000 jobs across multiple AWS regions. The new approach leverages internal dataflow graph analysis over generic external metrics, enabling more intelligent scaling decisions.
Read original on InfoQ ArchitectureNetflix, a major adopter of Apache Flink since 2017, initially developed a custom cluster-level autoscaler to manage its vast streaming workload. While effective for basic resource reduction, this approach proved insufficient for increasingly complex, stateful pipelines featuring branches, joins, and terabytes of state. The limitation stemmed from making scaling decisions at the cluster level, forcing all operators within a job to share the same parallelism, even if their processing requirements differed significantly.
The Challenge with Cluster-Level Autoscaling
The original autoscaler used aggregated telemetry like CPU, network utilization, Kafka lag, and data rates from Atlas and Mantis to adjust TaskManager counts. This coarse-grained approach led to suboptimal resource allocation. For example, in a stateful job with varying operator demands, some operators might be under-provisioned, leading to backpressure and latency, while others might be over-provisioned, wasting compute resources. This highlighted the need for a more granular, operator-aware scaling mechanism.
Key Learning: Granularity in Autoscaling
When designing autoscaling solutions for complex distributed systems, especially those with heterogeneous workloads or varying component demands (like streaming jobs with diverse operators), a finer-grained scaling unit often leads to better resource utilization and performance. Cluster-level scaling might be simpler to implement but can be inefficient for intricate dataflow graphs.
Adopting the Open-Source Flink Autoscaler
Netflix shifted to the open-source Apache Flink Autoscaler, based on FLIP-271 and research from the DS2 project. This new approach determines each operator's required parallelism by analyzing metrics like throughput and busy time to estimate its "True Processing Rate." It then walks the job graph, calculating individual vertex parallelism requirements. This contrasts with generic autoscalers like KEDA, which react to external metrics; Flink's autoscaler understands the internal dataflow and operator capacity.
- True Processing Rate: Estimates an operator's real capacity from its throughput and busy time, allowing for accurate parallelism calculation.
- Job Graph Analysis: Iterates through the Flink job graph to determine the optimal parallelism for each individual operator vertex.
- Internal Control Plane Integration: Netflix integrated the autoscaler with its internal Spring Boot service, using Temporal workflows to manage isolated autoscaling decisions per job.
- Customizations for Scale: Modifications included supporting up to 3,000 subtasks, server-side metric filtering, preserving forward-connected subgraphs during scaling, and handling sink backpressure.
This migration resulted in significant cost savings, with one team reporting a 58% reduction in annualized Flink compute expenditure, equating to approximately $1.1 million annually. Netflix targets a 0.45 utilization, lower than Flink's community default of 0.7, to mitigate aggressive rescaling of large stateful jobs, acknowledging the cost of state recovery during rescaling. Future plans include exploring Flink 2's disaggregated state architecture to further address this challenge.