Netflix's Evolution to Open Source Flink Autoscaler for Large-Scale Streaming
Netflix transitioned to the open-source Apache Flink Autoscaler for its 30,000+ streaming jobs to address limitations of its previous cluster-level scaling approach. This move, driven by the complexities of stateful pipelines, allowed for more granular, operator-level parallelism adjustments, leading to significant cost savings and improved resource utilization. The new system leverages Temporal workflows for isolated autoscaling decisions and integrates specific Flink enhancements.
Read original on InfoQ CloudThe Challenge of Scaling Stateful Streaming Jobs
Netflix's initial Flink autoscaler, developed in 2019, operated at a cluster level. While effective for simple pipelines, this approach proved insufficient for complex, stateful streaming jobs with diverse processing requirements across different operators. In a cluster-level scaling model, all operators in a job receive the same scaling decision, which is suboptimal for pipelines involving branches, joins, or terabytes of state, where individual components may have vastly different resource needs.
Transition to Apache Flink Autoscaler: Operator-Level Parallelism
The move to the open-source Apache Flink Autoscaler brought a crucial shift: the ability to reason about and adjust parallelism at the individual operator level. This new approach uses metrics exposed by the running job to estimate each operator's true processing rate, then walks the job graph to calculate and apply the required parallelism for individual vertices. This granular control allows for more efficient resource allocation, addressing the specific needs of heterogeneous streaming jobs.
Key Distinction
Unlike generic event-driven autoscalers (like KEDA), Flink's autoscaler has deep knowledge of the internal dataflow graph and operator capacity, enabling more intelligent and optimized scaling decisions.
Architectural Integration and Enhancements
- Internal Control Plane Integration: Netflix integrated the Flink Autoscaler with its existing internal control plane, opting not to deploy it directly via the Flink Kubernetes Operator.
- Temporal Workflows: A Spring Boot service utilizes Temporal workflows to isolate and manage autoscaling decisions for individual jobs, ensuring robust and independent scaling actions.
- Customizations for Scale: Netflix made specific modifications to Flink, including improved JobManager metric collection for up to 3,000 subtasks, server-side metric filtering, preservation of forward connected subgraphs during scaling, and handling of sink backpressure.
The adoption of the open-source Flink Autoscaler has yielded substantial benefits, with one team reporting a 58% reduction in annualized Flink compute expenditure, translating to approximately $1.1 million in annual savings. This demonstrates the critical importance of a well-designed autoscaling solution for large-scale, cost-sensitive distributed streaming platforms.