Netflix's Evolution of Flink Autoscaling: From Homegrown to Open Source

Netflix operates over 30,000 Apache Flink jobs, necessitating robust autoscaling to manage fluctuating loads efficiently and cost-effectively. Static provisioning for peak load is wasteful, while provisioning for average load leads to performance lags during surges. A critical challenge is that Flink scaling involves taking a savepoint, stopping, and restarting the job, which can take several minutes for large stateful jobs, making rapid scaling difficult and impacting availability. This context frames the fundamental requirement for intelligent and reliable autoscaling.

The Homegrown External Autoscaler: Initial Approach and Limitations

Netflix's first autoscaler, built around 2019, operated externally, consuming cluster-level metrics from Atlas (CPU, network, Kafka lag, etc.). It functioned as a streaming job on Mantis, scaling out easily to handle more Flink jobs. While this system reduced resource usage by 25-45% for simple pipelines, it had significant limitations:

• Coarse-grained Scaling: It reasoned about the entire cluster through container metrics and scaled only the total TaskManager count, moving all operators together. This was unsuitable for complex, multi-operator, stateful DAGs.
• Limited Visibility: As an external system, it couldn't see internal Flink job states (e.g., specific operator backpressure) beyond aggregate container metrics. Jobs could be degraded without showing high CPU utilization.
• Metric Dependency: Reliance on external telemetry systems meant that changes or inaccuracies in metric reporting could silently break the autoscaler's effectiveness.

Adopting the Apache Flink Autoscaler: Internal Reasoning

The open-source Apache Flink Autoscaler offers a more sophisticated approach by reasoning *from inside* the Flink job. Its core idea is to estimate each operator's True Processing Rate (TPR) – the throughput an operator could sustain if fully busy. By dividing observed throughput by the 'busy fraction' (time spent doing actual work), it extrapolates capacity to full utilization. This allows it to compute the required parallelism for *each operator vertex* in the job graph, preventing single-operator bottlenecks rather than resizing the entire cluster uniformly.

Scaling the Open-Source Autoscaler to Netflix's Needs

While the community provided the core logic, integrating and scaling the OSS autoscaler to Netflix's environment required significant engineering, particularly since Netflix's Flink platform uses its own control plane, not the Kubernetes Operator for Flink. They leveraged the autoscaler's refactored generic interfaces (context, state store, event handler, realizer) to plug it into their internal ecosystem.

• Temporal Workflow Orchestration: A Spring Boot application orchestrates scaling using Temporal, a durable workflow engine. A long-running workflow is created *per Flink job* to fetch metrics, run the scaling algorithm, and apply decisions. This workflow-per-job model isolates problematic jobs, preventing cascading failures across the entire fleet.
• Metric Collection Optimizations: For high-parallelism jobs, fetching metrics from the Flink JobManager was a bottleneck. Netflix optimized Flink's runtime by caching transient metric names, adding server-side filtering, and contributing improvements upstream to handle jobs with up to 3,000 subtasks.
• Flink Fork Enhancements: To ensure correctness and safety, Netflix implemented changes in their internal Flink fork, including detecting and scaling forward-chained subgraphs as a unit and adding async-sink backpressure detection to prevent scaling into an overloaded sink.
• Safety Checks: A realizer component performs pre-actuation safety checks, such as refusing to scale down jobs in regions undergoing failover or verifying sufficient disk space for checkpoint state.

Impact and Future Directions

The OSS-based autoscaler achieved significant cost savings (e.g., 58% reduction, $1.1M annually for one team) by dynamically adapting to load, continuous optimization, and enabling better bin-packing. Netflix also tuned the target utilization (0.45 vs. default 0.7) to prioritize stability over aggressive efficiency for large stateful jobs, accepting marginal cost for fewer, calmer rescales. Future plans include leveraging Flink 2's disaggregated state architecture to reduce state recovery bottlenecks during scaling events, further improving scaling performance and cost efficiency.