Netflix Migrates Batch Workloads to Kueue for Cloud-Native Job Queueing
Netflix successfully migrated its internal Compute Managed Batch (CMB) system to Kueue, an open-source Kubernetes-native job queuing system. This move allowed Netflix to leverage cloud-native features and innovation from the Kubernetes ecosystem while maintaining API parity for a seamless user experience. The migration highlights architectural considerations for replacing homegrown solutions with industry-standard, cloud-native alternatives.
Read original on InfoQ ArchitectureNetflix, known for its extensive use of cloud infrastructure, faced the challenge of maintaining and evolving its homegrown batch job execution system, Compute Managed Batch (CMB), built on top of its container platform, Titus. CMB managed capacity using tenant hierarchies and federated workloads across multiple Kubernetes clusters (cells). Over time, the capabilities of CMB became redundant with features evolving rapidly within the open-source Kubernetes ecosystem, making in-house development costly and integration less efficient.
The Shift to Kueue
The decision was made to migrate to Kueue, a Kubernetes-native solution for batch job management. Kueue offers a comprehensive set of features that align with modern distributed system requirements, including priority-based queuing, advanced resource management, multi-cluster and topology-aware scheduling, and robust observability. This strategic shift allowed Netflix to offload the burden of maintaining a bespoke system and benefit from community-driven innovation.
Key Migration Strategy
A crucial aspect of Netflix's migration strategy was maintaining API parity with the existing system. This approach significantly derisked the project by ensuring a seamless experience for existing users and allowing for a gradual, tenant-bound rollout with easy rollback capabilities.
Architectural Mapping and Benefits
The migration involved mapping Netflix's internal tenant model (tenants and leaf tenants) to Kueue's constructs (Cohorts, ClusterQueue, and LocalQueue resources). Resource flavors and nominal quotas in Kueue were used to configure capacity requirements, directly importing existing configurations from CMB. This careful mapping ensured that the new system could handle the required container launch rates and throughput from the outset. Furthermore, leveraging features like preemption-based fair sharing in Kueue significantly improved average resource utilization by enabling idle capacity lending among tenants while maintaining reservation semantics.
- Reduced Maintenance Overhead: Eliminated the need for Netflix to maintain a complex in-house batch processing system.
- Access to Cloud-Native Features: Gained advanced features like multi-cluster scheduling and comprehensive observability inherently supported by Kueue.
- Improved Resource Utilization: Enhanced efficiency through preemption-based fair sharing, allowing dynamic allocation of idle capacity.
- Seamless User Experience: API parity ensured a transparent migration for end-users, minimizing disruption.
Lessons learned included the importance of tackling the most complex use cases early to build confidence and fine-tuning performance through load testing in non-production environments. This case study demonstrates a successful transition from a monolithic or highly customized internal system to a cloud-native, open-source alternative, highlighting best practices for large-scale migrations in distributed environments.