Optimizing AI Inference on Kubernetes: Cost and Infrastructure Challenges
This article discusses the evolving landscape of running AI inference workloads on Kubernetes, highlighting the challenges related to cost management and resource utilization. It presents a case study from China Merchants Bank on unifying AI compute resources with Kubernetes and explores the need for Kubernetes' resource model to evolve to better account for the nuanced economics of large-scale AI inference, particularly the "cost per token" metric.
Read original on The New StackThe integration of AI inference workloads into cloud-native infrastructures, primarily Kubernetes, is gaining significant traction. As the focus shifts from AI model training to efficient model serving in production, organizations are exploring how Kubernetes can manage and scale these demanding workloads. However, the article points out a critical challenge: accurately counting the real cost of AI inference on Kubernetes, which extends beyond traditional resource metrics like CPU and memory.
Case Study: China Merchants Bank's AI Infrastructure
China Merchants Bank successfully unified 99% of its nearly 10,000 heterogeneous accelerator cards for AI computation using a Kubernetes-based cloud-native architecture. This implementation leveraged several CNCF projects to achieve significant improvements in resource utilization and cost efficiency:
- Kueue: For efficient job queueing and quota management of AI workloads.
- KEDA: For event-driven auto-scaling, allowing resources to scale based on demand.
- Prometheus: For robust systems monitoring and metric collection.
- HAMi: To facilitate sharing accelerator capacity across multiple Kubernetes workloads.
- Fluid: For accelerating access to large datasets, crucial for AI inference.
Impact of Cloud-Native AI Infrastructure
By adopting this architecture, China Merchants Bank increased average utilization of its AI compute resources from 35% to over 60% and reduced the cost of processing 1 million tokens by 60%. This demonstrates the potential of cloud-native approaches to optimize AI inference even in highly regulated environments.
The Challenge of AI Inference Economics on Kubernetes
While Kubernetes provides a powerful orchestration layer, its current resource model may not fully capture the nuanced economics of large-scale AI inference, where the primary metric is "cost per token." This cost is influenced by factors Kubernetes was not originally designed to manage, such as request mix, KV cache occupancy, the balance of prefill and decode operations, and granular memory/bandwidth consumption within accelerators post-pod-launch.
Industry experts suggest that for Kubernetes to remain an optimal platform for AI inference, its resource model needs to evolve. This evolution would likely involve a new scheduling and memory layer capable of computing the actual cost of serving a token. Such a layer would enable more informed, cost-based decisions for scheduling and serving inference workloads, transforming Kubernetes from a potential "tax" on inference economics into a more optimized solution.