Rethinking Kubernetes Scaling with Scale-from-Zero for Cost Optimization

The persistent challenge in infrastructure management is the cost associated with over-provisioning. Many organizations provision for peak load or worst-case scenarios, leading to significant idle capacity and wasted budget. This is particularly pronounced with expensive resources like GPUs and in complex hybrid/multi-cloud Kubernetes deployments where the "always-on" default compounds costs across numerous clusters and teams. The article introduces "scale-from-zero" as a paradigm shift to align infrastructure capacity directly with actual demand, rather than maintaining expensive hot standbys or permanently provisioned node pools.

The Problem: Idle Infrastructure and Hot Standby Costs

Traditional infrastructure provisioning often results in substantial idle resources. Examples include GPU node pools for monthly batch jobs, standby data centers for disaster recovery, and CI/CD runners waiting for pipeline triggers. While seemingly reasonable decisions at a micro-level, these add up to a major structural cost at enterprise scale. The "hot standby" model, in particular, is an expensive insurance policy that often sits idle, waiting for an event that may never occur.

Scale-from-Zero: Architecture and Workflow

Scale-from-zero aims to make node pools with zero running nodes visible and schedulable to the Kubernetes scheduler. This is achieved through capacity annotation, where metadata is attached to node pool definitions, informing the scheduler about the potential CPU, memory, and GPU resources that would exist if a node were running. This allows the scheduler to make placement decisions and queue pods against an empty pool, triggering an autoscaler to provision resources only when needed.

1. Workload creation: A pod with specific resource requirements is created.
2. Pending state: The scheduler finds no existing nodes satisfying requirements, leaving the pod pending.
3. Autoscaler evaluation: Cluster Autoscaler identifies a compatible scale-from-zero node pool based on capacity annotations.
4. Infrastructure request: Cluster API updates the MachineDeployment to request additional infrastructure.
5. VM provisioning: CAPX (Cluster API Provider for Nutanix) provisions the virtual machine, configures networking, and bootstraps Kubernetes components.
6. Node join: The new node joins the cluster and reports "Ready" status.
7. Workload binding: Kubernetes binds the pending workload to the new node.
8. Scale-down: Once demand subsides, the autoscaler removes the node, returning the pool to zero capacity.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: gpu-node-pool
  annotations:
    capacity.annotations.kubernetes.io/gpu: "1"
    capacity.annotations.kubernetes.io/cpu: "8"
    capacity.annotations.kubernetes.io/memory: "64Gi"
spec:
  replicas: 0 # Starts at zero
  selector:
    matchLabels:
      node-pool: gpu
  template:
    spec:
      metadata:
        labels:
          node-pool: gpu
      spec:
        taints:
        - key: "nvidia.com/gpu"
          effect: "NoSchedule"
        ...