Ensuring LLM Readiness in Kubernetes: Beyond Basic Health Checks

Deploying Large Language Models (LLMs) in Kubernetes presents unique challenges beyond traditional microservices. A common pitfall is relying solely on basic HTTP readiness probes, which can signal a pod as healthy even if the LLM within it is not fully loaded or ready to serve inference requests. This discrepancy can lead to service degradation, increased latency for initial requests, and an unreliable user experience, especially during deployments or scaling events.

The 'Ready-to-Inference' Gap

The article demonstrates through experiments that there's a significant time gap between when Kubernetes reports an LLM serving pod as 'Ready' (T1) and when it can successfully complete its first inference request (T4). This 'Ready -> inference gap' can range from several seconds to over ten seconds, depending on the model size and environment. This gap arises because 'Ready' typically only confirms the process is running and an API endpoint is reachable, not that the model is loaded into memory and prepared for computation.

Why Basic Readiness Probes Fall Short for LLMs

• Process is up, model is gone: Using ephemeral storage like `emptyDir` for models can lead to a running container without the actual model artifact after a pod restart.
• Model on disk, not loaded: The model file might exist on the container's disk, but it still needs to be loaded into memory. This loading process can be time-consuming and memory-intensive.
• Resource contention: Overlapping model residency, where multiple models try to load or reside in memory concurrently within resource constraints (e.g., cgroup limits), can cause failures (e.g., OOM kills) that a simple HTTP probe wouldn't detect.
• Cold vs. warm cache: Host-level caching can mask true model load times during recovery tests, making results appear better than they would be on a truly cold node.

Implementing Robust LLM Readiness Checks

To accurately determine LLM readiness, the probe must verify that the model is not only present but also fully loaded and capable of serving requests. The proposed solution involves an `exec` probe that performs a minimal inference request against the LLM's API. This ensures the entire inference pipeline, including model loading and actual computation, is functional before the pod is considered 'Ready' by Kubernetes.

readinessProbe:
  exec:
    command:
      - sh
      - -c
      - |
        curl -sf -X POST http://localhost:11434/api/generate \
        -H 'Content-Type: application/json' \
        -d '{"model":"llama3.2:1b","prompt":"ping","stream":false}' \
        | grep -q '"done":true'
  periodSeconds: 2
  failureThreshold: 1