Building an Operational Triage Dashboard for Kubernetes
This article details the journey of evolving a simple Bash script into OpsCart Watcher, an open-source operational triage dashboard for Kubernetes. It focuses on the architectural shift from merely detecting failures to prioritizing and contextualizing operational issues, addressing the challenge of "where to look" in complex distributed environments. The evolution highlights the importance of operational memory and deterministic assessments for effective incident response.
Read original on DZone MicroservicesThe Challenge of Kubernetes Observability
While traditional monitoring tools like Prometheus and Grafana excel at reporting System Level Objectives (SLOs) and overall cluster health, they often fall short in guiding operators to immediate, actionable problems. The article points out that having extensive visibility (metrics, logs, kubectl access) doesn't equate to knowing "what deserves attention right now?" This gap leads to overlooked issues, like a pod stuck in CrashLoopBackOff for days, yet appearing healthy on a metrics dashboard due to satisfying replica counts between crashes. The core problem is prioritization, not lack of data.
From Detection to Triage and Prioritization
The OpsCart Watcher evolved to address this by moving beyond simple fault detection. It aggregates similar issues (e.g., multiple crash-looping pods across namespaces) into a single operational problem and assigns a severity. This approach transforms a long list of individual resource states into a ranked list of critical operational problems, each accompanied by a severity, location, and the precise `kubectl` command to begin investigation. This is crucial for guiding engineers in complex, distributed systems.
Incident Score
The introduction of an "Incident Score" (0-100) across reliability, security, and waste provides a high-level metric for tracking operational health. This score not only quantifies the collective state of problems but also serves as a communication tool, making the impact of operational work tangible for management.
Adding Operational Memory
A significant architectural improvement was the introduction of an embedded database to store scan history. This "operational memory" allows the dashboard to track changes over time, distinguishing between new incidents and long-standing problems. This shift from answering "what is broken?" to "what changed?" is vital for effective incident response, as the urgency and approach to an issue depend heavily on its recency and trend.
Incident Score: 41/100 (Degraded)
Top 5 Things to Fix:
1. 4 pods crash-looping CRITICAL 4 pods
2. 3 image_pull_backoff issues CRITICAL 3 items
3. 1 privileged_container issue CRITICAL 1 item
4. 1 namespace missing NetworkPolicy HIGH 1 ns
5. 3 orphaned PVCs wasting money MEDIUM 80 GBDeterministic Investigation Guidance
The tool provides an "Investigation page" that offers rules-based assessments and recommended steps for debugging. This deterministic approach, based on patterns like restart counts and failure stability, guides engineers on where to start (e.g., "Check previous container logs"). This is a layer of actionable intelligence that goes beyond raw data provided by `kubectl` or other dashboards.