Leveraging LLMs for Automated Alert Investigation in Production Systems
This article details how Datadog fine-tuned a 9B parameter LLM (Qwen3.5-9B) to create a specialized agent for attributing changes responsible for production alerts. The system significantly reduces the cost of investigations while achieving high recall, demonstrating an effective application of AI/ML infrastructure to enhance operational efficiency in complex distributed systems.
Read original on Datadog BlogThe Challenge of Alert Investigation in Complex Systems
In large-scale distributed systems, identifying the root cause of production alerts is a significant challenge. Engineers often sift through vast amounts of telemetry data, logs, and change events to pinpoint the specific deployment, configuration change, or code push that triggered an issue. This manual process is time-consuming, expensive, and prone to human error, especially as system complexity grows. Automating this process requires a system capable of understanding context, correlating diverse data types, and reasoning about potential causal links.
Architecture of the LLM-Powered Investigation Agent
The core of the solution involves fine-tuning a large language model (LLM) to act as an expert system for change attribution. Instead of relying on a human-crafted rules engine, the LLM learns to identify relevant change events based on historical data. The input to the LLM includes alert context (metrics, logs, traces), recent deployments, configuration changes, and relevant infrastructure events. The output is a ranked list of potentially culpable changes.
- Data Ingestion and Preprocessing: Collects various signals like metrics, logs, traces, deployment records, and configuration changes from Datadog's extensive monitoring platform.
- Contextualization: Packages these diverse data points into a coherent prompt for the LLM, ensuring all relevant information is present and formatted appropriately.
- LLM Fine-tuning: Utilizes a smaller, specialized LLM (Qwen3.5-9B) rather than a very large general-purpose model, to reduce inference cost and latency while improving domain-specific accuracy through fine-tuning on incident data.
- Output Ranking and Evaluation: The LLM suggests possible causes, which are then ranked and presented to the user. Performance is evaluated against human expert investigations.
Why smaller, fine-tuned LLMs are beneficial for specific tasks
For specialized tasks like alert investigation, a smaller LLM fine-tuned on domain-specific data often outperforms larger, general-purpose models in terms of cost-efficiency, inference speed, and relevant accuracy. This is a critical design trade-off for integrating AI into production monitoring systems, balancing performance with operational expenditure.