Expedia's AI-Driven Service Telemetry Analyzer (STAR) for Incident Investigation
Expedia Group developed the Service Telemetry Analyzer (STAR), an AI-assisted observability platform designed to accelerate incident investigation. STAR combines operational metrics from services with Large Language Models (LLMs) through predefined diagnostic workflows, aiming to reduce the Time To Know (TTK) and Time To Recover (TTR) during production incidents. The system provides structured root cause assessments and recommended next steps, acting as an assistive tool rather than an autonomous agent.
Read original on InfoQ ArchitectureOverview of STAR's Architecture and Purpose
Expedia's Service Telemetry Analyzer (STAR) is an internal platform built to assist engineers in diagnosing production incidents faster. It integrates AI capabilities, specifically Large Language Models (LLMs), with traditional observability metrics. The core objective is to minimize Time To Know (TTK) and Time To Recover (TTR) by automating parts of the diagnostic process, enabling engineers to focus on validation and decision-making rather than manual data correlation.
System Design and Technology Stack
STAR is implemented as a FastAPI application, initially using background tasks. Due to I/O-bound operations involving telemetry retrieval from Datadog and requests to an internal generative AI gateway, Expedia migrated to a Celery-based asynchronous architecture with Redis serving as both the message broker and result backend. This asynchronous model allows STAR to concurrently process multiple analysis tasks while respecting external API rate limits, highlighting a key design decision for performance and resilience.
Design Choice: Asynchronous Processing
The transition to a Celery-Redis asynchronous architecture for STAR demonstrates a common pattern in distributed systems dealing with external API calls and I/O-bound workloads. By decoupling task execution from the main application thread, the system can handle higher concurrency, improve responsiveness, and gracefully manage rate limits and potential retries for external dependencies.
AI Workflow and Data Analysis
STAR's AI integration uses prompt chaining within a deterministic workflow. It collects standardized infrastructure telemetry from Kubernetes-based services and JVM applications, including metrics like throughput, latency, error rates, CPU/memory utilization, and JVM-specific data. This telemetry is analyzed using domain-specific prompts, consolidated into intermediate findings, and summarized into a final report suggesting potential root causes and next steps. Notably, the current implementation avoids advanced LLM features like function calling, RAG, or autonomous tool use, prioritizing consistent and predictable analyses via predefined workflows.
- Input Telemetry: Standardized metrics from Kubernetes and JVMs (throughput, latency, error rates, resource utilization).
- AI Processing: Predefined diagnostic workflows, prompt chaining with LLMs.
- Output: Structured root cause assessments and recommended actions.
- Current Limitations: No RAG, function calling, or autonomous tool use; focus on deterministic analysis.
The platform supports various use cases, including production incident investigation, post-incident analysis, Kubernetes troubleshooting, and JVM memory diagnostics. Future enhancements include incorporating service dependency information, additional operational metadata, and conversational interfaces.