Instacart's Blueberry: An AI-Powered Incident Response System for On-Call Engineers
Instacart developed Blueberry, an AI-assisted system to accelerate incident investigation for on-call engineers. It leverages multiple AI agents, operational data, and historical incident knowledge to provide context, generate root cause hypotheses, and support debugging within existing Slack workflows, significantly reducing time spent on initial context gathering.
Read original on InfoQ ArchitectureOverview of Blueberry's Architecture
Blueberry is designed as an AI-powered assistant for on-call engineers, aiming to streamline incident response by providing rapid context and initial diagnostic hypotheses. It operates within existing incident workflows, primarily integrated with Slack, to avoid disrupting engineers' established collaboration channels. The system's core value lies in its ability to synthesize information quickly, addressing the common challenge of engineers spending significant time gathering context before actual diagnosis can begin.
Key Architectural Components and Principles
- Multiple AI Agents: Blueberry orchestrates approximately 10 subagents in parallel when an alert is triggered. These agents work concurrently to gather information and formulate a hypothesis.
- Operational Data Integration: Instead of relying solely on general-purpose Large Language Models (LLMs), Blueberry connects to Instacart's internal operational knowledge bases. This includes incident history, service ownership data, deployment records, logs, metrics, and other debugging signals. This grounding in specific, reliable context is crucial for accurate recommendations.
- Tool-Aware Harness: The system incorporates a 'durable, tool-aware harness' that allows agents to retrieve information from connected systems while maintaining the state of the investigation. This ensures that the AI's recommendations are grounded in real-time and historical operational context.
- Human-in-the-Loop: A critical design decision is that Blueberry assists engineers rather than automating production changes. Engineers remain responsible for diagnosis, mitigation decisions, and remediation, with Blueberry acting as a force multiplier for information gathering and hypothesis generation.
Addressing Challenges in AI for Operations
The "Grounding" Problem
A primary challenge in applying AI to production operations is ensuring that the AI's recommendations are reliable and contextually accurate. Blueberry addresses this by tightly integrating with organization-specific data and using a tool-aware approach, effectively 'grounding' its AI agents in real operational knowledge rather than generic LLM outputs. This significantly improved diagnostic accuracy from the mid-60% range to the high 90% range.