Meta's DrP: Engineering Automated Incident Investigation

The Challenge with Manual Incident Investigation

As engineering organizations scale, manual incident investigation becomes unsustainable. Key issues include knowledge being trapped in individual engineers, making it inaccessible when that person is unavailable. Furthermore, rapidly evolving microservice architectures quickly render static runbooks and documentation obsolete. While one-off scripts can help, they often lack systematic testing, cross-service boundary capabilities, and clear ownership, eventually becoming another form of tribal knowledge.

DrP's Architectural Philosophy: Debugging as Software

Meta's DrP platform addresses these challenges by treating incident investigation as a software engineering problem. This involves codifying debugging workflows into 'analyzers' that are subject to the same development rigor as any other production code: code review, CI/CD, and automated testing. This shifts the paradigm from reactive, human-centric debugging to proactive, automated, and continuously improving investigative software.

• Analyzers: Programmatic workflows defined using an SDK, specifying data to pull, anomalies to detect, and decision trees to follow. They output structured, machine-readable findings.
• Code Review & Testing: Analyzers undergo code review and automated backtesting to verify their effectiveness against past incidents before deployment.
• CI/CD Integration: Changes to underlying systems prompt updates to analyzers through standard development workflows, ensuring their relevance.

Key System Design Capabilities of DrP

DrP's power stems from its ability to integrate and automate investigation across a complex microservices environment, going beyond isolated scripts.

• Cross-Service Chaining: Analyzers can chain together, passing context and invoking other analyzers across service boundaries. For example, an API analyzer detecting an issue can trigger a downstream storage service analyzer to find the root cause.
• Alert Lifecycle Integration: DrP integrates directly with alert systems, auto-triggering analyzers when an alert fires. The diagnosis is then surfaced alongside the alert, providing immediate context to on-call engineers.
• Shared Libraries: The SDK provides common investigation patterns like anomaly detection, time series correlation (e.g., correlating metric spikes with deploys), and dimension analysis (slicing metrics by region/device) to prevent reinvention.
• Feedback Loops & Organizational Learning: A post-processing system can automate remediation tasks (e.g., creating revert tasks or filing bugs). DrP Insights also analyzes investigation outputs to identify common root causes, fostering continuous organizational learning.