Designing Pensive AI: An LLM-Assisted Failure Diagnosis System at Netflix

Netflix operates millions of data and ML workloads daily, leading to approximately 100,000 failures on peak days. Diagnosing these failures is complex, requiring engineers to sift through extensive logs to determine root causes and appropriate actions. The original Pensive system used regular expressions and hand-authored Java rules, achieving about 50% self-service resolution. However, its coverage was limited by the manual effort required to create and maintain rules for evolving failure patterns. This motivated the development of Pensive AI, which integrates AI capabilities to broaden coverage without sacrificing the reliability of deterministic methods.

Pensive AI Architecture Overview

Pensive AI's design wraps AI functionalities around existing deterministic processes, aiming for broader coverage, low latency, controlled AI costs, and easier maintenance. The system features two complementary paths: Always-on error extraction and On-demand debugging. Both share log sources, an LLM gateway, tracing, and a cache, with archived logs feeding offline evaluations.

• Error Extraction: Improves existing capabilities by quickly identifying potential root causes in logs for every failed step. It's designed for low latency and cost, offering a fast initial look.
• On-Demand Debugging: A new capability providing deeper analysis by examining symptoms across logs, leveraging the LLM's knowledge and reasoning. This is invoked by the user when initial extraction is insufficient and has slightly higher latency and cost.

Error Extraction Design

To address the limitations of rule-based extraction and the slowness of direct LLM transcription, Pensive AI adopted a novel approach. Instead of having the LLM transcribe long stack traces, logs are line-numbered and presented to the model as `{lineNumber, text}` records. The LLM identifies only the start and end line numbers of an error span. Pensive then reconstructs the full error text from its own log copy. This significantly reduces output token generation, leading to lower cost and better latency, ensuring the displayed content directly reflects the original log and mitigates hallucinations.

On-Demand Debugging and Caching

Full AI-based debugging is run on-demand to control costs, as most failed steps do not require deep analysis. A crucial design decision was to treat the failed step attempt (not the user) as the unit of work. This enables consistent diagnoses for all investigators and, critically, allows for caching of debugging responses. Subsequent requests for the same failed attempt become cache reads, significantly reducing latency and further lowering AI inference costs. A conditional write mechanism handles concurrent requests, with only one caller initiating generation while others wait for the shared result, and responses are streamed to the client for a responsive user experience.

Data Model Evolution and Rollout Strategy

Migrating from a flattened error string to a list of errors with deep links presented a challenge. Pensive AI introduced the new data model alongside the legacy model, populating both. This allowed clients like Data Portal to prefer AI errors while falling back to legacy values, enabling a gradual, uncoordinated rollout. This strategy prevents breaking existing clients and gives teams flexibility to adopt the new capabilities at their own pace, demonstrating a robust approach to evolving data contracts in a large-scale system.