Implementing Online and Offline Rule Evaluation with Jev in Datadog
This article discusses Datadog's approach to implementing a unified rule evaluation system using TypeSafe's Jev library. It highlights the architecture for applying a single rubric across both real-time, online evaluations of live telemetry spans and offline evaluations within experimental analysis, focusing on consistency and efficiency in observability and alerting systems.
Read original on Datadog BlogDatadog leverages TypeSafe's Jev, an embedded expression language evaluator, to create a consistent and efficient mechanism for evaluating rules. This system is crucial for their observability platform, enabling them to apply the same evaluation logic whether they are processing live telemetry data or analyzing historical data for experiments.
Unified Evaluation Rubric
A core architectural decision is to use a single rubric (set of rules) for both online and offline evaluations. This reduces the cognitive load for engineers, prevents discrepancies between development/testing and production, and simplifies maintenance. The challenge lies in designing an evaluation engine that can perform efficiently in real-time, high-throughput scenarios and also be flexible enough for batch processing and complex analytical queries.
Online Evaluation on Live Spans
For online evaluation, Jev is integrated directly into the data processing pipeline of the Datadog Agent. As live telemetry spans (e.g., traces, metrics, logs) flow through the system, Jev rules are applied to determine if certain conditions are met, triggering alerts or further processing. This requires a low-latency, high-throughput evaluation capability, often involving in-memory processing and optimized rule matching.
Offline Evaluation in Experiments
Offline evaluations are used within Datadog's experimentation platform, where historical data is re-evaluated against the same rules. This helps in validating new detection logic, understanding the impact of rule changes, or analyzing past incidents with new insights. The system must be capable of processing large volumes of archived data efficiently, potentially in a distributed batch processing environment like Apache Flink or Spark.
Design Principle: Consistency
Using a unified rule engine for both real-time and historical data ensures that the logic applied during incident detection is identical to the logic used for post-mortem analysis or experimentation. This consistency is vital for building reliable and trustworthy observability platforms.