Designing Intelligent Monitoring Systems: Beyond Simple Thresholds

Traditional monitoring engines often fall into a common pitfall: alerting prematurely on new services or endpoints. This occurs when a monitor triggers an alert based on initial failures during a service's startup phase, DNS propagation, or certificate issuance, before the service has had a chance to stabilize or even become fully operational. This leads to false positives and alert fatigue, eroding trust in the monitoring system.

The "Awakening" State for New Monitors

The article introduces the concept of an AWAKENING state for new monitors. Instead of relying on a fixed time delay (like `new_group_delay` in Datadog), it advocates for a count-based approach. A monitor remains in AWAKENING until it has successfully processed a configured number of pulses (e.g., three pulses). During this state, even if all pulses are failures, no alerts are fired, acknowledging that the system is still learning about the monitored target.

State-Driven Monitoring Logic

private ReliabilityState evaluateReliability(
    double healthIndex,
    long totalCount,
    long pulsesSinceActivation) {
    if (pulsesSinceActivation < monitoringProperties.getAwakeningPulseThreshold()) {
        return ReliabilityState.AWAKENING;
    }
    if (healthIndex >= monitoringProperties.getHealthyThreshold()) {
        return ReliabilityState.HEALTHY;
    }
    return ReliabilityState.DEGRADED;
}

The monitoring system incorporates a state machine with states like AWAKENING, HEALTHY, and DEGRADED. New monitors begin in AWAKENING. Once the `awakeningPulseThreshold` is met, the monitor transitions to HEALTHY or DEGRADED based on its accumulated health index. Crucially, the alert pipeline is guarded to ignore monitors in the AWAKENING state, preventing unnecessary notifications.