Scalable HTTP/3 Network Probing with Prometheus Blackbox Exporter

The Challenge: HTTP/3 Observability Gap

When Slack began rolling out HTTP/3 support, built on the QUIC transport protocol (which uses UDP instead of TCP), they encountered a critical observability gap. Their existing commercial SaaS offerings and custom internal tools, including the Prometheus Blackbox Exporter, lacked the capability to probe HTTP/3 endpoints. This meant they couldn't get client-side visibility for metrics like round-trip times or regressions, essential for monitoring the health and performance of their new infrastructure.

Solution: Extending Prometheus Blackbox Exporter

To address this, Slack chose to extend their existing Prometheus Blackbox Exporter (BBE). The core of the solution involved integrating a QUIC-capable HTTP client, specifically `quic-go` due to its widespread adoption and Go language support, into BBE's HTTP client structure. This allowed BBE to send probes over HTTP/3 and collect metrics from these endpoints.

http3Transport := &http3.Transport{
    TLSClientConfig: tlsConfig,
    QUICConfig: &quic.Config{},
}
client = &http.Client{
    Transport: http3Transport,
}

• Unified Monitoring: A single pane of glass in Grafana for HTTP/1.1, HTTP/2, and HTTP/3 metrics, improving correlation and debugging.
• Reliable Alerts: Ability to create more accurate and reliable alerts based on HTTP/3 endpoint health and performance.
• Open Source Contribution: The HTTP/3 probing functionality was open-sourced, benefiting the broader Prometheus community and future-proofing observability for QUIC adoption.

Key Takeaways for System Design

1. Monitor First, Migrate Second: Emphasizes the critical importance of robust observability before migrating to new protocols or infrastructure. Proving the new system's stability and performance is crucial for long-term investment.
2. Open Source Contributions Pay Dividends: Contributing to open-source projects not only benefits the community by filling critical gaps (like HTTP/3 support in monitoring tools) but also ensures that the organization's specific needs are met and can be maintained within a shared ecosystem.
3. Architect for Extensibility: The ability to extend an existing tool like Prometheus Blackbox Exporter with new transport protocol support demonstrates the value of designing systems with clear architectural patterns that allow for future adaptations and new feature integration without a complete overhaul.