Migrating a High-Volume Metrics Pipeline to OpenTelemetry and Prometheus

Migrating a production-grade, high-volume metrics pipeline requires careful architectural planning, especially when transitioning between different instrumentation protocols and storage backends. Airbnb faced the challenge of moving from a StatsD-based system to one leveraging OpenTelemetry Protocol (OTLP) for instrumentation and Prometheus for storage, while ensuring scalability, correctness, and cost-efficiency.

Instrumentation and Dual-Write Migration Strategy

The initial architecture used StatsD with Veneur sidecars. The migration involved adopting OTLP as the recommended protocol for internal services and Prometheus for OSS workloads, with StatsD remaining for legacy applications. A dual-write approach was critical: a shared metrics library was updated to emit metrics simultaneously in both StatsD (for the old pipeline) and OTLP (for the new OpenTelemetry Collector). This allowed for a broad, low-friction rollout and validation against real data.

A key challenge with OTLP involved performance regressions in high-volume services due to high-cardinality metrics, leading to memory pressure and increased GC activity. This was mitigated by configuring delta temporality for these specific services, which reduces in-process memory burden by not retaining full state between exports. The trade-off is that unexpected failures can cause visible data gaps.

Streaming Aggregation with vmagent

To manage costs and scale, a streaming aggregation layer was essential. After evaluating and rejecting options like continuing with a custom Veneur fork, Prometheus recording rules (due to storing raw data first), m3aggregator (complexity), and OTel Collector/Vector (lack of scaling/features), Airbnb chose vmagent from VictoriaMetrics.

• vmagent supports streaming aggregation for Prometheus metrics.
• It offers sharding capabilities for horizontal scaling.
• Features a small codebase for ease of understanding and modification.
• Documentation is user-friendly and setup is straightforward.

The vmagent architecture consists of two layers: routers (stateless, shard metrics by consistent hashing of labels) and aggregators (perform aggregation and maintain in-memory state). This scalable design allowed Airbnb to ingest over 100 million samples per second, reducing costs significantly and providing a centralized point for metric transformations.

Solving Sparse Counter Undercounting with Zero Injection

A critical issue emerged with sparse counters in the Prometheus pipeline: queries using `rate()` or `increase()` consistently undercounted compared to the StatsD system. This occurred because Prometheus's cumulative counters, when reset (e.g., due to pod restarts) before incrementing again, lose the increment before `rate()` can derive a delta. This was particularly problematic for low-rate, high-dimensionality counters.