Integrating OpenTelemetry with Datadog for Observability

The Challenge of Vendor Lock-in in Observability

Many organizations face a dilemma when choosing observability tools: tightly integrating with a vendor provides rich features but can lead to vendor lock-in, making it difficult to switch or integrate other tools. OpenTelemetry (OTel) addresses this by providing vendor-neutral APIs, SDKs, and data specifications for collecting traces, metrics, and logs. This decouples telemetry generation from backend processing, offering significant architectural flexibility.

OpenTelemetry as an Observability Data Standard

OpenTelemetry defines a standardized way to instrument applications and collect telemetry data. This means that application code doesn't need to be rewritten if the observability backend changes. The OTel Collector, a proxy that receives, processes, and exports telemetry data, plays a crucial role in this architecture. It can be deployed as an agent, a gateway, or even within Kubernetes sidecars to aggregate and transform data before sending it to various observability platforms.

Integrating OTel with Datadog

The integration pattern involves instrumenting applications with OpenTelemetry SDKs, sending data to an OpenTelemetry Collector, and then configuring the collector to export data to Datadog. Datadog provides an OpenTelemetry exporter that translates OTel data into Datadog's native format. This ensures that features like APM, infrastructure monitoring, and log management in Datadog can still leverage the standardized OTel inputs. Key to this is preserving essential metadata and correlations (e.g., trace context linking logs and metrics).

System Design Considerations for OTel Adoption

• Instrumentation Strategy: Decide between automatic and manual instrumentation. Automatic instrumentation is faster but less granular, while manual offers precise control over custom metrics and spans.
• Collector Deployment: Determine the optimal deployment strategy for OTel Collectors (e.g., agent on each host, daemonset in Kubernetes, central gateway). This impacts latency, resource utilization, and data aggregation capabilities.
• Data Governance: Plan for data sampling, filtering, and redaction within the OTel Collector to manage costs and ensure compliance, especially before data leaves your infrastructure and is sent to a vendor.
• Migration Path: For existing systems, plan a phased migration from vendor-specific instrumentation to OpenTelemetry to minimize disruption and ensure continuous observability.