Distributed tracing: OpenTelemetry in practice — the good and the bad

we've been rolling out opentelemetry across 25 of our core services for the last six months, and the results are pretty interesting. on the good side, distributed tracing has been absolutely invaluable for debugging complex issues across multiple services, especially understanding latency hotspots. but it's not without its challenges. we're seeing a consistent 5-10% latency overhead in some services, which is non-trivial. the sheer volume of trace data is enormous, even with a 1% sampling rate, leading to significant storage and processing costs. what are others' experiences with opentelemetry at scale? how are you managing the data volume and balancing observability benefits against performance overhead and cost?