Uber's Automated Dependency Analysis in a Service Mesh
Uber developed an automated system to map and classify service dependencies within its microservices architecture to identify how downstream failures propagate. This system uses RPC middleware to correlate caller and callee failures, classifying dependencies as fail-close, fail-open, or unknown, offering faster feedback than traditional distributed tracing for rare failures.
Read original on InfoQ ArchitectureThe Challenge of Dependency Analysis at Scale
In a large-scale microservices environment like Uber's, with thousands of services, understanding how failures in one service affect others is crucial for reliability. Traditional distributed tracing, while powerful for individual request visibility, struggles with the analysis of rare failures across the entire system due to sampling limitations. If 99.9% of requests are successful, a 0.01% sampling rate means a failed request has a one-in-ten-million chance of being sampled, making it impractical for timely failure correlation analysis.
Uber's Approach: Middleware-based Failure Correlation
Instead of relying solely on distributed tracing, Uber engineers opted for an automated system built directly into their RPC framework, YARPC (for Go and Java), using inbound and outbound middleware. This approach focuses on aggregate metrics rather than individual trace details, providing faster feedback on dependency failure propagation.
Key Dependency Classifications
Fail-close: A dependency whose failure consistently causes its caller to fail. Fail-open: A dependency that can fail without causing its caller to fail. Unknown: Insufficient evidence to classify.
Technical Implementation Details
- Request ID Propagation: Inbound middleware generates a request ID and records caller service/endpoint in an in-memory `RequestTracker`. This ID propagates through Go's context (or similar mechanisms in Java).
- Outbound Call Recording: Outbound middleware records the downstream service, endpoint, and call result, linking it to the request ID.
- Metrics Emission: At request completion, middleware emits metrics capturing caller and callee failure states.
- Correlation Calculation: A failure correlation probability is calculated for each caller-dependency pair. Probabilities ">=0.8" are fail-close, "<=0.2" are fail-open, and intermediate values are unknown. Retries are handled separately to prevent skewing classifications.
This method trades the granular detail of individual request traces for broader coverage and quicker analysis of failure propagation patterns across the service mesh, enabling Uber to catalog dependencies for critical APIs and identify fail-close anomalies more efficiently.