Mastering Latency in System Design Interviews

Latency is a critical metric in system design, reflecting the user experience and the overall performance of a distributed system. A strong system design discussion goes beyond merely mentioning a cache; it delves into the specifics of how latency is perceived, measured, and managed across different components and scenarios.

Why Latency Matters in System Design

Understanding latency demonstrates an engineer's grasp of the physical costs of operations, the importance of thinking in distributions (not just averages), and the alignment between architectural choices and performance targets. A design that involves multiple cross-continental network hops for a single request, for instance, inherently contradicts a low-latency target.

Key Latency Tiers to Understand

1. Memory and Cache: Effectively instantaneous at the scale of a single request.
2. In-datacenter Network Hops: Fractions of a millisecond; can accumulate with many hops.
3. Disk and Database Operations: Tens of milliseconds (spinning disk) or less (SSDs); often the largest part of the latency budget.
4. Cross-Region/Cross-Ocean Trips: Tens to hundreds of milliseconds, necessitating regional data locality for read paths.

The Importance of Percentiles (p99) and Fan-Out

Averages hide the 'tail' latency that impacts a small but significant percentage of users. High-order percentiles (e.g., p99, p99.9) reveal the plausible worst-case experience. When a request fans out to multiple backend services, the slowest backend dictates the overall response time for that user. Even if individual backends are fast 99% of the time, a fan-out to 100 services means approximately two out of three user requests will experience tail latency (1 - 0.99^100 ≈ 0.63). Mitigations include reducing fan-out, returning partial results, or sending backup requests.

Building and Measuring a Latency Budget

Start by setting an end-to-end user-perceived latency target (e.g., 100ms for a fluid experience), then allocate a server-side budget. Walk through the request path, assigning rough latency costs to each tier (network, disk, compute) and making architectural decisions based on the budget. For example, a precomputed feed in a cache is faster than building it on demand via fan-out. Finally, measure latency for successful and failed requests separately, using bucketed histograms to capture the full distribution, and alert on the promised percentiles.