Optimizing DNS Cache Memory in Large-Scale Distributed Systems

Cloudflare's Big Pineapple platform, serving 1.1.1.1 and other DNS services, manages an enormous DNS cache. The article details the critical importance of memory efficiency at this scale, where even small per-entry savings translate into massive fleet-wide reductions. The optimizations also led to performance improvements, demonstrating that space efficiency can often go hand-in-hand with speed when managed carefully through techniques like improved memory locality and fewer allocations.

The Challenge: Scale and Overhead

With 250 billion DNS cache entries, any wasted byte per entry accumulates significantly. The challenge was compounded by features like EDNS Client Subnet (ECS), which necessitate caching multiple versions of DNS answers, further increasing entry count and memory footprint. Each cache entry comprises a key (the queried domain) and a value (DNS response, metadata), both of which were initially stored using standard Rust types that introduced unnecessary overhead at rest.

Memory Optimization Techniques

• Replacing `Vec` and `String` with `Box` and `Box`: Standard `Vec` and `String` carry capacity fields and over-allocate heap space for future growth. Since cached DNS responses are immutable, using `Box` and `Box` eliminated these 8-byte capacity fields and unnecessary heap reservations, saving 64 bytes per entry and over 15 TB fleet-wide.
• Consolidating Lists: Instead of separate lists for answer, authority, and additional record sections, a single list with `u16` offsets was used. This replaced 8-byte pointers and lengths for each list with 2-byte offsets, saving 28 bytes per entry and reducing padding.
• Conditional Owner Storage: Most DNS records have an owner identical to the queried domain. By inferring the owner from the cache key when it's the same, and only storing the full owner name when it differs (e.g., due to CNAMEs), heap allocations for owners were significantly reduced.
• Boxing Large Enum Variants: Rust enums are sized to their largest variant. To avoid excessive padding for small variants (like A/AAAA records, which constitute 80% of traffic but were padded to 144 bytes for NAPTR), larger variants were boxed. This moves their data to the heap, where they occupy only their actual size, saving up to 120 bytes per record for common types.
• Storing Records in Wire Format (Partial): The most impactful change involved storing only the record data as raw bytes (wire format) within a single `Box`, rather than parsed enum variants. This eliminated per-variant enum overhead, separate heap allocations, and improved CPU cache locality. While it meant sequential iteration for records and added complexity for features like round-robin, the performance benefits (5% lookup latency reduction) and memory savings were substantial.

These optimizations demonstrate that a deep understanding of language runtime, memory allocators (like `jemalloc`), and data access patterns is crucial for achieving extreme efficiency in high-performance distributed systems. The trade-offs between memory, CPU cache locality, and serialization/deserialization costs must be meticulously benchmarked and evaluated to find the optimal balance for specific workloads.

Benchmarking and Production Validation

Cloudflare used a custom allocator wrapping Rust's System allocator to precisely measure memory usage per cache entry. They also benchmarked insert throughput and lookup latency using randomly generated entries mimicking production traffic. Crucially, resident memory usage was monitored across production instances during rollout, validating that memory savings did not come at the expense of performance, but rather improved it.