Optimizing DNS Propagation: Targeted Builds and Scalable Zone Management
This article details Cloudflare's architectural changes to their DNS propagation pipeline, focusing on how they halved DNS build delays and significantly reduced memory usage. The improvements center around moving from full zone rebuilds to targeted, incremental builds, especially for large zones and zone setting changes, by analyzing event data and optimizing their build planning and scheduling processes. This approach addresses the scaling challenges posed by millions of DNS records and frequent updates in a global distributed system.
Read original on Cloudflare BlogCloudflare's DNS propagation pipeline is a critical component for distributing DNS record changes globally. As the number of events and the size of individual zones (some with over 20 million records) grew, the original system, which relied on full zone rebuilds, faced significant scalability challenges, particularly concerning memory consumption and build latency. This case study highlights their architectural shift towards a more efficient, targeted approach.
The Original DNS Propagation Pipeline
- Customer Initiated Change: DNS record or zone setting change via API/dashboard.
- Database Storage: Change is stored in a backend database.
- Kafka Event: A Kafka event is produced from the database change or another system.
- Kafka Aggregator: Combines events from multiple Kafka topics and forwards to the Zone Builder.
- Zone Builder: Reads zone data, converts it to keys for authoritative DNS servers, compares with Quicksilver (distributed KV store), and writes only changes. Quicksilver then propagates these changes.
Scaling Challenge: Full Builds
The original Zone Builder's simplicity—it didn't need to know *why* a zone changed, just rebuilt from current state—became its downfall. For zones with tens of millions of records, a full build meant reading, encoding, comparing every record, and holding the entire zone in memory, leading to OutOfMemory errors and blocking further changes. Zone settings changes were particularly problematic as they often triggered full rebuilds.
Architectural Evolution: Targeted Builds
To address the scalability issues, Cloudflare introduced a targeted build planner and a unified scheduler. This system intelligently identifies the minimal set of changes required for a build, drastically reducing memory usage and latency. The core problems solved were: events needing to be more specific, mapping changes to the smallest safe build, accounting for shared zone data, handling database changes between event and build, and combining work efficiently.
Key Components of the New System
- Enriched Event Producers: Events now include specific changed fields and their old/new values, enabling granular decision-making.
- Targeted Build Planner: This component determines the smallest safe build type (e.g., resources only, apex and metadata, full zone) by analyzing the event and ensuring the output is byte-for-byte identical to a full build. It includes a fallback to full builds for uncertainty.
- Database Read Snapshot: Planning happens twice: initially for a conservative work list, and then immediately before execution, a fresh publication snapshot (zone settings, network publishing status, shared metadata IDs) is read from read-only replicas to verify and adapt the build plan to the latest state.
- Unified Scheduler: Replaces separate schedulers for record and full-zone work. It combines multiple events for the same zone into a single, optimized build, merging work (e.g., record changes + wildcard changes) to perform the most comprehensive work needed by any event.
A crucial trade-off for very large zones (above a fixed record-count threshold) is that the Zone Builder will *never* escalate to a full build, even if uncertainty suggests it. Instead, it rebuilds the apex, metadata, and specifically identified names. This pragmatic decision acknowledges that a full build for such zones is infeasible and would block other operations, prioritizing availability over absolute theoretical correctness in rare edge cases.