Cloudflare optimizes DNS propagation with targeted build planner and scheduler

Cloudflare has overhauled its DNS propagation pipeline to handle increasingly massive DNS zones, some containing over 20 million records. Previously, changes to zone settings often triggered 'full builds' that required reading, encoding, and comparing every single record in a zone, which consumed excessive memory and time.

To solve this, the company introduced a targeted build planner and a new scheduler. These components allow the system to identify the smallest possible subset of data that needs updating—such as specific DNS resource keys or shared metadata—rather than rebuilding the entire zone. This optimization has halved the time required to write changes to their Quicksilver distributed key-value store.

Technical improvements to the Zone Builder

• Targeted Build Planner: A new logic layer that classifies changes into specific build types, such as 'Resources only' (for A, AAAA, TXT, or CNAME changes) or 'Resources and metadata' (for changes affecting nameservers or wildcards).
• Memory Optimization: By avoiding full zone reloads for setting changes, the system reduced peak memory consumption by 85%, preventing crashes in zones that exceed available RAM.
• Event Granularity: The event producer was updated to include specific changed fields and both previous and new values, allowing the planner to map events to the smallest safe build path.
• Work Scheduling: A new scheduler combines multiple pending work items for the same zone to maximize efficiency and prevent redundant processing of Kafka events.

Why it matters

While this is an internal infrastructure update, it demonstrates how large-scale distributed systems use Kafka and specialized planners to manage massive state changes. The techniques used to handle high-throughput DNS updates and minimize memory footprints are highly relevant to engineers building cloud-native networking tools and distributed key-value stores.