Cloudflare's Data Pipeline: Scaling for Exponential Growth
This article details Cloudflare's strategies to manage exponential data growth in their globally distributed data pipeline while minimizing hardware usage. It covers architectural shifts like schema decomposition, compression algorithm changes, replication rearchitecture, decentralized sampling, and early-stage aggregations to achieve significant resource cost reductions and enhance scalability.
Read original on Cloudflare BlogCloudflare's data team faces the challenge of processing billions of events per second with linearly growing hardware. To combat this, they implemented several architectural and technical optimizations over five years, effectively deferring two and a half years of hardware expansion. The core problem is managing massive traffic volume in a globally distributed system that feeds critical services like Cloudflare Logs, SQL API, and internal analytics.
Evolution of the Data Pipeline Architecture
The pipeline begins with Logfwdr collecting data from distributed sources, feeding it to centralized receivers, and then pushing to a buffering system called Buftee. Consumers then fetch data from Buftee and deliver it to final destinations like ClickHouse. The following key changes were made:
- Schema Decomposition (2022): Replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth usage by 35-40% by eliminating the need for consumers to ingest irrelevant fields. This trade-off involved duplicating some fields but significantly improved efficiency.
- Compression Algorithm Change (2024): Switched from LZ4 to Zstandard (zstd) for data compression. Despite initial concerns about CPU usage, careful rollout and monitoring showed minimal impact, resulting in a nearly 40% reduction in compressed traffic throughput. This allowed supporting 1.5x more traffic with the same network capacity.
- Rearchitected Pipeline Replication (2026): Moved from source-side double-writing for disaster resilience to late replication. Instead of Logfwdr sending two full copies of events across wide-area networks, each event is sent once to an available centralized receiver. Replication then occurs downstream from Buftee, after initial sampling, reducing wide-area transit traffic by 36 GB/s. This also leveraged better compression due to uniform partitioning of aggregated streams.
- Decentralized Sampling (2026): Implemented sampling earlier in the pipeline, within Logfwdr, in addition to existing sampling in centralized receivers. This double sampling technique, which shuffles records and samples based on the square root of the count, significantly reduced the number of events delivered and stored for high-volume keys while preserving detail for low-volume keys. This resulted in a 25GB/s reduction in compressed data on the wire.
- Early Aggregation (2026): Decentralized data aggregation by integrating aggregation functionality at multiple stages: Logfwdr, centralized receivers, after dedicated partitioned buffers, and finally in ClickHouse. This reduces the amount of data centralized and spreads out computational load, dropping compressed bytes traffic for one dataset from 9+ GB/s to <2 GB/s.
Impact and Key Takeaways
System Design Lessons
This case study demonstrates the critical importance of continuous optimization in large-scale distributed systems. Key lessons include: 1) Data Efficiency: Optimizing data representation (schema), compression, and volume reduction (sampling, aggregation) can yield massive savings. 2) Decentralization: Pushing processing closer to the data source and distributing workloads horizontally can mitigate bottlenecks. 3) Trade-offs: CPU for network, data completeness for storage/bandwidth (sampling), and early vs. late processing are constant considerations. 4) Observability: Metric tracking and phased rollouts are essential for validating changes in production.