Cloudflare Logpush Transformers: In-Process SQL Transformations for Log Exports
Cloudflare's Logpush Transformers enable in-process SQL-based filtering, reshaping, and redaction of logs before export to various destinations. This eliminates the need for external processing services, reducing costs and operational overhead. The system leverages Apache DataFusion and Apache Arrow for high-throughput, efficient execution within the existing Logpush delivery pipeline, ensuring reliability and predictable resource usage for processing over a billion events per second.
Read original on Cloudflare BlogCloudflare Logpush Transformers provide a significant architectural enhancement to log management by moving log processing closer to the source. Traditionally, log data would be exported in its raw form, then filtered, reshaped, or redacted by separate, customer-managed services. This approach introduced additional infrastructure, operational burden, and data transfer costs. Transformers embed this processing directly into Cloudflare's Logpush export workflow, allowing customers to define SQL queries that are applied *before* logs leave Cloudflare's infrastructure.
Architectural Motivation and Trade-offs
The primary motivation for Transformers was to reduce complexity and cost for users while maintaining Cloudflare's high standards for log delivery reliability and throughput. The initial idea of using Cloudflare Workers for transformations was discarded due to the overhead of network calls, which could introduce latency, timeouts, and retries into the critical log export path. This led to a key architectural decision: in-process execution.
In-Process vs. Out-of-Process Execution
The choice to embed the transformation engine directly within the Logpush delivery service (Bufstream) highlights a critical trade-off. While out-of-process solutions (like Cloudflare Workers) offer greater isolation and flexibility, they introduce network latency and distributed system complexities. In-process execution prioritizes performance, reliability, and simplified operational overhead by eliminating network hops and leveraging existing infrastructure resources.
Key Components and Technologies
- Bufstream (Logpush Delivery Service): Cloudflare's distributed, high-throughput service responsible for delivering logs. It runs across hundreds of Kubernetes pods and claims data from Buftee (Cloudflare's in-house message broker).
- Apache DataFusion: A Rust-based query engine built on Apache Arrow, chosen for its SQL planner and vectorized, columnar execution engine. It's used to execute customer-defined SQL transformation queries.
- Apache Arrow: A language-independent, columnar in-memory format that enables efficient data exchange between Go (Bufstream) and Rust (DataFusion) across an FFI boundary. This minimizes serialization/deserialization overhead and ensures type consistency.
- Cap'n Proto: Used for encoding records within Bufstream before transformation.
The integration of DataFusion (Rust) into Bufstream (Go) required careful handling of memory ownership and efficient data transfer. Apache Arrow's IPC (Inter-Process Communication) format was crucial here, allowing Go to pass pointers to Arrow buffers to Rust for the duration of the CGO call, with Rust explicitly managing and releasing its own output buffers. This design avoids cross-runtime memory management issues and optimizes performance by operating on batches of columnar data.
Reliability and Failure Handling
Transformers are designed to fail closed. If a transformation cannot be executed successfully, Bufstream will not advance the job's delivery offset and will re-attempt the transformation. This prevents potentially malformed or untransformed data from reaching downstream systems, which could violate schema or redaction assumptions. To prevent a continuously failing job from consuming excessive resources, Bufstream implements progressive retry delays and automatically disables jobs that remain in a failing state for an extended period.