Cloudflare Blog Migration to EmDash: A Customer Zero Case Study in Scalable CMS Architecture
This article details Cloudflare's migration of its blog to EmDash, a new internal CMS built on Astro and Workers. It highlights the 'Customer Zero' philosophy, where Cloudflare dogfoods its own products to validate scale, security, and usability. The migration involved significant architectural decisions around caching, performance testing, and a phased rollout strategy to ensure zero downtime and high availability under variable traffic loads.
Read original on Cloudflare BlogCloudflare's 'Customer Zero' Philosophy
Cloudflare embraces a 'Customer Zero' culture, where its own products are used internally at a massive scale before being offered to external customers. This approach rigorously tests products for scalability, security, and usability in a real-world, demanding production environment. For the EmDash CMS, Cloudflare became its initial, most critical user, identifying and addressing gaps related to large-scale content management, localization, SEO, and the usability of the administrative editor, especially concerning scheduled posts.
System Design for Scalability and Performance
The Cloudflare blog experiences highly variable traffic, from a baseline of 75 RPS to spikes over 5,000 RPS, sometimes reaching 7,000 RPS during bursts or DDoS attacks. To handle this, a robust architecture focusing on performance and resilience was designed. Key components included: EmDash running on Cloudflare Workers, leveraging Workers Cache, a custom EmDash object cache built on Workers KV, and Hyperdrive integration with PlanetScale. This multi-layered caching strategy ensures high availability and low latency, serving 99.5% of static files and 70% of requests directly from cache.
Performance Testing Strategy
Cloudflare utilized k6, an open-source performance testing tool, to simulate various load scenarios: a gradual ramp-up, a breakpoint test to identify failure thresholds, and an immediate burst test. Metrics evaluated included availability (less than 0.01% 5xx errors) and latency (P95 < 500ms, P99 < 1000ms). These tests were crucial in validating the proposed EmDash setup's ability to withstand significant traffic without degradation.
Architectural Stack
- Cloudflare Workers: Serverless execution environment for EmDash logic.
- Workers Cache: Edge caching layer for high-speed content delivery.
- Workers KV: Key-value store used for a custom EmDash object cache, built for this specific use case.
- Hyperdrive with PlanetScale: Managed database integration for persistent storage, benefiting from Hyperdrive's optimized database connectivity.
Caching Strategy
The multiple layers of caching (Workers Cache, custom Workers KV object cache) are critical for both frontend performance and reducing the load on the database. This pattern is essential for highly trafficked sites, ensuring responsiveness and resilience against traffic spikes.
Gradual Rollout and Zero-Downtime Migration
To achieve zero downtime, a proxy Worker was deployed to intelligently route traffic between the legacy blog and the new EmDash site. This allowed for a phased rollout, starting with 1% of traffic and incrementally increasing. The proxy Worker also provided a seamless fallback mechanism to the legacy blog in case of any 500 errors on the new platform. A key optimization was the use of a direct worker-to-worker connection via service binding, reducing latency compared to public hostname routing.