Cloudflare Migrates cdnjs to Serverless Developer Platform
Cloudflare successfully migrated its cdnjs service, handling 9 billion daily requests, to its internal Developer Platform. This architectural shift replaced a hybrid on-premise/Google Cloud setup with Cloudflare Workers, R2, Workflows, KV, and Durable Objects, showcasing a strategic move towards serverless architecture for a high-scale CDN. The migration emphasized maintaining existing URLs and integrity hashes, highlighting challenges in distributed system transitions.
Read original on InfoQ ArchitectureMigration to Cloudflare Developer Platform
Cloudflare undertook a significant architectural migration of cdnjs, an open-source CDN for JavaScript and CSS libraries, to its proprietary Developer Platform. This move consolidated publishing and serving infrastructure that was previously fragmented across Cloudflare's own systems and Google Cloud Platform. The new architecture leverages a suite of Cloudflare's serverless and edge computing offerings, demonstrating a "dogfooding" approach to their internal products for a critical, high-traffic service.
Old Architecture Overview
The previous architecture for cdnjs was split. While file serving had partially moved to Workers and Workers KV in 2020 (with external origins as fallback), the publishing path remained highly distributed. It involved Google Cloud Functions for package release checks, Google Cloud Storage for package files, Pub/Sub for messaging, and a VM for repository synchronization. This setup, with 26 Cloud Functions sharded alphabetically, managed a GitHub repository exceeding 1.1 TB and duplicated file representation in both GitHub and KV.
New Architecture Components and Flow
- R2 (Object Storage): Serves as the *source of truth* for published files, replacing Google Cloud Storage.
- KV (Key-Value Store): Stores package metadata, versions, and Subresource Integrity (SRI) hashes.
- Workers (Serverless Functions): Handle incoming requests for CDN content and serve files, leveraging Workers Cache for high hit rates.
- Workflows (Orchestration): Manages the entire package ingestion and publishing pipeline, including scheduled checks for new releases, package downloads, processing, minification, compression, and metadata updates.
- Containers: Used for resource-intensive tasks like compression, which require buffering entire libraries in memory, circumventing current Worker limitations. Cloudflare is exploring streaming support to potentially move this to Workers.
The migration achieved a 98.6% cache hit rate for cdnjs, handling 9 billion daily requests across 330+ data centers. A key requirement was preserving existing URLs, package contents, and SRI hashes, which is critical for CDN integrity and client-side caching. DigitalOcean Spaces provides an additional fallback mirror for published content.
Key System Design Takeaways
This case study demonstrates the power of a unified serverless platform for high-scale services. It highlights the challenges of migrating complex, distributed systems while maintaining data integrity (SRI hashes) and backward compatibility (URLs). The use of Workflows for orchestration and specialized Containers for compute-intensive tasks within a serverless ecosystem illustrates pragmatic design choices to overcome platform limitations and achieve high performance and reliability.
Platform Enhancements and Trade-offs
The migration itself pushed the limits of Cloudflare's Developer Platform, leading to significant increases in Worker subrequest limits (1,000 to 10 million) and Workflow steps (1,024 to 10,000, configurable up to 25,000). This exemplifies how internal 'dogfooding' of a platform can drive its evolution and robustness. The decision to use Containers for compression, rather than Workers, was a trade-off to manage memory constraints for processing large files, with future plans to enable streaming directly within Workers.