Scaling Git Infrastructure: GitHub's Hybrid Cloud Migration to Agent-Scale
GitHub details its strategic migration of Git infrastructure to Azure, driven by exponential traffic growth and data center capacity constraints. The article highlights architectural shifts like minimizing coordination through per-reference write serialization and decoupling storage from compute to enhance scalability and throughput for their Tier 0 Git service, especially for the busiest repositories.
Read original on GitHub EngineeringGitHub's journey to scale its Git infrastructure for "agent-scale" traffic involves a hybrid cloud migration strategy, moving repository replicas from their data centers to Azure. This move was necessitated by rapid traffic growth, power capacity limits in existing data centers, and supply chain constraints. The core challenge was to migrate a Tier 0 service, critical to all GitHub operations, without compromising availability or performance during the transition.
Overcoming Cross-Region Write Latency with Per-Reference Serialization
Initially, the cross-region latency (around 25ms round trip between data centers and Azure) became a bottleneck for the busiest repositories. The existing architecture used a repository-wide lock for writes, where the slowest healthy replica dictated the write latency and thus the overall throughput. This serialization meant only one write could be in flight at a time for a given repository. To mitigate this, GitHub introduced per-reference serialization.
Minimizing Coordination for Higher Throughput
By shifting the quorum's scope from the entire repository to individual Git references, writes to different branches or tags can proceed concurrently. This design decision significantly increases write throughput by removing the single bottleneck, allowing multiple writes to be in flight simultaneously, even with cross-region latency.
Decoupling Storage from Compute for Read Scalability
Git repositories are typically heavily read-skewed (100-1,000 reads for every write). In the traditional architecture, reads and writes contended for the same resources. GitHub's new approach decouples storage from compute, using Azure Blob Storage as the durable source of truth. This is facilitated by a "virtual replica" that acts as a quorum member, streaming writes to object storage but serving no reads.
- Virtual Write Replica: A special quorum member that ensures writes are durable in object storage. It coalesces transactions on disjoint references into a single set of network I/Os to object storage, optimizing concurrent writes.
- Virtual Read Replicas: Once data is durable in object storage, an arbitrary number of virtual read replicas can be scaled out on demand. They warm their cache from Azure Blob Storage and serve read traffic without joining the write quorum, using object storage as the source of truth and reconciling local state in the background. This significantly boosts read scale.