GitHub's New Git Storage Architecture: Scaling for Agentic Development

The Challenge: Scaling Git for Agentic Development

The rise of AI agents means a significant shift in how Git repositories are used. Instead of human developers pushing code in bursts, agents generate sustained high-volume writes, often thousands per second. This necessitates a fundamental re-think of Git storage to support both horizontal scale (many developers across many repositories) and, crucially, vertical scale (many agents working on a single, busy repository). The goal was to achieve at least two orders of magnitude more scale for single repositories than previously possible.

• Accept thousands of writes per second.
• Support multiple writer nodes for CPU-intensive pack indexing and object connectivity checks.
• Minimize commit latency for agents.
• Record changes in order for each reference, with quick linearization across references.
• Efficiently catch up virtual replicas with minimal data transfer.
• Keep reads cheap, especially when no changes have occurred.
• Allow each component to scale independently.
• Improve overall reliability and global scale.

Rubicon Architecture: Core Principles and Components

Project Rubicon is built on two core principles: minimize coordination and decouple storage from compute. This enables independent scaling of different parts of the Git infrastructure. The architecture includes several key components:

• Spokes: Handles routing, load balancing, placement, and tenant isolation, maintaining horizontal scale.
• Virtual Replicas: A fleet of disposable compute nodes that serve reads and writes. They have local disk caches that can be rebuilt on demand. Traffic is routed to keep caches warm, but correctness does not depend on a specific replica.
• Compaction Nodes: A background fleet dedicated to merging packs and garbage collection directly against durable storage, offloading this work from virtual replicas.
• Azure Blob Storage: Stores immutable Git objects (packs and indexes) and various read accelerators. Its high aggregate throughput for distinct keys makes it ideal for write-once data.
• Ref-CAS (Reference Compare-and-Swap) Service: A custom replicated log service for mutable reference state and pack manifests. This small consensus group manages ordered updates for references.

Write and Read Flow

1. Write: A virtual replica indexes an incoming pack, uploads it to Blob Storage, then commits a single transaction to Ref-CAS to update the pack manifest and references. Compaction nodes later optimize packs.
2. Read: A virtual replica checks the latest reference version with Ref-CAS, pulls necessary packs from Blob Storage into its cache, and serves the read.

Ref-CAS: Minimizing Coordination for Reference State

The biggest challenge for high-throughput writes was managing reference state. Initial attempts to store references directly in Blob Storage faced limitations:

• Single-blob write-ahead log: Offered correct writes and cheap reads, but serialized all writes to a single blob, limiting throughput to hundreds per second due to conditional write constraints.
• Segmented blobs: Improved write throughput by allowing parallel writes to different segments, but made reads more expensive as they had to check all segments, becoming as slow as the slowest segment.