Anthropic's Scion: A Client-Server Source Control System for AI-Scale Monorepos

The Challenge of AI-Scale Source Control

Anthropic faced a unique challenge with its source control infrastructure: a rapid increase in AI-generated code leading to an exponential growth in daily merges (from 2,500 to over 30,000) and an enormous monorepo size (100 GB history, 25 GB compressed working copy). This resulted in an astounding 25 million CI checkouts per day, generating PBs of traffic. Traditional Git, with its distributed nature, became a bottleneck due to the high costs associated with repeated data transfer, storage, and maintenance for numerous, often partial, client repositories.

Why Git Fell Short for Anthropic's Workload

Git's core strength lies in its local independence, allowing developers to perform operations offline. However, for Anthropic's automated CI jobs and agents, this independence was not valuable. Managing hundreds of internal Git mirrors, keeping them in sync, and dealing with outdated seeds proved unsustainable. The partial repository data on clients also complicated operations requiring historical context. Anthropic's workflow already relied on a single authoritative shared repository, making Git's client-side independence an architectural impedance mismatch.

Introducing Scion: A Client-Server Asymmetric Design

Anthropic's solution, Scion, adopts a client-server asymmetric model. The server is responsible for most repository operations (historical queries, diffs, commits, merges), while the client (scn) maintains a limited cache and manages local files. This shifts the burden of historical data and complex operations from individual clients to a centralized, scalable service. Commit creation, for instance, becomes a server operation, eliminating the separate 'push' step found in Git. While this introduces a strong dependency on service availability, Scion is designed as a distributed service, not a single server.

• Server Responsibilities: Repository operations, historical data queries, diff calculations, commit creation, merges, rebases.
• Client Responsibilities: Maintaining a limited cache, preparing files on disk, watching local filesystem changes, providing developer/job interface.
• Benefit: Reduces client-side data management and processing, leveraging server-side power for complex tasks.

Content Addressing and Caching in Scion

Scion extensively uses content-addressing for its source control objects, recognizing them by their cryptographic hash. This makes objects immutable: a specific ID always refers to the same data. This immutability simplifies caching significantly. Cached historical objects don't require freshness checks, as their content cannot change. The system's design is heavily optimized around requesting and caching these immutable objects efficiently. Changesets (durable records of changes), blobs (file contents), and trees (directory structures) are managed this way, with metadata in Spanner and file contents in object storage.

Scalability through Shared APIs and Hierarchical Caching

Scion scales by separating answering repository questions (e.g., blame, history) from delivering working directory files, leveraging shared source control APIs, and a hierarchy of caches. Repository operations are handled by a shared service. The server-derived root tree, rather than a stored one, allows recording changes based on affected data and structures without processing the entire repository, which is crucial for handling high change rates in a monorepo. This architecture prioritizes efficient data distribution and processing for an environment with massive, concurrent consumers.