OpenAI's Pando: Scaling Git for Monorepos with a Custom Backend
This article details how OpenAI developed Pando, a custom Git backend, to address scalability bottlenecks in their monorepo caused by rapid commit growth from human and agentic developers. Pando significantly reduced push latency and improved reliability by fundamentally re-architecting how Git servers handle branch consistency and packfile generation, while maintaining compatibility with the standard Git client protocol.
Read original on The New StackOpenAI's challenge with their monorepo highlights a growing problem for large organizations and highly active development environments: traditional Git server architectures struggle to keep up with massive numbers of commits, especially with the rise of agentic coding tools. Their existing Git backend experienced reliability issues and slow operations, prompting the need for a new solution.
Why Standard Git Servers Struggle
Conventional Git servers face two primary bottlenecks: sequential coordination of all branch updates for repository-wide consistency, and the dynamic generation of custom packfiles for each client fetch. Both operations are computationally expensive and become severe bottlenecks under high load, leading to increased latency and reduced reliability. The article notes that listing millions of branches for object negotiation can also create large network payloads.
Pando's Architectural Innovations
Pando tackles these issues by making two key architectural shifts. First, it abandons the requirement for repository-wide consistent state across all branches, instead handling each branch independently. This allows concurrent pushes to unrelated branches without blocking. Second, it optimizes packfile handling: rather than generating new packfiles per fetch, Pando stores incoming packfiles in blob storage and serves pre-existing ones from a caching service (Express) using `packfile-uri`.
- Decoupled Branch Writes: Each branch is managed independently, reducing contention.
- Persistent Packfiles: Packfiles are stored and reused, dramatically cutting server-side computation on fetches.
- Layered Storage & Caching: Blob storage for packfiles, a key-value store for metadata, and regional caches (Express) for high hit rates.
- Background Compaction: Similar to log-structured databases, background workers merge small packs into larger ones over time, improving fetch efficiency.
Trade-offs
Pando's approach introduces a trade-off: a client viewing multiple branches simultaneously might see them at slightly different points in time. However, OpenAI argues that in practice, users primarily care about the consistency of a *single* branch. Another trade-off is that advertising only the trunk branch means repeat pushes to feature branches resend data not on trunk, but this is deemed acceptable given small median pack sizes.
Impact and Open Source Contributions
Pando, built in Rust, cut push latency by nearly 5x and serves close to a petabyte of Git data daily. While Pando itself isn't open-sourced, OpenAI is contributing numerous fixes and enhancements back to the core Git project, particularly around `packfile-uri` handling, concurrent downloads, tracing, and explicit 'haves' features. This demonstrates a commitment to improving the broader Git ecosystem even with proprietary backends.