Optimizing Monorepo Size for Developer Velocity at Scale
Dropbox faced significant developer velocity challenges due to their 87GB monorepo, impacting clone times and CI/CD. This case study details their investigation into the unexpected Git compression behavior caused by their i18n file structure and their collaborative solution with GitHub to reduce the repository size to 20GB, significantly improving development workflows.
Read original on Dropbox TechDropbox, like many large tech companies, utilizes a monorepo approach for much of its backend infrastructure. While this architecture simplifies cross-service development, it introduced severe performance bottlenecks as the repository grew to an unmanageable 87GB. This size led to initial clone times exceeding an hour, slowed down critical CI/CD pipelines, and approached GitHub's 100GB hard limit, posing significant operational risks. The core problem wasn't merely the volume of code, but how Git was storing it.
The Unseen Culprit: Git's Delta Compression Heuristics
Initial investigations into common causes like large binaries or committed dependencies proved fruitless. The real issue was identified in Git's delta compression. Git optimizes storage by representing file versions as deltas (differences) against similar files. By default, Git uses only the last 16 characters of a file path to find candidates for delta comparison. Dropbox's internationalization (i18n) file structure, specifically `i18n/metaserver/[language]/LC_MESSAGES/[filename].po`, meant the language code appeared earlier in the path, preventing Git from efficiently comparing files within the same language. This led to larger, less efficient deltas being generated for routine translation updates.
Key System Design Takeaway
This case highlights that infrastructure and tool choices, even at a low level like version control compression, can have profound impacts on developer velocity and operational stability at scale. Assumptions embedded in tools may not hold for unique system architectures or data structures, necessitating deep understanding and sometimes platform-level collaboration.
Collaborative Solution with GitHub
Dropbox's team locally validated their hypothesis using Git's experimental `--path-walk` flag, which restructured the repository from over 80GB to 20GB. However, this flag was incompatible with GitHub's server-side optimizations (like bitmaps and delta islands) crucial for fast clones and fetches. The permanent solution required collaboration with GitHub, who recommended an aggressive repack using tuned `window` and `depth` parameters. This method allows Git to search more thoroughly for similar objects and permits more layers of deltas, improving compression without sacrificing server-side optimizations. The repack took approximately nine hours but successfully reduced the repository size from 84GB to 20GB.
Gradual Rollout and Impact
The production repack was treated as a critical infrastructure change, with a test mirror first undergoing the process and close monitoring of fetch duration, push success rates, and API latency. GitHub then gradually rolled out the change across production replicas over a week. The result was a dramatic reduction from 87GB to 20GB, cutting clone times from over an hour to under 15 minutes. This not only improved developer onboarding and CI/CD efficiency but also mitigated the risk of hitting GitHub's hard limits, ensuring stability and performance at scale.
- Repository size reduced by 77% (87GB to 20GB).
- Clone times dropped from over 60 minutes to under 15 minutes.
- CI pipelines became faster and more reliable.
- Operational risk from approaching GitHub's size limit was eliminated.