Scaling Git at OpenAI | OpenAI
OpenAI built Pando, a new storage and serving backend for Git that keeps the Git client and protocol but handles many concurrent pushes and very high read volume. It now runs their monorepo and cut median push latency nearly 5x.
- Pando coordinates each branch independently and uses causal consistency instead of repository-wide ordering, so pushes to unrelated branches don't contend on one coordination point.
- It reuses packfiles across fetches rather than building one per client, with local caches hitting over 99%, which lets clones and fetches saturate 10 Gbps links on standard cloud VMs.
- In production it delivers nearly a petabyte of Git data per day, and the team contributed upstream changes to Git such as packfile-uri improvements and "explicit haves."
OpenAI's monorepo got hammered once Codex made engineers far more parallel. More branches, more pushes, more merges, and more fetches and clones from dev environments, research jobs, and CI. By the end of last year, slow fetches and pushes, failed merges, and reliability dipping below 90% at times made it clear the existing setup couldn't keep up. The team's response was Pando, a new backend for Git repositories that sits behind the normal Git protocol and client, so engineers and agents didn't have to change how they work. Pando now handles all development in the monorepo, cut median push latency by nearly 5x, and serves close to a petabyte of Git data per day.
The core design trades a little consistency for scale. Writes are coordinated per branch, so pushes to unrelated branches don't queue behind one global lock. The cost is that a reader looking at several branches may see them at different points in time. On the read side, Pando stops generating a unique packfile for every client and instead reuses packfiles across fetches, so the server can send the same bytes to many clients with little computation. Pushed packfiles land as immutable blobs in a blob store, and the branch's history is tracked with causal consistency rather than a single repository-wide ordering.
The authors compare Pando against two conventional approaches they seriously evaluated. The first is a three-phase commit design, where every replica holds a full copy of the repo and a majority must accept each ref update, requiring multiple rounds of coordination. The second orders all updates through a shared write-ahead log in blob storage, where the manifest becomes a contention point and replicas fall behind when writes outpace indexing. The team also says it is contributing its changes from its Git fork back upstream, since client-side improvements should help the wider community and not only OpenAI.