The storage engine

Most hosted forges are built around Git’s on-disk storage and its ref model. neosource is not. Underneath the protocols is a content-addressed storage engine written for this platform, and nearly everything that makes neosource behave differently comes from it.

You do not need to know any of this to use neosource. Push with Git or jj and it behaves like a forge. This page is for people who want to know what is actually underneath.

Three ideas

The engine rests on three things.

Immutable, content-addressed objects. Every piece of repository content — file blobs, trees, commits — is stored under a hash of its own bytes, so an object can never change meaning after it is written, and content already present is not stored again.

Dedup is deliberately bounded: it applies within a workspace and across a fork network, not globally across tenants. Sharing storage between unrelated customers would leak information about what they hold and entangle their data; tenant isolation is worth more than the bytes it costs.

An append-only operation log. Every mutation of a repository is recorded as an operation: what changed, and which operation it followed. The log is only ever appended to. Nothing rewrites history in place, so the state of a repository at any recorded point is recoverable rather than reconstructed.

Compare-and-swap operation heads. The current state of a repository is a small pointer — its operation heads. Advancing that pointer is a single atomic compare-and-swap: read the current head, do the work, then swap the pointer only if it still points where it did when you started. If another writer got there first, the swap fails and the loser retries against the new state.

Why this matters

Git’s model for updating a ref assumes one authority that can take a lock. That assumption is fine on a laptop, and it is the reason hosted Git forges need substantial distributed machinery to make ref updates safe across more than one node: something has to arbitrate, and that arbiter tends to become the write bottleneck.

An append-only operation log with compare-and-swap on operation heads does not need that arbiter. Multiple API nodes can accept writes to the same repository concurrently, because correctness comes from the atomic swap rather than from a lock held somewhere else. Contention is resolved by retrying against the new state, not by queueing behind a single writer.

This is a claim about the write path, not a promise about any particular throughput number. The point is structural: the coordination cost that a single-node ref model forces on a hosted forge is not in this design.

What follows from it

Crash safety comes from ordering. A write authorizes, stores its immutable objects, appends the operation record, and only then advances the operation heads. A crash before the heads advance leaves the objects orphaned but the repository untouched — no corruption, no half-applied push. A crash after them leaves the canonical state correct and only caches stale.

Indexes are derived, never authoritative. Search indexes, projections, and caches are all rebuildable from the objects and the operation log. They are version-keyed and disposable. If a derived structure is wrong, it is discarded and rebuilt; the canonical state is never at risk from a bad index.

Storage tiers are chosen per workload. Object bytes, the content-addressed manifest (operation log, operation heads, pack catalog), and operational data like accounts and pull requests live in different substrates suited to their access patterns — each behind an abstraction rather than a hard dependency on one product. See Platform posture for what that means for portability, and Large files for how content-addressing changes the economics of big blobs.

An engine of our own

This is a novel distributed storage engine, built for this platform: content-addressed objects, an append-only operation log, and lock-free atomic advances of operation heads. Concurrent writers do not queue behind a lock and do not corrupt each other — a write either advances the head atomically or retries against the state that beat it.

It is a storage engine for distributed version control rather than a Git implementation — the canonical bytes are Git-format packs (that format won on measurement, and it is what makes a Git clone a direct transfer rather than a transcode), but the coordination layer underneath is not Git’s: no ref locking, no on-disk repo, no Git object database as the primitive. That is what lets jj ride over the same objects today as side-data, and lets additional clients over the same snapshot model fit the same way over time.

Using neosource requires none of this. You push with Git or jj and you get a forge — see Protocols.