The storage engine

Most hosted forges are built around Git’s on-disk storage and its ref model. neosource is not. Underneath the protocols is a content-addressed storage engine written for this platform, and nearly everything that makes neosource behave differently comes from it.

You do not need to know any of this to use neosource. Push with Git and it behaves like a forge. This page is for people who want to know what is actually underneath.

Three ideas

The engine rests on three things.

Immutable, content-addressed objects. Every piece of repository content — file blobs, trees, commits — is stored under a hash of its own bytes. Content that already exists is never stored twice, and an object can never change meaning after it is written. Identical content pushed to a thousand repositories is stored once.

An append-only operation log. Every mutation of a repository is recorded as an operation: what changed, and which operation it followed. The log is only ever appended to. Nothing rewrites history in place, so the state of a repository at any recorded point is recoverable rather than reconstructed.

Compare-and-swap operation heads. The current state of a repository is a small pointer — its operation heads. Advancing that pointer is a single atomic compare-and-swap: read the current head, do the work, then swap the pointer only if it still points where it did when you started. If another writer got there first, the swap fails and the loser retries against the new state.

Why this matters

Git’s model for updating a ref assumes one authority that can take a lock. That assumption is fine on a laptop, and it is the reason hosted Git forges need substantial distributed machinery to make ref updates safe across more than one node: something has to arbitrate, and that arbiter tends to become the write bottleneck.

An append-only operation log with compare-and-swap on operation heads does not need that arbiter. Multiple API nodes can accept writes to the same repository concurrently, because correctness comes from the atomic swap rather than from a lock held somewhere else. Contention is resolved by retrying against the new state, not by queueing behind a single writer.

This is a claim about the write path, not a promise about any particular throughput number. The point is structural: the coordination cost that a single-node ref model forces on a hosted forge is not in this design.

What follows from it

Crash safety comes from ordering. A write authorizes, stores its immutable objects, appends the operation record, and only then advances the operation heads. A crash before the heads advance leaves the objects orphaned but the repository untouched — no corruption, no half-applied push. A crash after them leaves the canonical state correct and only caches stale.

Indexes are derived, never authoritative. Search indexes, projections, and caches are all rebuildable from the objects and the operation log. They are version-keyed and disposable. If a derived structure is wrong, it is discarded and rebuilt; the canonical state is never at risk from a bad index.

Storage tiers are chosen per workload. Object bytes, the content-addressed manifest (operation log, operation heads, object and pack catalogs), and operational data like accounts and pull requests live in different substrates suited to their access patterns — each behind an abstraction rather than a hard dependency on one product. See Platform posture for what that means for portability, and Large files for how content-addressing changes the economics of big blobs.

Where the design comes from

The engine’s design lineage is jj’s model: the operation log, lock-free concurrency, and content-addressed storage are ideas jj demonstrated, and they map unusually well onto a hosted platform that has to accept concurrent writes across machines.

The implementation is our own. jj-lib is not a dependency; the semantics are implemented in-house. neosource is not “jj running on a server”, and using neosource does not require learning jj — see Protocols: Git and jj.