core

module

v0.0.7 Latest Latest Go to latest Published: Sep 9, 2026 License: GPL-3.0, LGPL-3.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/amber-store/core

Links

Open Source Insights

README ¶

Amber-Store Core

A content-addressable store for filesystem trees — arbitrarily deep directories and files, where file content is split by content-defined chunking and every object is identified by a fixed 32-byte key derived from a hash of its content.

Status: working implementation. This is the local core: a Go library (plus a minimal CLI) that owns a store in a local directory — no sockets, no networking, no authentication. It is the foundation the amber server is built on, and it embeds directly into other projects (e.g. the JOBS engine). The design is specified in architecture/.

What it is

Amber-Store models a POSIX-style filesystem as an immutable, deduplicated Merkle structure in a content-addressed store:

Files are split into chunks by content-defined chunking (CDC); each chunk is a Blob. Large files get a multi-level FileNode index for O(log n) random-access seek.
Directories are prolly trees — sorted maps from name to entry, chunked at entry boundaries — so a directory with >100K entries can be looked up, iterated, and mutated with sub-O(n) memory.
Metadata (type, mode, uid, gid, mtime, optional xattrs) lives in the parent directory entry. Symlinks and special files are stored inline; regular files and subdirectories reference content by key.
Deduplication is automatic: identical content yields an identical key, and editing one entry re-writes only the O(log n) objects on its path — the rest is structurally shared with the previous tree.

Design goals

Very large directories (>100K entries) processed with less than O(n) memory.
Directory entries carry all filesystem metadata; file content is reached through the store by key.
Deterministic, implementation-independent encoding so any reader recomputes the same key for the same content.

Object types

Type	Role
`Blob`	Raw file-content byte chunk (a CDC leaf).
`FileNode`	File chunk-index node (file content tree).
`DirLeaf`	A run of complete directory entries (prolly-tree leaf).
`DirNode`	Directory index node (directory tree).
`XattrSet`	Spilled extended attributes, when too large to inline.

Encoding

Keys are 32 bytes: a type/length header plus a truncated BLAKE3 hash of the content.
Structured objects use deterministic CBOR (RFC 8949 §4.2); Blobs are raw bytes. Deterministic encoding is required because the key's hash is taken over the serialized bytes.

The library

The packages compose loosely; consumers wire them together and own their store-directory layout. The conventional layout (which the CLI uses) is <dir>/packstore for objects and <dir>/refs for references. A store directory is single-owner: never open one from two live processes.

Package	Role
`key`	The 32-byte content key: type, length, truncated BLAKE3 hash.
`fstree`	Tree objects (encode/decode), bottom-up builders, and the read paths: entry lookup, ordered listing, content streaming, reachable-set walks, completeness checks.
`chunkers`	Content-defined byte chunking (ultracdc) and item chunking for tree nodes.
`ingest`	Build a tree from a local directory (or single file): `Objects` streams every built object plus the resolved root; `Dir` writes straight into a packstore; `Scan` sizes progress displays. Honors `.amberignore`.
`amberignore`	`.gitignore`-semantics exclusion for ingestion.
`packstore`	The local object store: append-only pack segments with parallel, deduplicating, verifying writers.
`refstore`	Pebble-backed name → record map for references.
`reference`	The reference record: canonical CBOR encoding and validation; signature fields carried opaquely.
`amberpack`	The flat pack stream format (`key + payload` records, no root) used for transfer and storage.
`inbox`	Durable pack receiving: persist incoming packs, then drain them into a packstore.
`tarexport`	Stream a stored tree as a PAX tar.
`tarextract`	Materialize such a tar onto the filesystem, restoring metadata.
`cborx`	Shared deterministic-CBOR helpers.

A minimal embedding looks like:

objects, _ := packstore.Open(filepath.Join(dir, "packstore"))
defer objects.Close()

root, stats, _ := ingest.Dir(objects, "./some/dir", ingest.Opts{})
_ = tarexport.Write(w, root, objects.Get)

The CLI

Every command operates directly on the store directory given by --store or $AMBER_STORE, creating it as needed. Ingest a directory or a single file — the root key (hex) is printed to stdout; a directory root is a DirNode, a single-file root is the file's content key:

amber-store --store ./store ingest ./some/dir           # print the root key
amber-store --store ./store ingest --ref backups/home ./some/dir  # also name it

Inspect and export by key or reference, optionally addressing a subdirectory with KEY/PATH or ref:NAME[@PATH]:

amber-store --store ./store ls KEY[/PATH]               # list entries, ls -l style (--keys adds content keys)
amber-store --store ./store ls ref:backups/home@sub/dir # ref:NAME[@PATH] works wherever KEY[/PATH] does
amber-store --store ./store export ref:backups/home -o tree.tar  # PAX tar (default: stdout)
amber-store --store ./store restore ref:backups/home ./dest      # recreate the tree on disk
amber-store --store ./store ref list                    # references: name, key, created, user
amber-store --store ./store ref set NAME KEY            # name an existing key
amber-store --store ./store ref get NAME                # print the key a name points at
amber-store --store ./store ref rm NAME                 # delete the name; objects stay

Ingest parallelism is set with --jobs (default: number of CPUs). Chunking is tunable with --min/--avg/--max (ultracdc byte chunking) and --item-bits (index/entry chunking); --xattr-inline-max controls when extended attributes spill to an XattrSet object.

.amberignore files exclude entries from ingestion, with .gitignore semantics: negation (!pattern), ** globs, directory-only (name/) and anchored (/name) patterns; a file in any subdirectory applies to that subtree and composes with inherited patterns (last match wins). Ignored directories are pruned without being read. The .amberignore files themselves are always stored, so a restored tree re-ingests to the same root. --no-ignore disables all ignore processing.

Architecture

Document	Contents
`architecture/keys.md`	The 32-byte lookup key: header byte, payload length, truncated hash.
`architecture/types.md`	The type model: object types, filesystem entry types, length-field semantics.
`architecture/fstree.md`	On-the-wire CBOR layout of every type, the chunkers, tree construction, and read paths.
`architecture/amberpack.md`	The flat pack stream: record framing, CRCs, recovery.
`architecture/references.md`	Named pointers to keys: record layout, name rules, storage.

Development

The repository uses a Nix flake (with direnv) to provide the Go toolchain.

direnv allow        # or: nix develop
go build ./...
go test ./...

Module: github.com/amber-store/core
Go: 1.26+

License

Licensed under the GNU Lesser General Public License, version 3 only (LGPL-3.0-only). See LICENSE for the LGPL terms and COPYING for the GPL terms incorporated by the LGPL.

Third-party notices retain their stated licenses.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
amberignore Package amberignore filters ingest trees through .amberignore files with gitignore semantics: patterns compose per directory down the tree and support negation, ** globs, dir-only (trailing /) and anchored (leading /) forms; the last matching pattern wins.	Package amberignore filters ingest trees through .amberignore files with gitignore semantics: patterns compose per directory down the tree and support negation, ** globs, dir-only (trailing /) and anchored (leading /) forms; the last matching pattern wins.
amberpack Package amberpack defines the Amber-Store pack format.	Package amberpack defines the Amber-Store pack format.
cborx Package cborx provides the minimal canonical-CBOR encoding the fstree needs for byte-string-keyed maps (extended attributes), which fxamacker cannot produce from a Go map[string]...	Package cborx provides the minimal canonical-CBOR encoding the fstree needs for byte-string-keyed maps (extended attributes), which fxamacker cannot produce from a Go map[string]...
chunkers Package chunkers provides the two content-defined chunkers the fstree uses: a byte chunker (ultracdc) for file content, and an item chunker for the index/entry streams (architecture/fstree.md, "Content-defined chunking").	Package chunkers provides the two content-defined chunkers the fstree uses: a byte chunker (ultracdc) for file content, and an item chunker for the index/entry streams (architecture/fstree.md, "Content-defined chunking").
cmd
amber-bench command amber-bench: an ingest → delete → gc benchmark for amber-store.	amber-bench: an ingest → delete → gc benchmark for amber-store.
amber-store command
fstree Package fstree encodes the Amber-Store filesystem tree objects (FileNode, DirLeaf, DirNode, XattrSet, Blob) as deterministic CBOR per architecture/fstree.md, and builds files and directories bottom-up by streaming.	Package fstree encodes the Amber-Store filesystem tree objects (FileNode, DirLeaf, DirNode, XattrSet, Blob) as deterministic CBOR per architecture/fstree.md, and builds files and directories bottom-up by streaming.
gc Package gc implements the mark-and-sweep collector of architecture/mark-sweep-gc.md, a port of Mic92's bitmap GC (draganm/amber-store#9): a cycle marks every key reachable from the references' roots into a packstore.MarkSet — one bit per sealed record, slotted by the packs' own footer indexes — and sweeps by rewriting the packs whose dead ratio crosses the line (packstore.Compact).	Package gc implements the mark-and-sweep collector of architecture/mark-sweep-gc.md, a port of Mic92's bitmap GC (draganm/amber-store#9): a cycle marks every key reachable from the references' roots into a packstore.MarkSet — one bit per sealed record, slotted by the packs' own footer indexes — and sweeps by rewriting the packs whose dead ratio crosses the line (packstore.Compact).
inbox Package inbox stores authenticated packs that have been received but not yet processed into the packstore.	Package inbox stores authenticated packs that have been received but not yet processed into the packstore.
ingest Package ingest builds content-addressed filesystem trees from a local directory or a single regular file: it walks the source, applies .amberignore filtering, splits file content by content-defined chunking, and streams every built object (children before parents) to the consumer.	Package ingest builds content-addressed filesystem trees from a local directory or a single regular file: it walks the source, applies .amberignore filtering, splits file content by content-defined chunking, and streams every built object (children before parents) to the consumer.
key Package key implements the 32-byte Amber-Store lookup key: a content address that encodes a CAS object type, a logical payload length, and a truncated BLAKE3 hash of the payload's serialized bytes.	Package key implements the 32-byte Amber-Store lookup key: a content address that encodes a CAS object type, a logical payload length, and a truncated BLAKE3 hash of the payload's serialized bytes.
packstore Package packstore persists Amber-Store CAS objects in log-structured, append-only segment (pack) files.	Package packstore persists Amber-Store CAS objects in log-structured, append-only segment (pack) files.
reference Package reference defines the named-pointer record: a global name pointing at a store key, with creator, creation time, and an optional opaque signature.	Package reference defines the named-pointer record: a global name pointing at a store key, with creator, creation time, and an optional opaque signature.
refstore Package refstore persists reference records in a Pebble DB: name bytes → CBOR record bytes, stored verbatim.	Package refstore persists reference records in a Pebble DB: name bytes → CBOR record bytes, stored verbatim.
tarexport Package tarexport traverses an Amber-Store CAS from a directory key and writes a PAX-format tar of the filesystem tree.	Package tarexport traverses an Amber-Store CAS from a directory key and writes a PAX-format tar of the filesystem tree.
tarextract Package tarextract extracts a PAX tar (as produced by tarexport) into a directory, restoring permissions, ownership (when running as root), extended attributes (best-effort), nanosecond mtimes, symlinks, fifos, and device nodes.	Package tarextract extracts a PAX tar (as produced by tarexport) into a directory, restoring permissions, ownership (when running as root), extended attributes (best-effort), nanosecond mtimes, symlinks, fifos, and device nodes.