lode

README

Lode

Structured persistence for object storage.

Lode is an embeddable Go persistence framework that provides snapshots, metadata, and safe write semantics on object storage, without requiring a database. It offers Datasets for structured collections of named objects and, in an experimental package (lode/volume), Volumes for sparse, resumable storage of byte ranges.

---

Quick Start

Install

go get github.com/pithecene/lode

Writing and Reading a Blob

package main

import (
    "context"
    "fmt"
    "github.com/pithecene/lode/lode"
)

func main() {
    ctx := context.Background()

    // Create a dataset on filesystem storage.
    ds, _ := lode.NewDataset("mydata", lode.NewFSFactory("/tmp/lode-demo"))

    // Write a blob (by default, raw bytes without a codec).
    snap, _ := ds.Write(ctx, []any{[]byte("hello world")}, lode.Metadata{"source": "demo"})
    fmt.Println("Created snapshot:", snap.ID)

    // Read the blob back.
    data, _ := ds.Read(ctx, snap.ID)
    fmt.Println("Data:", string(data[0].([]byte)))
}

Writing Structured Records

// With a codec, Write accepts structured records.
ds, _ := lode.NewDataset("events",
    lode.NewFSFactory("/tmp/lode-demo"),
    lode.WithCodec(lode.NewJSONLCodec()),
)

records := lode.R(
    lode.D{"id": 1, "event": "signup"},
    lode.D{"id": 2, "event": "login"},
)
snap, _ := ds.Write(ctx, records, lode.Metadata{"batch": "1"})

Lode stores data on the filesystem (NewFSFactory), in memory (NewMemoryFactory), or on S3-compatible storage (lode/s3). The storage location must exist before a dataset is constructed (see Storage Prerequisites). Streaming writes and the experimental Volume API are described in PUBLIC_API.md, and complete programs are listed in examples/.

---

Core Ideas

Dataset (structured collections):

• Each write produces a new, immutable snapshot.
• Each snapshot references a manifest that describes its data files and metadata.
• Storage, format, compression, and partitioning are independent.

Volume (sparse byte ranges; experimental package lode/volume):

• A volume is a sparse, resumable byte address space.
• Commits record which ranges exist; gaps are never filled with zeros.
• Each commit produces a cumulative manifest.

Both:

• A snapshot is visible only once its manifest is persisted.
• Metadata is explicit and persisted, never inferred.
• Every store detects concurrent writers through CAS.

---

Guarantees

• Immutable snapshots: data files and manifests never change once written.
• Atomic commits: a snapshot is visible only once its manifest exists.
• Explicit metadata: every snapshot carries metadata supplied by the caller, never inferred.
• Safe writes: existing objects are never overwritten, and every store detects concurrent writers through CAS.

Lode does not coordinate distributed writers, execute queries, compact data, or remove anything without being asked; the concepts guide gives the details.

---

Documentation

Topic	Where
Storage model, commits, and choosing a write API	Concepts guide
Complete API usage, options, and common pitfalls	PUBLIC_API.md
Expiring snapshots and reclaiming residual data	Retention guide
Snapshot ID format and ordering	Snapshot ID guide
Normative behavior	docs/contracts/
Runnable programs	examples/
Performance	docs/BENCHMARKS.md

---

Status

Lode is at v0.11.0 and under active development. Its APIs are stabilizing; the breaking changes still under consideration for v1.0 are listed in docs/V1_API_DECISIONS.md, and each release is described in CHANGELOG.md.

---

License

Apache License 2.0 (see LICENSE). Copyright © 2026 Pithecene LLC.

Contact: contact@pithecene.sh. To report a vulnerability, follow SECURITY.md.