lode

README

Lode

Structured persistence for object storage.

Lode is an embeddable Go persistence framework that provides snapshots, metadata, and safe write semantics on object storage, without requiring a database. It offers Datasets, which hold structured collections of named objects, and Volumes, which store sparse, resumable byte ranges and reside in an experimental package (lode/volume).

---

Quick Start

Install

go get github.com/pithecene/lode

Writing and Reading a Blob

package main

import (
    "context"
    "fmt"
    "github.com/pithecene/lode/lode"
)

func main() {
    ctx := context.Background()

    // Open a root over filesystem storage, and a blob dataset in it.
    blobs, _ := lode.Open(lode.FS("/tmp/lode-demo")).Blobs(ctx, "mydata")

    // Write a blob: one []byte per snapshot.
    snap, _ := blobs.Write(ctx, []byte("hello world"), lode.D{"source": "demo"})
    fmt.Println("Created snapshot:", snap.ID)

    // Read the blob back.
    data, _ := blobs.Read(ctx, snap.ID)
    fmt.Println("Data:", string(data))
}

Writing Structured Records

// The codec fixes the record type: here Event, encoded as JSON Lines.
type Event struct {
    ID    int    `json:"id"`
    Event string `json:"event"`
    Day   string `json:"day"`
}

// Batch is the snapshot's metadata.
type Batch struct {
    Number int `json:"number"`
}

root := lode.Open(lode.FS("/tmp/lode-demo"))
events, _ := root.Records(ctx, "events", lode.JSONL[Event](),
    lode.WithLayout(lode.Hive("day").By(func(e Event) lode.Partition {
        return lode.Partition{"day": e.Day}
    })))

snap, _ := events.Write(ctx, []Event{
    {ID: 1, Event: "signup", Day: "2026-10-22"},
    {ID: 2, Event: "login", Day: "2026-10-22"},
}, Batch{Number: 1})
got, _ := events.Read(ctx, snap.ID) // []Event

var batch Batch
_ = snap.Metadata.Decode(&batch)

// A reader opens a view: it takes the layout from the manifests.
view, _ := lode.OpenReadOnly(lode.FS("/tmp/lode-demo")).Records(ctx, "events", lode.JSONL[Event]())

Lode stores data on the filesystem (lode.FS), in memory (lode.Memory), or on S3-compatible storage (s3.Factory). The storage location must exist before the caller opens a dataset (see Storage Prerequisites). PUBLIC_API.md describes streaming writes and the experimental Volume API, and examples/ contains complete programs.

---

Core Ideas

A Dataset, which holds structured collections, follows these rules:

• Each write produces a new, immutable snapshot.
• Each snapshot references a manifest that describes its data files and metadata.
• A caller chooses storage, format, compression, and partitioning independently of one another.

A Volume, which stores sparse byte ranges and resides in the experimental package lode/volume, follows these rules:

• A Volume is a sparse, resumable byte address space.
• Commits record which ranges exist, and Lode never fills a gap with zeros.
• Each commit produces a cumulative manifest.

Datasets and Volumes share these rules:

• A snapshot becomes visible only after Lode persists its manifest.
• Lode persists the metadata that the caller supplies and never infers metadata.
• Every store detects concurrent writers through compare-and-swap (CAS).

---

Guarantees

• Immutable snapshots: Lode never modifies a data file or manifest after it persists the object.
• Atomic commits: a snapshot is visible only once its manifest exists.
• Explicit metadata: every snapshot carries metadata that the caller supplies, and Lode never infers metadata.
• Safe writes: Lode never overwrites an existing object, and every store detects concurrent writers through CAS.

Lode does not coordinate distributed writers, execute queries, compact data, or delete objects unless the caller requests deletion. The concepts guide defines each guarantee and each exclusion.

Lode has two limitations by design. A backend that lacks conditional writes supports writers in one process only, through SingleProcess, so writers in separate processes require external coordination (Concurrency and CAS). On an S3-compatible service other than AWS S3, the guarantees depend on conditional writes and consistency that Lode does not verify (Other S3-Compatible Services).

---

Documentation

Pithecene publishes the guides, the public API reference, and the contracts at pithecene.sh/docs/lode. The repository holds the same documents:

Topic	Where
Storage model, commits, and write API selection	Concepts guide
Rules of the API that most often surprise a caller	Usage pitfalls guide
Complete API usage and options	PUBLIC_API.md
Expiring snapshots and reclaiming residual data	Retention guide
Snapshot ID format and ordering	Snapshot ID guide
Normative behavior	docs/contracts/
Runnable programs	examples/
Performance	docs/BENCHMARKS.md
Documentation conventions, for maintainers	docs/maintainers/

---

Status

The current release of Lode is v0.13.0. docs/plans/V1_API_DECISIONS.md lists the breaking changes still under consideration for v1.0, and CHANGELOG.md describes each release.

---

License

Pithecene LLC distributes Lode under the Apache License 2.0 (see LICENSE). Copyright © 2026 Pithecene LLC.

Pithecene answers inquiries at contact@pithecene.sh. SECURITY.md describes the procedure for reporting a vulnerability.