bolttable

package module
v1.0.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

README

bolttable

Go Reference Go version Mutation

bolttable gives your Go code typed tables, declared indexes and a change feed over one bbolt file.

It replaces the hand-written bucket code, index upkeep and migration ladder that grow around a bbolt store. It depends only on bbolt and the standard library, needs Go 1.27.2 or later and is licensed under Apache-2.0.

Why use it

  • Tables are generic over key and value types. A key codec you choose sets the byte layout and the scan order, without reflection or struct tags.
  • An index maps a row to its entries in the row's transaction. Verify rescans every table, reports each index entry or counter that disagrees, and can rewrite it.
  • Each transaction that changes a non-silent table records one change at the next revision. Since returns changes after a revision you hold, and OnCommit receives them in order.
  • Each schema namespace has its own format digest. A changed declaration fails Open naming only that namespace, and ResetNamespaces empties it while the others keep their rows.
  • Stats reports size and free pages, Backup copies it online, and CompactIfFree shrinks it before open.

Consider storm or bolthold if you want tables from struct tags and queries built at run time, or bbolt alone for a few buckets.

Install

go get github.com/cplieger/bolttable@latest

Usage

var (
    schema   = bolttable.NewSchema("app")
    accounts = bolttable.Must(schema.Namespace("accounts"))
    users    = bolttable.Must(bolttable.Define(accounts, 0x01, "users",
        bolttable.Uint64Key{}, bolttable.JSON[user](1), bolttable.TableOptions{}))
)

db, err := bolttable.Open(path, schema, bolttable.Options{Retention: 1000})
if err != nil {
    return err
}
defer db.Close()

err = db.Update(ctx, func(tx *bolttable.Tx) error {
    id, err := users.NextID(tx)
    if err != nil {
        return err
    }
    return users.Put(tx, id, &user{Email: "ada@example.com"})
})

The package example on pkg.go.dev adds a unique index, a counter and OnCommit, and go test keeps it true.

API

  • NewSchema, Schema.Namespace, Define, DefineIndex, DefineCounter, CascadePrefix and CascadeIndex declare the layout. Must wraps them for package-level variables.
  • Open, DB.Update, DB.Batch and DB.View run transactions. Table has Get, Put, Delete, Scan, NextID and Subject. Index has Lookup and Scan, and Counter has Value.
  • Tx.Mark, Tx.Touch and Tx.AfterCommit add to a transaction's change. ReadTx.Since and ReadTx.Rev read the feed, and Options.BeforeCommit chooses what each change records.
  • DB.Verify, DB.Stats, DB.Backup, CompactIfFree and ResetNamespaces look after the file.
  • ErrFormat, ErrConflict, ErrCorrupt, ErrCommit and ErrPublished are the errors a caller can match, with *FormatError and *ConflictError carrying the details.

The full reference is on pkg.go.dev.

Contributing

Issues and pull requests are welcome. The shared contributing rules apply.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

Apache-2.0. See LICENSE.

Documentation

Overview

Package bolttable provides typed tables over a bbolt file: namespaces of tables with declared indexes, counters and cascades, a revision counter with a change feed, a per-namespace format digest with reset, and verification, statistics, online backup and offline compaction.

A Schema is declared once, usually through Must, and sealed by Open. Keys go through a KeyCodec, so their byte layout, and with it the scan order, is the caller's; values through a Codec, JSON by default. Every declaration that shapes stored bytes enters its namespace's digest, and Open refuses a file whose stored digests differ with a *FormatError.

A transaction that writes a non-silent table, Touches a subject or sets a Mark reserves the next revision and records one Change, which Since returns and OnCommit receives in revision order. Every slice the package returns or keeps is a copy.

Example
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/cplieger/bolttable"
)

type user struct {
	Email string `json:"email"`
	Admin bool   `json:"admin"`
}

var (
	schema   = bolttable.NewSchema("app")
	accounts = bolttable.Must(schema.Namespace("accounts"))
	users    = bolttable.Must(bolttable.Define(accounts, 0x01, "users",
		bolttable.Uint64Key{}, bolttable.JSON[user](1), bolttable.TableOptions{}))
	byEmail = bolttable.Must(bolttable.DefineIndex(users, "by_email",
		bolttable.IndexOptions{Unique: true, Version: 1},
		func(_ uint64, u *user) []bolttable.Entry { return []bolttable.Entry{{Key: []byte(u.Email)}} }))
	admins = bolttable.Must(bolttable.DefineCounter(users, "admins", 1,
		func(_ uint64, u *user) int64 {
			if u.Admin {
				return 1
			}
			return 0
		}))
)

func main() {
	dir, err := os.MkdirTemp("", "bolttable-example")
	if err != nil {
		log.Fatal(err)
	}
	defer func() { _ = os.RemoveAll(dir) }()
	db, err := bolttable.Open(filepath.Join(dir, "app.bolt"), schema, bolttable.Options{
		Retention: 1000,
		OnCommit:  func(c bolttable.Change) { fmt.Println("revision", c.Rev, "subjects", len(c.Subjects)) },
	})
	if err != nil {
		log.Fatal(err)
	}
	defer func() { _ = db.Close() }()
	ctx := context.Background()

	err = db.Update(ctx, func(tx *bolttable.Tx) error {
		id, err := users.NextID(tx)
		if err != nil {
			return err
		}
		return users.Put(tx, id, &user{Email: "ada@example.com", Admin: true})
	})
	if err != nil {
		log.Fatal(err)
	}

	err = db.View(ctx, func(r *bolttable.ReadTx) error {
		id, _, found, err := byEmail.Lookup(r, []byte("ada@example.com"))
		if err != nil || !found {
			return err
		}
		n, err := admins.Value(r)
		fmt.Println("user", id, "admins", n)
		return err
	})
	if err != nil {
		log.Fatal(err)
	}
}
Output:
revision 1 subjects 1
user 1 admins 1

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	// ErrFormat matches a *FormatError: the file's stored format differs from
	// the schema.
	ErrFormat = errors.New("bolttable: stored format differs from the schema")
	// ErrConflict matches a *ConflictError: a Put would give a unique index
	// key a second row.
	ErrConflict = errors.New("bolttable: unique index conflict")
	// ErrCorrupt reports stored bytes that do not decode. Its message names
	// the table or index and a fingerprint of the stored key, never key or
	// row bytes, and it drops the codec's own error, whose text may hold
	// them.
	ErrCorrupt = errors.New("bolttable: row does not decode")
	// ErrCommit wraps a failure of bbolt itself to begin or commit a
	// transaction, so a caller tells a storage fault from its own refusal.
	ErrCommit = errors.New("bolttable: commit failed")
	// ErrPublished reports that Backup linked a complete copy at its path,
	// then failed to remove its temporary name or to fsync the directory.
	ErrPublished = errors.New("bolttable: backup published, not made durable")
)

Functions

func CascadeIndex

func CascadeIndex[PK, PV, CK, CV any](parent *Table[PK, PV], via *Index[CK, CV], version uint32, prefix func(PK) []byte) error

CascadeIndex makes deleting a parent row delete, in the same transaction, every row of via's table holding an entry of via whose index key starts with prefix(parent key), and recursively their own cascades. Parent and index share one namespace.

func CascadePrefix

func CascadePrefix[PK, PV, CK, CV any](parent *Table[PK, PV], child *Table[CK, CV], version uint32, prefix func(PK) []byte) error

CascadePrefix makes deleting a parent row delete, in the same transaction, every child row whose encoded key starts with prefix(parent key), and recursively their own cascades. Parent and child share one namespace.

func CompactIfFree

func CompactIfFree(path string, minFreeRatio float64, minFreeBytes int64) (bool, error)

CompactIfFree rewrites the file at path without its free pages when they reach both minFreeRatio of the file and minFreeBytes, and reports whether it did. An absent path answers (false, nil). It runs before Open and needs the caller's exclusive access: bbolt's lock is a flock on the open file's inode (https://github.com/etcd-io/bbolt/blob/v1.5.0/bolt_unix.go#L32), so a process waiting in Open across the rename would acquire the old one. The copy is written to <path>.compact-tmp (a stale one is removed first), synced, and renamed over path, so a crash at any point leaves path whole.

func Must

func Must[T any](v T, err error) T

Must returns v, panicking with err when err is not nil; for package-level declarations.

func ResetNamespaces

func ResetNamespaces(path string, s *Schema, names []string) error

ResetNamespaces empties the named namespaces of the existing file at path, sealing s; pass it every name a *FormatError lists. It runs before Open and needs the caller's exclusive access. Each name must be stored or declared, or be "bolttable", which recreates the library's bucket (revision head and change log) at the current layout number. One bbolt transaction deletes the stored named buckets and recreates the declared ones, every table starting at the highest sequence its namespace stored, so NextID never reissues an id across a prefix move, a rename or a removed table.

Types

type BytesKey

type BytesKey struct{}

BytesKey stores a []byte key as is; Put refuses an empty one.

func (BytesKey) AppendKey

func (BytesKey) AppendKey(dst, k []byte) []byte

AppendKey appends k.

func (BytesKey) ParseKey

func (BytesKey) ParseKey(b []byte) ([]byte, error)

ParseKey returns a copy of b.

func (BytesKey) Version

func (BytesKey) Version() uint32

Version is 1.

type Change

type Change struct {
	Subjects []Subject
	Marks    []string
	Rev      uint64
}

Change is one committed revision: the subjects BeforeCommit returned (or the touched ones) and every mark.

type Codec

type Codec[V any] interface {
	Version() uint32
	Append(dst []byte, v *V) ([]byte, error)
	Decode(b []byte, v *V) error
}

Codec encodes a table's values. Version enters the namespace's format digest, so an encoding change bumps it. Append must be deterministic: a Put whose encoding equals the stored bytes writes nothing. Decode receives a copy it may keep.

func JSON

func JSON[V any](version uint32) Codec[V]

JSON returns a Codec over encoding/json at the given version.

type ConflictError

type ConflictError struct {
	Index string
}

ConflictError names the unique index a Put collided on; errors.Is(err, ErrConflict) matches it. The transaction stays usable: the refused Put wrote nothing.

func (*ConflictError) Error

func (e *ConflictError) Error() string

func (*ConflictError) Is

func (*ConflictError) Is(target error) bool

Is reports whether target is ErrConflict.

type Counter

type Counter[K, V any] struct {
	// contains filtered or unexported fields
}

Counter keeps the sum of a weight over every row of its table, updated in the same transaction as the row.

func DefineCounter

func DefineCounter[K, V any](t *Table[K, V], name string, version uint32, weight func(k K, v *V) int64) (*Counter[K, V], error)

DefineCounter declares a counter summing weight over the table's rows. It fails for a nil table or function, a zero version, a name already used in the namespace, and after Open.

func (*Counter[K, V]) Value

func (c *Counter[K, V]) Value(r *ReadTx) (int64, error)

Value returns the counter's total.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is an open bolttable file.

func Open

func Open(path string, s *Schema, o Options) (*DB, error)

Open opens or creates the file at path, mode 0600, under s, sealing s. A file whose stored format differs from s fails with a *FormatError and is left byte-for-byte unchanged; only when nothing mismatches does Open create the namespaces the file lacks. Open fails when another process holds the file for 5 s.

func (*DB) Backup

func (db *DB) Backup(ctx context.Context, path string) error

Backup writes a compacted copy of the file to path, mode 0600, from one read transaction while the database stays open. It walks into an exclusively created .<base>.bolttable-tmp beside path and publishes it with a hard link, which fails for any existing path, one created during the walk included. A failure before the link leaves no path and removes the temporary file, never one Backup did not create. After the link path is complete, and a failed removal of the temporary name or directory fsync wraps ErrPublished.

func (*DB) Batch

func (db *DB) Batch(ctx context.Context, fn func(*Tx) error) error

Batch is Update through bbolt's grouped commit: concurrent calls share one transaction and fsync. fn may run more than once (bbolt reruns a group after one call fails, and the failed call alone: https://github.com/etcd-io/bbolt/blob/v1.5.0/db.go#L1024); only its last run's writes, change and AfterCommit functions survive. Each call keeps its own subjects, marks and revision.

func (*DB) Close

func (db *DB) Close() error

Close closes the file, waiting for open transactions.

func (*DB) Epoch

func (db *DB) Epoch() [16]byte

Epoch returns the random identity of this open: a reader holding a revision from another epoch must resync.

func (*DB) Stats

func (db *DB) Stats(ctx context.Context) (Stats, error)

Stats reads the file's statistics in one read transaction, walking every table's pages.

func (*DB) Update

func (db *DB) Update(ctx context.Context, fn func(*Tx) error) error

Update runs fn in one bbolt read-write transaction. fn's error, or one from BeforeCommit, rolls it back and is returned as is; a failure of bbolt itself wraps ErrCommit. A context already done runs nothing.

func (*DB) Verify

func (db *DB) Verify(ctx context.Context, repair bool) (VerifyReport, error)

Verify rescans every table in one transaction and recomputes every index entry and counter. With repair it rewrites index entries and counters, and only those: it never rewrites or deletes a row, so an undecodable row stays reported with Repaired false, and its remedy is a restore. A repair writes no revision.

func (*DB) View

func (db *DB) View(ctx context.Context, fn func(*ReadTx) error) error

View runs fn in one read-only transaction. A context already done runs nothing.

type Entry

type Entry struct {
	Key, Value []byte
}

Entry is one index entry a row produces. Key is the index key; Value is the projection a lookup or scan returns without reading the row.

type FormatError

type FormatError struct {
	Namespaces []NamespaceDigest
}

FormatError names every namespace whose stored format differs from the schema; errors.Is(err, ErrFormat) matches it. Every name it lists is one ResetNamespaces accepts.

func (*FormatError) Error

func (e *FormatError) Error() string

func (*FormatError) Is

func (*FormatError) Is(target error) bool

Is reports whether target is ErrFormat.

type Index

type Index[K, V any] struct {
	// contains filtered or unexported fields
}

Index is a declared index over a table. Each row maps to any number of entries: none makes the index sparse, several make it per-element.

func DefineIndex

func DefineIndex[K, V any](t *Table[K, V], name string, o IndexOptions, entries func(k K, v *V) []Entry) (*Index[K, V], error)

DefineIndex declares an index whose entries entries computes from a row. It fails for a nil table or function, a zero version, a name already used in the namespace, and after Open.

func (*Index[K, V]) Lookup

func (ix *Index[K, V]) Lookup(r *ReadTx, key []byte) (k K, proj []byte, found bool, err error)

Lookup returns the row key and projection under a unique index key; found is false when no row has it. It fails on a non-unique index.

func (*Index[K, V]) Scan

func (ix *Index[K, V]) Scan(r *ReadTx, rg Range, fn func(ikey []byte, k K, proj []byte) (bool, error)) error

Scan calls fn for each entry whose index key falls in rg, in index-key order, until fn returns false or an error. Range compares index keys, so a resumable scan needs an index key unique per row.

type IndexOptions

type IndexOptions struct {
	Unique  bool
	Version uint32
}

IndexOptions configures an index. A Unique index gives each key at most one row. Version enters the namespace's format digest and must not be zero.

type KeyCodec

type KeyCodec[K any] interface {
	Version() uint32
	AppendKey(dst []byte, k K) []byte
	ParseKey(b []byte) (K, error)
}

KeyCodec lays a table's keys out as bytes; the byte order is the order Scan visits. Version enters the namespace's format digest, so a layout change bumps it. ParseKey receives a copy it may keep.

type Mismatch

type Mismatch struct {
	Name                        string
	Missing, Extra, Undecodable int
	Repaired                    bool
}

Mismatch is one table, index or counter, named namespace.name, whose stored state differs from a rescan of its rows. For a table, Undecodable counts rows that do not decode; their index entries count as neither missing nor extra. For an index, Missing and Extra count entries. For a counter, a stored total differing from the exact sum of its rows' weights, or a sum outside int64, is one Missing and one Extra, and Undecodable counts rows its sum could not include, which leaves it unchecked.

type Namespace

type Namespace struct {
	// contains filtered or unexported fields
}

Namespace is one top-level bucket holding its tables, with its own format digest, so one namespace's format change never resets another's rows. Prefixes are unique across the schema, so a Subject stays prefix plus key.

type NamespaceDigest

type NamespaceDigest struct {
	Name         string
	Stored, Want [32]byte
}

NamespaceDigest is one mismatched top-level bucket. A zero Want is a stored bucket the schema does not declare, or the library's own bucket; a zero Stored is a bucket holding no digest, or any bucket of a file whose layout is not this library's.

type NamespaceStats

type NamespaceStats struct {
	Name   string
	Tables []TableStats
	Format [32]byte
}

NamespaceStats is one namespace's stored format digest and its tables.

type Options

type Options struct {
	// BeforeCommit runs once per revision-reserving transaction, before bbolt
	// commits, and returns the subjects to record. It may write silent tables
	// and add marks; a non-silent write or Touch fails the transaction, and
	// its error rolls everything back.
	BeforeCommit func(*Tx, *Pending) ([]Subject, error)
	// OnCommit receives each committed change in revision order. It must not
	// block: it runs under the lock that orders deliveries.
	OnCommit func(Change)
	// Retention is how many of the newest changes Since can return; it must
	// not be negative.
	Retention int
}

Options configures an open DB. The zero value records Touched as each change's subjects, delivers nothing and retains no change, so every Since below the head answers full.

type Pending

type Pending struct {
	Touched []Subject
	Marks   []string
	Rev     uint64
}

Pending is the revision a transaction is about to commit, handed to BeforeCommit. Its slices are copies.

type Range

type Range struct {
	Prefix  []byte
	From    []byte
	Reverse bool
}

Range selects the keys a Scan visits: those starting with Prefix, from From onward (inclusive; when Reverse, From is the inclusive upper bound), in byte order or, when Reverse, its reverse. Nil Prefix and From select every key.

type ReadTx

type ReadTx struct {
	// contains filtered or unexported fields
}

ReadTx reads one consistent snapshot. It is valid only inside the function View passed it to, or while its Tx is.

func (*ReadTx) Rev

func (r *ReadTx) Rev() uint64

Rev returns the head revision: the newest committed one, 0 before any.

func (*ReadTx) Since

func (r *ReadTx) Since(after uint64) ([]Change, bool, error)

Since returns every retained change with Rev above after, in order. full reports that the log cannot answer: a change after after has been trimmed (after+1 is below the oldest retained revision), or after is above the head; changes is then nil.

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

Schema is the set of namespaces a file holds. Declarations build it before the first Open or ResetNamespaces; either seals it, and a declaration after that fails.

func NewSchema

func NewSchema(name string) *Schema

NewSchema returns an empty schema. Its name enters every namespace's format digest.

func (*Schema) Namespace

func (s *Schema) Namespace(name string) (*Namespace, error)

Namespace declares a namespace. It fails for an empty or duplicate name, the library's own bucket name "bolttable", and after Open.

type Stats

type Stats struct {
	Namespaces []NamespaceStats
	Rev        uint64
	Oldest     uint64
	Commits    uint64
	FileBytes  int64
	FreeBytes  int64
}

Stats describes an open file. Rev and Oldest bound what Since returns. Commits counts every successful bbolt commit since Open: one per Batch group, and silent-only, byte-equal and empty writes too, since bbolt commits a meta page for each; failed and rolled-back transactions do not count (https://github.com/etcd-io/bbolt/blob/v1.5.0/tx.go#L267).

type Subject

type Subject []byte

Subject names one row in the change feed: a table's prefix byte plus the encoded key.

type Table

type Table[K, V any] struct {
	// contains filtered or unexported fields
}

Table is a typed table declared in a namespace.

func Define

func Define[K, V any](ns *Namespace, prefix byte, name string, k KeyCodec[K], v Codec[V], o TableOptions) (*Table[K, V], error)

Define declares a table under a prefix byte unique across the schema and a name unique in its namespace. It fails for a nil namespace or codec, a zero codec version, a duplicate prefix or name, and after Open.

func (*Table[K, V]) Delete

func (t *Table[K, V]) Delete(tx *Tx, k K) (bool, error)

Delete removes the row under k, its index entries and counter weights, and runs the table's cascades; it reports whether a row was there.

func (*Table[K, V]) Get

func (t *Table[K, V]) Get(r *ReadTx, k K) (v V, found bool, err error)

Get returns the row under k; found is false for an absent row. A stored value that does not decode is ErrCorrupt.

func (*Table[K, V]) NextID

func (t *Table[K, V]) NextID(tx *Tx) (uint64, error)

NextID returns the table's next sequence number, starting at 1. It records no subject; on a non-silent table it fails the transaction inside BeforeCommit, as a write does. After ResetNamespaces it stays above every id the namespace issued before.

func (*Table[K, V]) Put

func (t *Table[K, V]) Put(tx *Tx, k K, v *V) error

Put stores v under k, updating every index and counter of the table in the same transaction. A value whose encoding equals the stored bytes writes nothing and records no subject. It fails with a *ConflictError when a unique index key belongs to another row, writing nothing; a counter leaving int64 fails it and the whole transaction.

func (*Table[K, V]) Scan

func (t *Table[K, V]) Scan(r *ReadTx, rg Range, fn func(K, *V) (bool, error)) error

Scan calls fn for each row in rg until fn returns false or an error. A key or value that does not decode stops the scan with ErrCorrupt. Inside a write transaction fn may write the table it scans.

func (*Table[K, V]) Subject

func (t *Table[K, V]) Subject(k K) Subject

Subject returns the change-feed subject of the row under k.

type TableOptions

type TableOptions struct {
	Silent bool
}

TableOptions configures a table. A Silent table's writes reserve no revision, record no subject and never reach Since; a non-silent write inside BeforeCommit fails the transaction.

type TableStats

type TableStats struct {
	Name  string
	Rows  int
	Bytes int64
}

TableStats is one table's row count and the bytes its pages use.

type Tx

type Tx struct {
	// contains filtered or unexported fields
}

Tx is a read-write transaction. It is valid only inside the function Update or Batch passed it to.

func (*Tx) AfterCommit

func (tx *Tx) AfterCommit(fn func())

AfterCommit registers fn to run once after this transaction commits, never after a rollback or a discarded Batch run; for signals about silent writes.

func (*Tx) Mark

func (tx *Tx) Mark(m string)

Mark records a mark in this transaction's change; a transaction with a mark reserves a revision even when it wrote nothing.

func (*Tx) Read

func (tx *Tx) Read() *ReadTx

Read returns a ReadTx over this transaction, seeing its own writes.

func (*Tx) Touch

func (tx *Tx) Touch(s Subject)

Touch records s as a subject of this transaction's change without a write. Inside BeforeCommit it fails the transaction.

type Uint64Key

type Uint64Key struct{}

Uint64Key encodes a uint64 big-endian, so byte order is numeric order.

func (Uint64Key) AppendKey

func (Uint64Key) AppendKey(dst []byte, k uint64) []byte

AppendKey appends k's 8 big-endian bytes.

func (Uint64Key) ParseKey

func (Uint64Key) ParseKey(b []byte) (uint64, error)

ParseKey refuses anything but 8 bytes.

func (Uint64Key) Version

func (Uint64Key) Version() uint32

Version is 1.

type VerifyReport

type VerifyReport struct {
	Mismatches []Mismatch
}

VerifyReport lists every mismatch Verify found.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL