bolttable

package module
v1.2.0-dev.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 19, 2026 License: Apache-2.0 Imports: 30 Imported by: 0

README

bolttable

Go Reference Go version Mutation

bolttable gives your Go code typed tables, declared indexes and a change feed over one bbolt file.

It replaces the hand-written bucket code, index upkeep, migration ladder and startup steps that grow around a bbolt store. It depends only on bbolt, golang.org/x/sync, golang.org/x/sys and the standard library and is licensed under Apache-2.0. It runs on Linux, macOS and the BSDs; on Windows and other platforms, Open, OpenReadOnly and LockPath return an error wrapping errors.ErrUnsupported.

Why use it

  • Tables are generic over key and value types. A key codec you choose sets the byte layout and the scan order, without reflection or struct tags.
  • An index maps a row to its entries in the row's transaction. Verify rescans every table, reports each index entry or counter that disagrees, and can rewrite it.
  • Each transaction that changes a non-silent table records one change at the next revision. Since returns changes after a revision you hold, a page at a time, and OnCommit receives every commit in commit order.
  • Each schema namespace has its own format digest. A changed declaration fails Open naming only that namespace, unless your reset policy allows it: Open then keeps a copy of the file, empties that namespace, and the others keep their rows.
  • Open takes a lock file beside the data file and does every startup step under it. It removes stranded temporary files, runs your own check of the stored file before it changes anything, can move a corrupt or truncated file aside, resets, compacts and reports what it did. A second process gets ErrLocked after a wait you choose.
  • Live is one handle to a live file that every store holds in place of a *DB. Swap replaces the file under all of them while it stays locked, and hooks you register reload what you keep in memory at each open. OpenFailure turns an open error into a fixed class (damaged, format, newer, unreadable or unwritable) for an outage page or health check that must not show error text.
  • OpenReadOnly opens a file for a report or a restore check and has no write method. Space reads the file's size, the bytes its data uses and the bytes bbolt can reuse, without a transaction, Backup copies the file while it stays open, and WriteHealth tracks, per class of write, whether commits are failing and since when, for a health check.

Consider storm or bolthold if you want tables from struct tags and queries built at run time, or bbolt alone for a few buckets.

Install

go get github.com/cplieger/bolttable@latest

Usage

var (
    schema   = bolttable.NewSchema("app")
    accounts = bolttable.Must(schema.Namespace("accounts"))
    users    = bolttable.Must(bolttable.Define(accounts, 0x01, "users",
        bolttable.Uint64Key{}, bolttable.JSON[user](1), bolttable.TableOptions{}))
)

db, err := bolttable.Open(path, schema, bolttable.Options{
    Retention: 1000,
    LockWait:  5 * time.Second,
    // A format change in a namespace you can rebuild resets it, keeping two
    // copies of the file; any other format change fails Open.
    Reset: bolttable.ResetPolicy{
        Allow: func(n bolttable.NamespaceDigest) bool { return n.Name == "cache" },
        Keep:  2,
    },
})
if err != nil {
    return err
}
defer db.Close()

err = db.Update(ctx, func(tx *bolttable.Tx) error {
    id, err := users.NextID(tx)
    if err != nil {
        return err
    }
    return users.Put(tx, id, &user{Email: "ada@example.com"})
})

The package example on pkg.go.dev adds a unique index, a counter, a subject that names no row, OnCommit and a page of the feed, and go test keeps it true.

Replacing a live file

When a process must replace its own file, a restore from backup for instance, build its stores on a Live instead of a *DB:

live, err := bolttable.NewLive(path, schema, bolttable.Options{LockWait: 5 * time.Second})
if err != nil {
    return err // an invalid option, or another process holds the lock
}
defer live.Close()
live.OnOpen(reloadCache) // runs at every open, before any call is admitted
if err := live.Open(); err != nil {
    class, _ := bolttable.OpenFailure(err)
    slog.Error("store unavailable", "class", class, "err", err)
}

// Later, put a staged copy in place of the file.
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
err = live.Swap(ctx, func() error { return os.Rename(staged, path) })

NewLive holds the lock for the handle's whole life: the Options.Lock you pass, which it never releases, or one it takes, which Close releases. Until Close, that lock's Release fails, and an Open or a second NewLive under it returns ErrLocked. A handle whose file failed to open stays usable: its calls return ErrClosed and Err says why, until an Open or a Swap opens it.

During a Swap, every call on the handle returns ErrClosed at once. Swap waits for the calls already running, then closes the file, runs your rename, reopens the file with the same schema and options, and runs the OnOpen hooks before it admits calls again. The wait is cooperative: no bbolt transaction reads a context inside its function, so ctx bounds only how long Swap waits and never cuts a running call short. When ctx ends first, Swap closes nothing, admits calls again and returns ctx's error. A failed rename, open or hook leaves the handle closed with Err set. A Swap on a closed handle skips the close, so it can also replace a file that never opened.

A writer that runs on its own schedule should treat ErrClosed as a skipped write, not a failure. A size check reads Space through the handle, which answers the open file's sizes, or ErrClosed while no file is open or a swap is running. Work that reads in one transaction and writes in a later one takes Pin at its start and runs on that *DB: after a swap it returns ErrClosed, so nothing read from the old file lands in the new one.

Options.Inspect checks a stored file in a read-only transaction before Open resets, compacts or creates anything, and its error refuses the open with the file left as it was. A damaged page the check reads is ErrInvalidFile, which Options.InvalidFile handles like any unreadable file, while a panic in your own check reaches you. A check that finds a version stamp above the running build's own wraps ErrNewerBuild, which OpenFailure reports as newer.

API

  • NewSchema, Schema.Namespace, Define, DefineIndex, DefineCounter, CascadePrefix and CascadeIndex declare the layout. Must wraps them for package-level variables, and Schema.Prefixes lists the prefixes of the tables that record changes.
  • Open runs the open procedure that Options.LockWait, Lock, Reset, Compact, InvalidFile and Inspect set, and DB.OpenReport tells what it did. LockPath takes the same lock for your own file operations, and OpenReadOnly returns a *ReadDB.
  • NewLive returns a *Live, the swappable handle, with OnOpen, Open, Err, Refuse, View, Update, Batch, Backup, Space, Epoch, Pin, Swap and Close. Epoch stays the old file's until a Swap or Close has closed it, which waits for its transactions and their commit callbacks, and is zero while no file is open.
  • DB.Update, DB.Batch and DB.View run transactions. Table has Get, Put, Delete, Scan, NextID, Subject and Prefix. Index has Lookup and Scan, and Counter has Value. A Range scans a prefix, a [From, To) span or one Exact key.
  • Tx.Mark, Tx.Touch and Tx.AfterCommit add to a transaction's change, and Synthetic builds a subject that names no row. ReadTx.Since pages the feed and ReadTx.Rev reads its head. Options.BeforeCommit chooses what each change records.
  • DB.Verify, DB.Stats, DB.Space and DB.Backup look after the file. WriteHealth records write outcomes, and Options.CommitFault lets a test fail chosen commits.
  • ErrFormat, ErrConflict, ErrCorrupt, ErrCommit, ErrPublished, ErrLocked, ErrInvalidFile, ErrReset, ErrClosed and ErrNewerBuild are the errors a caller can match, with *FormatError and *ConflictError carrying the details. DB.Close waits for the writes in progress, their OnCommit and AfterCommit callbacks included, so a callback that closes its own DB, or closes or swaps a Live serving it, deadlocks, and so does one that opens or refuses that Live while it is being closed or swapped. After DB.Close, every call that returns an error returns one matching both ErrClosed and bbolt's ErrDatabaseNotOpen. OpenFailure maps an open error to its outage class.

The full reference is on pkg.go.dev.

Contributing

Issues and pull requests are welcome. The shared contributing rules apply.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI assistance. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

Apache-2.0. See LICENSE.

Documentation

Overview

Package bolttable provides typed tables over a bbolt file: namespaces of tables with declared indexes, counters and cascades, a revision counter with a change feed, a per-namespace format digest with reset, and verification, statistics, online backup and compaction at open.

A Schema is declared once, usually through Must, and sealed by Open. Keys go through a KeyCodec, so their byte layout, and with it the scan order, is the caller's; values through a Codec, JSON by default. Every declaration that shapes stored bytes enters its namespace's digest, and Open refuses a file whose stored digests differ with a *FormatError unless its ResetPolicy allows the reset.

Open takes the exclusive lock on <path>.lock, a file it never removes, and under it removes stranded temporary files, inspects the file without writing, moves an invalid file aside or resets a changed namespace when Options allow it, compacts, and opens the file for writing; OpenReport tells what it did. OpenReadOnly holds the same lock shared when <path>.lock exists, leaves an absent lock file absent, and writes nothing. LockPath takes the lock for a caller's own file operations, and Options.Lock runs Open under it.

A Live is the one handle to a live file that every store of it holds. Swap replaces the file under the handle: calls meanwhile answer ErrClosed, calls in flight finish first, and OnOpen hooks run at each open before any call is admitted. OpenFailure names an open error's outage class.

A transaction that writes a non-silent table, Touches a subject or sets a Mark reserves the next revision and records one Change, which Since returns and OnCommit receives in revision order. Every slice the package returns or keeps is a copy.

Example
package main

import (
	"bytes"
	"context"
	"fmt"
	"log"
	"os"
	"path/filepath"
	"slices"
	"time"

	"github.com/cplieger/bolttable"
)

type user struct {
	Email string `json:"email"`
	Admin bool   `json:"admin"`
}

var (
	schema   = bolttable.NewSchema("app")
	accounts = bolttable.Must(schema.Namespace("accounts"))
	users    = bolttable.Must(bolttable.Define(accounts, 0x01, "users",
		bolttable.Uint64Key{}, bolttable.JSON[user](1), bolttable.TableOptions{}))
	byEmail = bolttable.Must(bolttable.DefineIndex(users, "by_email",
		bolttable.IndexOptions{Unique: true, Version: 1},
		func(_ uint64, u *user) []bolttable.Entry { return []bolttable.Entry{{Key: []byte(u.Email)}} }))
	admins = bolttable.Must(bolttable.DefineCounter(users, "admins", 1,
		func(_ uint64, u *user) int64 {
			if u.Admin {
				return 1
			}
			return 0
		}))
)

func main() {
	dir, err := os.MkdirTemp("", "bolttable-example")
	if err != nil {
		log.Fatal(err)
	}
	defer func() { _ = os.RemoveAll(dir) }()
	db, err := bolttable.Open(filepath.Join(dir, "app.bolt"), schema, bolttable.Options{
		Retention: 1000,
		LockWait:  5 * time.Second,
		// A format change in a namespace the app can rebuild resets it,
		// keeping two copies of the file; any other one fails Open.
		Reset: bolttable.ResetPolicy{
			Allow: func(n bolttable.NamespaceDigest) bool { return n.Name == "cache" },
			Keep:  2,
		},
		OnCommit: func(c bolttable.Change) { fmt.Println("revision", c.Rev, "subjects", len(c.Subjects)) },
	})
	if err != nil {
		log.Fatal(err)
	}
	defer func() { _ = db.Close() }()
	ctx := context.Background()

	err = db.Update(ctx, func(tx *bolttable.Tx) error {
		id, err := users.NextID(tx)
		if err != nil {
			return err
		}
		tx.Touch(bolttable.Synthetic([]byte("signups")))
		return users.Put(tx, id, &user{Email: "ada@example.com", Admin: true})
	})
	if err != nil {
		log.Fatal(err)
	}

	err = db.View(ctx, func(r *bolttable.ReadTx) error {
		id, _, found, err := byEmail.Lookup(r, []byte("ada@example.com"))
		if err != nil || !found {
			return err
		}
		n, err := admins.Value(r)
		if err != nil {
			return err
		}
		fmt.Println("user", id, "admins", n)
		page, err := r.Since(0, 100)
		for _, c := range page.Changes {
			fmt.Printf("change %d: %d subjects, signups touched %v\n", c.Rev, len(c.Subjects), slices.ContainsFunc(c.Subjects, isSignups))
		}
		return err
	})
	if err != nil {
		log.Fatal(err)
	}
}

func isSignups(s bolttable.Subject) bool {
	return bytes.Equal(s, bolttable.Synthetic([]byte("signups")))
}
Output:
revision 1 subjects 2
user 1 admins 1
change 1: 2 subjects, signups touched true

Index

Examples

Constants

View Source
const SyntheticPrefix byte = 0xFF

SyntheticPrefix starts every subject that names no row. Define refuses it.

Variables

View Source
var (
	// ErrFormat matches a *FormatError: the file's stored format differs from
	// the schema.
	ErrFormat = errors.New("bolttable: stored format differs from the schema")
	// ErrConflict matches a *ConflictError: a Put would give a unique index
	// key a second row.
	ErrConflict = errors.New("bolttable: unique index conflict")
	// ErrCorrupt reports stored bytes that do not decode. Its message names
	// the table or index and a fingerprint of the stored key, never key or
	// row bytes, and it drops the codec's own error, whose text may hold
	// them.
	ErrCorrupt = errors.New("bolttable: row does not decode")
	// ErrCommit wraps a failure of bbolt itself to begin or commit a
	// transaction, so a caller tells a storage fault from its own refusal. A
	// failed page write keeps its *os.PathError and a failed fdatasync its
	// syscall.Errno, but bbolt flattens a failure to grow the file to text
	// (https://github.com/etcd-io/bbolt/blob/v1.5.0/db.go#L1245), so a caller
	// classing causes by errno falls back to "other" when none matches. A
	// call on a closed DB answers ErrClosed instead.
	ErrCommit = errors.New("bolttable: commit failed")
	// ErrPublished reports that Backup linked a complete copy at its path,
	// then failed to remove its temporary name or to fsync the directory.
	ErrPublished = errors.New("bolttable: backup published, not made durable")
	// ErrLocked reports that another process, or another handle in this one,
	// holds the file.
	ErrLocked = errors.New("bolttable: database file is locked")
	// ErrInvalidFile reports a file bbolt cannot read as a database: both
	// meta pages rejected, a page size too small to hold a meta page, fewer
	// bytes than its meta records, a freelist page bbolt cannot load or that
	// lists a page it must never reuse, or a page read that faulted or
	// panicked; bbolt's "file size too small" stays unclassified. A panic
	// inside bbolt's open, or a rollback that panics twice, leaves the file
	// mapped but not locked, and bbolt walks a NoFreelistSync file on an
	// unguarded goroutine.
	ErrInvalidFile = errors.New("bolttable: invalid database file")
	// ErrReset reports that Open could not run a reset Options.Reset allowed:
	// keeping the copy, or the reset transaction, failed, and the file is
	// left as it was. A file found unreadable during the reset is
	// ErrInvalidFile instead.
	ErrReset = errors.New("bolttable: reset failed")
	// ErrClosed reports a call on a closed DB or ReadDB, where it also
	// matches bbolt's ErrDatabaseNotOpen, and a call on a Live that is closed
	// or swapping. A writer that runs on its own
	// schedule reads it as a skipped write, never as a storage fault: it
	// never wraps ErrCommit.
	ErrClosed = errors.New("bolttable: database is closed")
	// ErrNewerBuild is what an Options.Inspect check wraps when the file
	// carries a store version stamp above the running build's own.
	ErrNewerBuild = errors.New("bolttable: file written by a newer build")
)

Functions

func CascadeIndex

func CascadeIndex[PK, PV, CK, CV any](parent *Table[PK, PV], via *Index[CK, CV], version uint32, prefix func(PK) []byte) error

CascadeIndex makes deleting a parent row delete, in the same transaction, every row of via's table holding an entry of via whose index key starts with prefix(parent key), and recursively their own cascades. Parent and index share one namespace.

func CascadePrefix

func CascadePrefix[PK, PV, CK, CV any](parent *Table[PK, PV], child *Table[CK, CV], version uint32, prefix func(PK) []byte) error

CascadePrefix makes deleting a parent row delete, in the same transaction, every child row whose encoded key starts with prefix(parent key), and recursively their own cascades. Parent and child share one namespace.

func Must

func Must[T any](v T, err error) T

Must returns v, panicking with err when err is not nil; for package-level declarations.

func OpenFailure

func OpenFailure(err error) (class string, ok bool)

OpenFailure names why a file cannot open as the outage class an app reports in place of the error, checked in this order: "newer" for ErrNewerBuild, "damaged" for ErrInvalidFile or ErrCorrupt, "format" for ErrFormat, "unwritable" for ErrReset or ErrCommit, which a hook's write at open returns, and "unreadable" for any other error. ok is false for nil and for ErrLocked, which is no outage.

Types

type BytesKey

type BytesKey struct{}

BytesKey stores a []byte key as is; Put refuses an empty one.

func (BytesKey) AppendKey

func (BytesKey) AppendKey(dst, k []byte) []byte

AppendKey appends k.

func (BytesKey) ParseKey

func (BytesKey) ParseKey(b []byte) ([]byte, error)

ParseKey returns a copy of b.

func (BytesKey) Version

func (BytesKey) Version() uint32

Version is 1.

type Change

type Change struct {
	Subjects []Subject
	Marks    []string
	Rev      uint64
}

Change is one committed revision: the subjects BeforeCommit returned (or the touched ones) and every mark.

type ChangePage

type ChangePage struct {
	// Changes are in revision order; empty, never nil, unless Full.
	Changes []Change
	// Full reports that the log cannot answer from after: a change after it
	// has been trimmed (after+1 is below the oldest retained revision), or
	// after is above the head. Changes is then nil and More false.
	Full bool
	// More reports that retained changes follow the last one returned.
	More bool
}

ChangePage is one page of the change log.

type Codec

type Codec[V any] interface {
	Version() uint32
	Append(dst []byte, v *V) ([]byte, error)
	Decode(b []byte, v *V) error
}

Codec encodes a table's values. Version enters the namespace's format digest, so an encoding change bumps it. Append must be deterministic: a Put whose encoding equals the stored bytes writes nothing. Decode receives a copy it may keep.

func JSON

func JSON[V any](version uint32) Codec[V]

JSON returns a Codec over encoding/json at the given version.

type CompactPolicy

type CompactPolicy struct {
	MinFreeRatio float64
	MinFreeBytes int64
}

CompactPolicy is when Open rewrites the file without its free pages: when its free bytes, free and pending pages times the page size, reach both MinFreeRatio of the file size and MinFreeBytes. The zero value never compacts. MinFreeRatio must lie in [0, 1] and MinFreeBytes must not be negative. A compaction that fails leaves the file whole and does not fail Open: OpenReport.CompactErr holds it.

type ConflictError

type ConflictError struct {
	Index string
}

ConflictError names the unique index a Put collided on; errors.Is(err, ErrConflict) matches it. The transaction stays usable: the refused Put wrote nothing.

func (*ConflictError) Error

func (e *ConflictError) Error() string

func (*ConflictError) Is

func (*ConflictError) Is(target error) bool

Is reports whether target is ErrConflict.

type Counter

type Counter[K, V any] struct {
	// contains filtered or unexported fields
}

Counter keeps the sum of a weight over every row of its table, updated in the same transaction as the row.

func DefineCounter

func DefineCounter[K, V any](t *Table[K, V], name string, version uint32, weight func(k K, v *V) int64) (*Counter[K, V], error)

DefineCounter declares a counter summing weight over the table's rows. It fails for a nil table or function, a zero version, a name already used in the namespace, and after Open.

func (*Counter[K, V]) Value

func (c *Counter[K, V]) Value(r *ReadTx) (int64, error)

Value returns the counter's total.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is an open bolttable file.

func Open

func Open(path string, s *Schema, o Options) (*DB, error)

Open opens or creates the file at path, mode 0600, under s, sealing s. Under the exclusive lock on path, taken or adopted from o.Lock and held until Close, it removes stranded temporary files, inspects a stored file without writing, applies o.InvalidFile, o.Reset and o.Compact, and creates the namespaces the file lacks; OpenReport tells what it did. A format change o.Reset refuses fails with the *FormatError, the file unchanged, and another holder of the lock with ErrLocked. The path follows LockPath's rule and the lock its filesystem and platform limits.

func (*DB) Backup

func (db *DB) Backup(ctx context.Context, path string) error

Backup writes a compacted copy of the file to path, mode 0600, from one read transaction while the database stays open. It walks into an exclusively created .<base>.bolttable-tmp beside path and publishes it with a hard link, which fails for any existing path, one created during the walk included. A failure before the link leaves no path and removes the temporary file, never one Backup did not create. After the link path is complete, and a failed removal of the temporary name or directory fsync wraps ErrPublished. A path naming no file is refused as LockPath refuses it.

func (*DB) Batch

func (db *DB) Batch(ctx context.Context, fn func(*Tx) error) error

Batch is Update through bbolt's grouped commit: concurrent calls share one transaction and fsync, in groups bounded by Options.MaxBatchSize and MaxBatchDelay. fn may run more than once (bbolt reruns a group after one call fails, and the failed call alone: https://github.com/etcd-io/bbolt/blob/v1.5.0/db.go#L1024); only its last run's writes, change and AfterCommit functions survive. Each call keeps its own subjects, marks and revision; a group's callbacks run in the order its calls ran. Batch returns and honors ctx as Update does, except that a call in a group cannot leave it before the group runs.

func (*DB) Close

func (db *DB) Close() error

Close refuses new writes, waits for the writes in progress, their OnCommit and AfterCommit callbacks included, and for open transactions, closes the file, then releases the lock Open took or frees an adopted one, which stays held. A second Close returns nil. Every later call that returns an error answers one matching both ErrClosed and bbolt's ErrDatabaseNotOpen. Close from inside a transaction or a commit callback of the same DB deadlocks.

func (*DB) Epoch

func (db *DB) Epoch() [16]byte

Epoch returns the random identity of this open: a reader holding a revision from another epoch must resync.

func (*DB) OpenReport

func (db *DB) OpenReport() OpenReport

OpenReport returns what Open did to the file before it opened it.

func (*DB) Space

func (db *DB) Space() (Space, error)

Space reads the file's size, in-use and free bytes without a transaction or a page walk: the high-water mark comes from the file's meta pages, read without mapping. After Close it returns an error matching both ErrClosed and bbolt's ErrDatabaseNotOpen.

func (*DB) Stats

func (db *DB) Stats(ctx context.Context) (Stats, error)

Stats reads the file's statistics in one read transaction, walking every table's pages.

func (*DB) Update

func (db *DB) Update(ctx context.Context, fn func(*Tx) error) error

Update runs fn in one bbolt read-write transaction. fn's error, or one from BeforeCommit, rolls it back and is returned as is; a failure of bbolt itself wraps ErrCommit. A context that ends before the commit begins returns ctx.Err(), at once while the call waits for another writer and after a rollback once fn ran; the commit itself never reads ctx. Update returns after its own commit's callbacks ran; a call that fails in its transaction returns after the callbacks of the commits before it.

func (*DB) Verify

func (db *DB) Verify(ctx context.Context, repair bool) (VerifyReport, error)

Verify rescans every table in one transaction and recomputes every index entry and counter. With repair it rewrites index entries and counters, and only those: it never rewrites or deletes a row, so an undecodable row stays reported with Repaired false, and its remedy is a restore. A repair writes no revision and honors ctx as Update does.

func (*DB) View

func (db *DB) View(ctx context.Context, fn func(*ReadTx) error) error

View runs fn in one read-only transaction. A context already done runs nothing.

type Entry

type Entry struct {
	Key, Value []byte
}

Entry is one index entry a row produces. Key is the index key; Value is the projection a lookup or scan returns without reading the row.

type FormatError

type FormatError struct {
	Namespaces []NamespaceDigest
	// Missing, sorted, is what OpenReadOnly found absent and Open would
	// create: each declared namespace the file has no bucket for, and
	// "bolttable" when it lacks the library bucket or its change log. Open
	// never fills it.
	Missing []string
}

FormatError names every namespace whose stored format differs from the schema; errors.Is(err, ErrFormat) matches it. Every name in Namespaces is one Reset.Allow is asked about and Reset.Force accepts.

func (*FormatError) Error

func (e *FormatError) Error() string

func (*FormatError) Is

func (*FormatError) Is(target error) bool

Is reports whether target is ErrFormat.

type Index

type Index[K, V any] struct {
	// contains filtered or unexported fields
}

Index is a declared index over a table. Each row maps to any number of entries: none makes the index sparse, several make it per-element.

func DefineIndex

func DefineIndex[K, V any](t *Table[K, V], name string, o IndexOptions, entries func(k K, v *V) []Entry) (*Index[K, V], error)

DefineIndex declares an index whose entries entries computes from a row. It fails for a nil table or function, a zero version, a name already used in the namespace, and after Open.

func (*Index[K, V]) Lookup

func (ix *Index[K, V]) Lookup(r *ReadTx, key []byte) (k K, proj []byte, found bool, err error)

Lookup returns the row key and projection under a unique index key; found is false when no row has it. It fails on a non-unique index.

func (*Index[K, V]) Scan

func (ix *Index[K, V]) Scan(r *ReadTx, rg Range, fn func(ikey []byte, k K, proj []byte) (bool, error)) error

Scan calls fn for each entry whose index key falls in rg, in index-key order, until fn returns false or an error.

type IndexOptions

type IndexOptions struct {
	Unique  bool
	Version uint32
}

IndexOptions configures an index. A Unique index gives each key at most one row. Version enters the namespace's format digest and must not be zero.

type InvalidFilePolicy

type InvalidFilePolicy uint8

InvalidFilePolicy is what Open does with a file wrapping ErrInvalidFile.

const (
	// InvalidFileFail returns the error and leaves the file untouched.
	InvalidFileFail InvalidFilePolicy = iota
	// InvalidFileMoveAside renames the file to <path>.<stamp>.invalid, stamp
	// the UTC time as 20060102T150405.000000000Z, and opens a new empty file.
	// Open moves at most one file aside: a new file that is invalid fails.
	InvalidFileMoveAside
)

type KeyCodec

type KeyCodec[K any] interface {
	Version() uint32
	AppendKey(dst []byte, k K) []byte
	ParseKey(b []byte) (K, error)
}

KeyCodec lays a table's keys out as bytes; the byte order is the order Scan visits. Version enters the namespace's format digest, so a layout change bumps it. ParseKey receives a copy it may keep.

type Live

type Live struct {
	// contains filtered or unexported fields
}

Live is the one handle to a live file. Every store of the file holds it, never a *DB, so a Swap replaces the file under all of them and nothing keeps writing to the file it replaced.

func NewLive

func NewLive(path string, s *Schema, o Options) (*Live, error)

NewLive checks o as Open does and holds path's lock for the handle's life: o.Lock when set, which the handle adopts and never releases, else one it takes with LockPath(path, o.LockWait), which Close releases. Until Close the lock's Release fails, and an Open or NewLive under it answers ErrLocked. It opens nothing: the handle is closed until Open or Swap opens it.

func (*Live) Backup

func (l *Live) Backup(ctx context.Context, path string) error

Backup is (*DB).Backup on the open DB; a closed or swapping handle answers ErrClosed.

func (*Live) Batch

func (l *Live) Batch(ctx context.Context, fn func(*Tx) error) error

Batch is (*DB).Batch on the open DB; a closed or swapping handle answers ErrClosed.

func (*Live) Close

func (l *Live) Close() error

Close refuses new calls, closes the DB as (*DB).Close does, unbinds the lock and releases one NewLive took, never an adopted one. It is final: every later View, Update, Batch, Backup, Space, Pin, Open and Swap answers ErrClosed, Err answers ErrClosed and Epoch zero. A second Close returns nil.

func (*Live) Epoch

func (l *Live) Epoch() [16]byte

Epoch is the open DB's epoch, kept until that DB's close returns, which waits for its transactions and the callbacks of its commits, and zero while no DB is open.

func (*Live) Err

func (l *Live) Err() error

Err is nil while the handle is open or swapping; otherwise why it is closed: the error of the open, rename, hook or DB close that closed it, ErrClosed after Close, or an error saying it was never opened.

func (*Live) OnOpen

func (l *Live) OnOpen(fn func(db *DB) error)

OnOpen registers fn to run at every open of the file, the first and each swap's, in registration order and before any call is admitted. fn works on db. View, Update, Batch, Backup, Space and Pin on the handle answer ErrClosed inside it; Open, Swap, Close and Refuse wait for the open it runs in, so a hook never calls them. Its error closes the handle with that error, and the open answers it. OnOpen after the handle's first Open or Swap panics, since that hook would miss an open.

func (*Live) Open

func (l *Live) Open() error

Open opens the file under the held lock and runs the OnOpen hooks. A failure leaves the handle closed with Err set and answers it. Open of an open handle is an error; after Close it answers ErrClosed.

func (*Live) Pin

func (l *Live) Pin() (*DB, error)

Pin returns the open DB, for work spanning transactions that must not cross a swap: after a Swap that DB is closed, so its next call answers ErrClosed. Drop it when the work ends. A closed or swapping handle answers ErrClosed.

func (*Live) Refuse

func (l *Live) Refuse(err error)

Refuse sets Err to err on a handle that is not open, such as a blocked start's, until an Open or Swap opens it; every call answers ErrClosed wrapping err. On an open handle, after Close, or with a nil err it does nothing. It waits for an Open, Swap or Close in progress.

func (*Live) Space

func (l *Live) Space() (Space, error)

Space is (*DB).Space on the open DB; a closed or swapping handle answers ErrClosed.

func (*Live) Swap

func (l *Live) Swap(ctx context.Context, rename func() error) error

Swap replaces the file: it refuses new calls with ErrClosed, waits for the calls in flight until ctx ends (then admits calls again, closes nothing and answers ctx's error), closes the DB as (*DB).Close does, calls rename, reopens the file with the same schema and options, runs the OnOpen hooks and admits calls. The wait is cooperative: no bbolt transaction reads ctx inside its function. A failed close, rename, open or hook leaves the handle closed with Err set and answers the error. On a closed handle Swap skips the close; after Close it answers ErrClosed.

func (*Live) Update

func (l *Live) Update(ctx context.Context, fn func(*Tx) error) error

Update is (*DB).Update on the open DB; a closed or swapping handle answers ErrClosed.

func (*Live) View

func (l *Live) View(ctx context.Context, fn func(*ReadTx) error) error

View is (*DB).View on the open DB; a closed or swapping handle answers ErrClosed.

type Lock

type Lock struct {
	// contains filtered or unexported fields
}

Lock is the exclusive process lock on a database path, the file <path>.lock. The lock assumes a local filesystem (see LockPath).

func LockPath

func LockPath(path string, wait time.Duration) (*Lock, error)

LockPath takes the exclusive lock on path, waiting up to wait: zero tries once, negative is an error, and another holder fails it with ErrLocked. A path naming no file (empty, or ending in a separator, "." or "..") is an error before anything is created. It creates <path>.lock, mode 0600, and never removes or renames it. Pass the Lock as Options.Lock to Open path under it. The lock, flock(2) on Linux, the BSDs and Darwin, excludes other processes and other handles in this one only on a local filesystem: NFS and, since Linux 5.5, CIFS emulate flock with per-process byte-range locks. Other platforms, Windows included, get an error wrapping errors.ErrUnsupported.

func (*Lock) Release

func (l *Lock) Release() error

Release releases the lock. A second Release returns nil. While an open DB runs under the lock (Options.Lock) or a Live holds it, Release fails and the lock stays held: Close the DB or the Live first.

type Mismatch

type Mismatch struct {
	Name                        string
	Missing, Extra, Undecodable int
	Repaired                    bool
}

Mismatch is one table, index or counter, named namespace.name, whose stored state differs from a rescan of its rows. For a table, Undecodable counts rows that do not decode; their index entries count as neither missing nor extra. For an index, Missing and Extra count entries. For a counter, a stored total differing from the exact sum of its rows' weights, or a sum outside int64, is one Missing and one Extra, and Undecodable counts rows its sum could not include, which leaves it unchecked.

type Namespace

type Namespace struct {
	// contains filtered or unexported fields
}

Namespace is one top-level bucket holding its tables, with its own format digest, so one namespace's format change never resets another's rows. Prefixes are unique across the schema, so a Subject stays prefix plus key.

type NamespaceDigest

type NamespaceDigest struct {
	Name         string
	Stored, Want [32]byte
}

NamespaceDigest is one mismatched top-level bucket. A zero Want is a stored bucket the schema does not declare, or the library's own bucket; a zero Stored is a bucket holding no digest, or any bucket of a file whose layout is not this library's. A declared namespace the file lacks is a FormatError.Missing name, never a NamespaceDigest.

type NamespaceStats

type NamespaceStats struct {
	Name   string
	Tables []TableStats
	Format [32]byte
}

NamespaceStats is one namespace's stored format digest and its tables.

type OpenReport

type OpenReport struct {
	// Invalid is why the file was moved aside; it wraps ErrInvalidFile.
	Invalid error
	// CompactErr is a compaction that failed and left the file as it was,
	// or a directory fsync that failed after it, with Compacted set.
	CompactErr error
	// MovedAside is the invalid file's new path; "" when none.
	MovedAside string
	// Kept is the .old copy taken before the reset; "" when none.
	Kept string
	// Reset is every name reset, sorted, with the digest stored and the one
	// recreated; Want is zero for a name the schema does not declare.
	Reset     []NamespaceDigest
	Compacted bool
}

OpenReport is what Open did to the file before it opened it.

type Options

type Options struct {
	// BeforeCommit runs once per revision-reserving transaction, before bbolt
	// commits, and returns the subjects to record. It may write silent tables
	// and add marks; a non-silent write or Touch fails the transaction, and
	// its error rolls everything back. It may return subjects naming no row,
	// built with Synthetic, which reach Since and OnCommit unchanged; a
	// subject starting with neither a declared prefix nor SyntheticPrefix
	// fails the transaction.
	BeforeCommit func(*Tx, *Pending) ([]Subject, error)
	// OnCommit receives each committed change. OnCommit and every AfterCommit
	// function run in commit order across the DB, silent-only and empty
	// commits included: each commit's callbacks return before the next
	// commit's start. They run one at a time, outside every bbolt lock, and
	// must not block, start a write, close the DB, or call Close, Swap, Open
	// or Refuse on a Live serving it: blocking, a write, a close, Close and
	// Swap deadlock, and Open and Refuse deadlock while a Close or Swap of
	// that Live runs. A View is allowed. Close waits for them.
	OnCommit func(Change)
	// CommitFault, for tests of a caller's write-fault handling, is asked
	// once per run of every Update and Batch call whose context is not done
	// after fn, BeforeCommit and the change record, with that context;
	// Verify's repair never asks it. A non-nil error fails the run as a
	// failed bbolt commit would: it rolls back, runs no callback, and the
	// call returns an error wrapping ErrCommit and the hook's error. In a
	// Batch group it fails only the calls it selects, on every rerun, where a
	// real commit failure fails the whole group. Production leaves it nil.
	CommitFault func(ctx context.Context) error
	// Inspect runs once on a stored file, in a read-only transaction after
	// the format comparison and before Open resets, compacts or creates
	// anything. Its error refuses the open, returned as is, with the file and
	// the lock as they were. A table, index, counter, namespace or change log
	// the file lacks reads as empty; an absent or empty file is not inspected.
	// A memory fault or a bbolt panic reading a page is ErrInvalidFile, which
	// InvalidFile decides on; a panic in fn's own code reaches the caller.
	Inspect func(*ReadTx) error
	// Lock is a lock the caller took with LockPath(path, …) and still holds.
	// Open runs under it instead of taking its own, and Close leaves it
	// held; one a Live holds fails Open with ErrLocked. nil has Open take the
	// lock and Close release it. Setting it with a non-zero LockWait is an
	// error.
	Lock *Lock
	// Reset decides what Open resets.
	Reset ResetPolicy
	// Compact decides when Open compacts the file.
	Compact CompactPolicy
	// Retention is how many of the newest changes Since can return; it must
	// not be negative.
	Retention int
	// MaxBatchSize and MaxBatchDelay bound a Batch group: it runs when it
	// holds MaxBatchSize calls or MaxBatchDelay after its first call,
	// whichever comes first. Zero means 1000 calls and 10 ms; negative is an
	// error.
	MaxBatchSize  int
	MaxBatchDelay time.Duration
	// LockWait is how long Open waits for another holder of <path>.lock,
	// polling every 50 ms. Zero tries once; negative is an error.
	LockWait time.Duration
	// InvalidFile is what Open does with a file wrapping ErrInvalidFile.
	InvalidFile InvalidFilePolicy
}

Options configures an open DB. The zero value records Touched as each change's subjects, delivers nothing and retains no change, so every Since below the head answers full.

type Pending

type Pending struct {
	Touched []Subject
	Marks   []string
	Rev     uint64
}

Pending is the revision a transaction is about to commit, handed to BeforeCommit. Its slices are copies.

type Range

type Range struct {
	Prefix  []byte
	From    []byte
	To      []byte
	Reverse bool
}

Range selects keys k with Prefix as a prefix, From <= k (when From is not nil) and k < To (when To is not nil), visited in byte order or, when Reverse, in reverse. A table compares its encoded keys; an index compares index keys, and a non-unique index visits a key's rows in primary-key order. From >= To selects nothing. A page resumes exactly (forward: From set to the last key plus 0x00; reverse: To set to the last key) only when each key names one row: a table, a unique index, or a non-unique index whose key carries something unique per row, such as a sequence.

func Exact

func Exact(key []byte) Range

Exact selects exactly key: a table's one row, or every entry of an index key, whether the index is unique or not. It copies key.

type ReadDB

type ReadDB struct {
	// contains filtered or unexported fields
}

ReadDB is a file opened by OpenReadOnly. It has no write method.

func OpenReadOnly

func OpenReadOnly(path string, s *Schema, o ReadOptions) (rdb *ReadDB, err error)

OpenReadOnly opens the file at path for reading only under s, sealing s: it creates, resets, compacts and writes nothing. It holds the shared lock on an existing <path>.lock until Close, so a writer's Open fails with ErrLocked meanwhile, and fails with ErrLocked when a writer keeps it past o.LockWait. An absent or empty file wraps fs.ErrNotExist, an unreadable one ErrInvalidFile; a file Open would reset or complete fails with a *FormatError whose Missing names what Open would create. The path follows LockPath's rule and the lock its filesystem and platform limits.

func (*ReadDB) Backup

func (db *ReadDB) Backup(ctx context.Context, path string) error

Backup writes a compacted copy of the file to path as DB.Backup does.

func (*ReadDB) Close

func (db *ReadDB) Close() error

Close closes the file, waiting for open transactions, and releases the shared lock. A second Close returns nil.

func (*ReadDB) Epoch

func (db *ReadDB) Epoch() [16]byte

Epoch returns the random identity of this open.

func (*ReadDB) Space

func (db *ReadDB) Space() (Space, error)

Space reads the file's size, in-use and free bytes as DB.Space does.

func (*ReadDB) Stats

func (db *ReadDB) Stats(ctx context.Context) (Stats, error)

Stats reads the file's statistics as DB.Stats does; Commits is 0.

func (*ReadDB) Verify

func (db *ReadDB) Verify(ctx context.Context) (VerifyReport, error)

Verify rescans every table and recomputes every index entry and counter, as DB.Verify does; it never repairs.

func (*ReadDB) View

func (db *ReadDB) View(ctx context.Context, fn func(*ReadTx) error) error

View runs fn in one read-only transaction. A context already done runs nothing.

type ReadOptions

type ReadOptions struct {
	// Retention is how many of the newest changes Since can return, as
	// Options.Retention.
	Retention int
	// LockWait is how long OpenReadOnly waits for a writer holding
	// <path>.lock, as Options.LockWait.
	LockWait time.Duration
}

ReadOptions configures OpenReadOnly.

type ReadTx

type ReadTx struct {
	// contains filtered or unexported fields
}

ReadTx reads one consistent snapshot. It is valid only inside the function View passed it to, or while its Tx is.

func (*ReadTx) Rev

func (r *ReadTx) Rev() (uint64, error)

Rev returns the head revision: the newest committed one, 0 before any. It fails after the transaction ended, and with ErrCorrupt when the library bucket, the change log or the head is damaged.

func (*ReadTx) Since

func (r *ReadTx) Since(after uint64, limit int) (ChangePage, error)

Since returns at most limit retained changes with Rev above after, in order. limit below 1 is an error. To continue, call again with after set to the last returned Rev; pages read in one transaction agree, and a later transaction may find the log trimmed past after, which it reports as Full.

type ResetPolicy

type ResetPolicy struct {
	// Allow reports whether a namespace a *FormatError lists may be reset.
	// Open asks it once per listed name, in order, and resets only when it
	// accepts every one; nil accepts none. Its panic reaches the caller.
	Allow func(NamespaceDigest) bool
	// Force names namespaces Open resets even when their format matches.
	// Each must be stored in the file, declared by the schema, or
	// "bolttable"; a new file stores nothing, so there it resets nothing.
	Force []string
	// Keep is how many .old copies of the file to keep, the new one
	// included: before a reset Open copies the file to
	// <path>.<8 hex>.<stamp>.old, the hex from the first reset name's stored
	// digest, and removes the older copies. Zero takes no copy and removes
	// none; negative is an error.
	Keep int
}

ResetPolicy decides what Open resets. The zero value resets nothing.

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

Schema is the set of namespaces a file holds. Declarations build it before the first Open, which seals it; a declaration after that fails.

func NewSchema

func NewSchema(name string) *Schema

NewSchema returns an empty schema. Its name enters every namespace's format digest.

func (*Schema) Namespace

func (s *Schema) Namespace(name string) (*Namespace, error)

Namespace declares a namespace. It fails for an empty or duplicate name, the library's own bucket name "bolttable", and after Open.

func (*Schema) Prefixes

func (s *Schema) Prefixes() []byte

Prefixes returns the prefix of every non-silent table, ascending: the tables whose writes record a subject. A silent table is left out.

type Space

type Space struct {
	// FileBytes is the data file's size, including the tail bbolt grows the
	// file by ahead of use, which holds no page yet.
	FileBytes int64
	// InUseBytes is the pages below bbolt's high-water mark, as the newest
	// valid meta page records it, less FreeBytes.
	InUseBytes int64
	// FreeBytes is free and pending pages times the page size, as of the
	// last write commit or the open. Pending pages count: they become
	// reusable once the readers pinning them finish, and compaction drops
	// them.
	FreeBytes int64
}

Space is the file's size, the bytes its data uses and the bytes bbolt can reuse. InUseBytes + FreeBytes is at most FileBytes; the rest is bbolt's preallocated tail.

type Stats

type Stats struct {
	Namespaces []NamespaceStats
	Rev        uint64
	Oldest     uint64
	Commits    uint64
	Space      Space
}

Stats describes an open file. Rev and Oldest bound what Since returns. Commits counts every bbolt commit since Open whose meta page reached the file: one per Batch group, and silent-only, byte-equal and empty writes too. A rolled-back transaction does not count; one whose fdatasync failed after the meta write does, though its call returned ErrCommit, since bbolt's next transaction id moves past it (https://github.com/etcd-io/bbolt/blob/v1.5.0/tx.go#L607).

type Subject

type Subject []byte

Subject names one row in the change feed: a table's prefix byte plus the encoded key, or SyntheticPrefix plus caller-owned bytes for a subject naming no row.

func Synthetic

func Synthetic(key []byte) Subject

Synthetic returns the subject SyntheticPrefix followed by a copy of key.

type Table

type Table[K, V any] struct {
	// contains filtered or unexported fields
}

Table is a typed table declared in a namespace.

func Define

func Define[K, V any](ns *Namespace, prefix byte, name string, k KeyCodec[K], v Codec[V], o TableOptions) (*Table[K, V], error)

Define declares a table under a prefix byte unique across the schema and a name unique in its namespace. It fails for a nil namespace or codec, a zero codec version, a duplicate prefix or name, SyntheticPrefix, and after Open.

func (*Table[K, V]) Delete

func (t *Table[K, V]) Delete(tx *Tx, k K) (bool, error)

Delete removes the row under k, its index entries and counter weights, and runs the table's cascades; it reports whether a row was there.

func (*Table[K, V]) Get

func (t *Table[K, V]) Get(r *ReadTx, k K) (v V, found bool, err error)

Get returns the row under k; found is false for an absent row. A stored value that does not decode is ErrCorrupt.

func (*Table[K, V]) NextID

func (t *Table[K, V]) NextID(tx *Tx) (uint64, error)

NextID returns the table's next sequence number, starting at 1. It records no subject; on a non-silent table it fails the transaction inside BeforeCommit, as a write does. After Open resets the namespace it stays above every id the namespace issued before.

func (*Table[K, V]) Prefix

func (t *Table[K, V]) Prefix() byte

Prefix returns the table's prefix byte, the first byte of each of its rows' subjects.

func (*Table[K, V]) Put

func (t *Table[K, V]) Put(tx *Tx, k K, v *V) error

Put stores v under k, updating every index and counter of the table in the same transaction. A value whose encoding equals the stored bytes writes nothing and records no subject. It fails with a *ConflictError when a unique index key belongs to another row, writing nothing; a counter leaving int64 fails it and the whole transaction.

func (*Table[K, V]) Scan

func (t *Table[K, V]) Scan(r *ReadTx, rg Range, fn func(K, *V) (bool, error)) error

Scan calls fn for each row in rg until fn returns false or an error. A key or value that does not decode stops the scan with ErrCorrupt. Inside a write transaction fn may write the table it scans.

func (*Table[K, V]) Subject

func (t *Table[K, V]) Subject(k K) Subject

Subject returns the change-feed subject of the row under k.

type TableOptions

type TableOptions struct {
	Silent bool
}

TableOptions configures a table. A Silent table's writes reserve no revision, record no subject and never reach Since; a non-silent write inside BeforeCommit fails the transaction.

type TableStats

type TableStats struct {
	Name  string
	Rows  int
	Bytes int64
}

TableStats is one table's row count and the bytes its pages use.

type Tx

type Tx struct {
	// contains filtered or unexported fields
}

Tx is a read-write transaction. It is valid only inside the function Update or Batch passed it to.

func (*Tx) AfterCommit

func (tx *Tx) AfterCommit(fn func())

AfterCommit registers fn to run once after this transaction commits, never after a rollback or a discarded Batch run; for signals about silent writes. It runs after OnCommit and earlier AfterCommit functions, in commit order across the DB, under OnCommit's rules, which name the DB and Live calls that deadlock inside it (see Options.OnCommit).

func (*Tx) Mark

func (tx *Tx) Mark(m string)

Mark records a mark in this transaction's change; a transaction with a mark reserves a revision even when it wrote nothing.

func (*Tx) Read

func (tx *Tx) Read() *ReadTx

Read returns a ReadTx over this transaction, seeing its own writes.

func (*Tx) Touch

func (tx *Tx) Touch(s Subject)

Touch records s as a subject of this transaction's change without a write. s must start with a declared table prefix or SyntheticPrefix; another s, or a Touch inside BeforeCommit, fails the transaction.

type Uint64Key

type Uint64Key struct{}

Uint64Key encodes a uint64 big-endian, so byte order is numeric order.

func (Uint64Key) AppendKey

func (Uint64Key) AppendKey(dst []byte, k uint64) []byte

AppendKey appends k's 8 big-endian bytes.

func (Uint64Key) ParseKey

func (Uint64Key) ParseKey(b []byte) (uint64, error)

ParseKey refuses anything but 8 bytes.

func (Uint64Key) Version

func (Uint64Key) Version() uint32

Version is 1.

type VerifyReport

type VerifyReport struct {
	Mismatches []Mismatch
}

VerifyReport lists every mismatch Verify found.

type WriteHealth

type WriteHealth struct {
	// contains filtered or unexported fields
}

WriteHealth keeps the outcome of writes per caller-named class. The zero value is ready; it is safe for concurrent use and must not be copied after first use.

func (*WriteHealth) Class

func (h *WriteHealth) Class(class string) (WriteRecord, bool)

Class returns the record of class; false when Record never changed it.

func (*WriteHealth) Record

func (h *WriteHealth) Record(class string, err error) WriteRecord

Record notes the result of one Update or Batch call of class and returns the class's record after it. nil records a commit, which clears a failing run of that class only; an error wrapping ErrCommit records a failure; every other error (the function's own, *ConflictError, ErrCorrupt, ErrClosed, a context error) changes nothing.

type WriteRecord

type WriteRecord struct {
	// LastErr is the latest ErrCommit, kept after a recovery.
	LastErr error
	// FirstFailed starts the current failing run; zero when not failing.
	FirstFailed time.Time
	LastFailed  time.Time
	LastOK      time.Time
	// Failures counts every ErrCommit recorded and is never reset.
	Failures uint64
	// Failing reports that the latest commit or failure recorded failed.
	Failing bool
}

WriteRecord is one class's write outcomes since its WriteHealth was made.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL