backend

Documentation

Overview

Package backend defines the L1 storage seam (DESIGN.md §3, §5): a common Backend interface (Read/Write/List/Delete over whole-object keys) with interchangeable implementations. The in-memory backend (Memory, in this package) is first-class — the reference and test default — so the whole engine runs with no disk or object store. The file backend lives in backend/file. The s3 backend and compare-and-swap (CAS) are M5.

Index

• Variables
• type Backend
  • func Cached(inner Backend, maxBytes int64) Backend
  • func Memory() Backend
• type CacheStats

Variables

var ErrNotExist = errors.New("backend: key does not exist")

ErrNotExist is the sentinel returned (wrapped) by Backend.Read and Backend.Delete when a key is absent. Test for it with errors.Is.

Types

type Backend

type Backend interface {
	// IsEphemeral reports whether the backend stores data only in RAM (dropped on
	// process exit). [Memory] is ephemeral; file and s3 are not.
	IsEphemeral() bool

	// PutIfAbsent stores data under key only if the key does not already exist. It
	// returns true if the write happened, false if the key was already present (no
	// change). Like [Backend.Write] it is atomic per object. It is the compare-and-swap
	// primitive for manifest commits.
	PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)

	// Write stores data under key, overwriting any existing value. The write is
	// atomic per object: a reader never observes a partially written value. The
	// implementation takes ownership semantics by copying data as needed; callers may
	// reuse the buffer after Write returns.
	Write(ctx context.Context, key string, data []byte) error

	// Read returns the value stored under key. It returns an error satisfying
	// errors.Is(err, [ErrNotExist]) if the key is absent. The returned slice is owned
	// by the caller (implementations return a fresh copy, never aliased state).
	Read(ctx context.Context, key string) ([]byte, error)

	// List returns, sorted ascending, every key with the given prefix (empty prefix
	// lists all keys).
	List(ctx context.Context, prefix string) ([]string, error)

	// Delete removes key. It returns an error satisfying errors.Is(err, [ErrNotExist])
	// if the key is absent.
	Delete(ctx context.Context, key string) error
}

Backend is the L1 storage seam (DESIGN.md §3, §5): a common interface over interchangeable implementations — memory (ephemeral, the reference), file, and (later) s3. The same engine code path runs over all three.

Data is addressed by an opaque, slash-delimited string key (e.g. a time-bucketed object path or a file-relative path). Values are whole objects: the part format (`block`) maps one part to a key prefix and one object per column/marks/manifest, so whole-object Read/Write is sufficient and gives per-object write atomicity. All methods are safe for concurrent use.

Ranged/streaming reads are deliberately not part of this interface: a part column is read whole, and the multi-key layout already gives projection pushdown (read only the referenced column objects) without ranged reads.

Backend.PutIfAbsent is the conditional-write primitive (added in M5) on which atomic manifest / block-list commits build: a versioned manifest key is written only if no writer has claimed that version, so single-writer-wins coordination needs no Raft (it maps to S3 If-None-Match, a filesystem exclusive create, and a guarded map insert).

func Cached

func Cached(inner Backend, maxBytes int64) Backend

Cached wraps a Backend with a bounded in-memory LRU over read objects — the object-store read cache. It targets the cold tier (file/S3), where a part column is otherwise re-read over the network on every query: because part objects are write-once immutable, a cached value is never stale, so the only invalidation is eviction (by byte budget) and an explicit Write/Delete of the same key (manifest/index objects, which the wrapper keeps coherent). List and PutIfAbsent are passed through.

maxBytes is the cache's total value-byte budget; objects larger than it are not cached (they would evict everything else). maxBytes ≤ 0 disables caching (the inner backend is returned unchanged). The wrapper preserves the Backend copy semantics: stored and returned slices are private copies, so a caller may retain or mutate them freely.

func Memory

func Memory() Backend

Memory returns an ephemeral in-memory Backend (DESIGN.md §5): the whole engine runs over it with no disk or object store; objects live in RAM and are dropped when the process exits. It is the reference implementation and the default in tests.

type CacheStats

type CacheStats struct {
	Hits, Misses int64
	Bytes        int64 // resident value bytes
	Items        int   // resident objects
}

CacheStats is a snapshot of a cached backend's effectiveness.