store

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 21, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

store

JSON document store on top of BadgerDB with account-scoped keys for multi-tenant data. This is the persistence layer of the services built on bboxsys/platform.

go get github.com/bboxsys/store

Licensed under the Apache License 2.0 (LICENSE). The source is published for the projects that build on it; it is maintained by Black Box Systems and is not seeking outside contributions.

Usage

import "github.com/bboxsys/store"

st, err := store.NewStore(&store.Options{
    Location: "/var/lib/myapp/data.db",
    Logger:   myLogger, // optional; anything with Debug/Info/Warn/Error(string)
})
if err != nil {
    return err
}
defer st.Close()

// Unscoped
err = st.PutOne("job:123", job)
err = st.GetOne("job:123", &job)

// Account-scoped: keys become "{accountID}:wpt:{key}"
err = st.PutScoped(accountID, "wpt", key, waypoint)
err = st.ListScoped(accountID, "wpt", &waypoints)
n, err := st.GetScopedCount(accountID, &store.Query{Prefix: "wpt"})

Keys are prefix:id. Account-scoped keys are {accountID}:{prefix}:{key}; that layout is a contract shared by every consumer, so do not change it.

See docs/store-guide.md for the full guide: CRUD, queries and filters, streaming, pagination, TTL and meta bytes, metrics.

Logging

Options.Logger accepts any value implementing the four-method store.Logger interface. When nil, the store creates a zerolog console logger tagged with Options.Application and Options.Component. store.NewZerologLogger adapts an existing zerolog.Logger.

Development

go test ./...
go test -short ./...   # skips the TTL expiry test

Tests use t.TempDir() and can run in parallel.

Documentation

Overview

Package store is a JSON document store on top of BadgerDB.

Keys are strings of the form "prefix:id"; values are the JSON encoding of the stored object. Account-scoped helpers prepend an account ID to build "{accountID}:{prefix}:{key}" for multi-tenant data.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ScopedKey

func ScopedKey(accountID, prefix, key string) string

ScopedKey builds an account-scoped key: {accountID}:{prefix}:{key}

func ScopedPrefix

func ScopedPrefix(accountID, prefix string) string

ScopedPrefix builds an account-scoped prefix for scanning: {accountID}:{prefix}:

Types

type BadgerLogger

type BadgerLogger struct {
	// contains filtered or unexported fields
}

BadgerLogger adapts a Logger to badger's Logger interface.

func NewBadgerLogger

func NewBadgerLogger(l Logger) *BadgerLogger

NewBadgerLogger creates a new badger-compatible logger.

func (*BadgerLogger) Debugf

func (bl *BadgerLogger) Debugf(format string, args ...interface{})

Debugf implements badger.Logger.

func (*BadgerLogger) Errorf

func (bl *BadgerLogger) Errorf(format string, args ...interface{})

Errorf implements badger.Logger.

func (*BadgerLogger) Infof

func (bl *BadgerLogger) Infof(format string, args ...interface{})

Infof implements badger.Logger.

func (*BadgerLogger) Warningf

func (bl *BadgerLogger) Warningf(format string, args ...interface{})

Warningf implements badger.Logger.

type Filter

type Filter struct {
	Path  string `json:"path"`
	Value string `json:"value"`
}

Filter matches documents whose gjson Path evaluates to Value.

type LevelMetrics

type LevelMetrics struct {
	Level     int   `json:"level"`
	NumTables int   `json:"num_tables"`
	Size      int64 `json:"size"`
}

LevelMetrics contains per-level table information from BadgerDB.

type Logger

type Logger interface {
	Debug(msg string)
	Info(msg string)
	Warn(msg string)
	Error(msg string)
}

Logger is the minimal logging surface the store needs. Any logger with these four methods can be passed in Options; when none is given the store builds a zerolog console logger.

func NewZerologLogger

func NewZerologLogger(l zerolog.Logger) Logger

NewZerologLogger wraps a zerolog.Logger as a store Logger.

type Metrics

type Metrics struct {
	SizeLsm          int64          `json:"size_lsm"`
	SizeVLog         int64          `json:"size_vlog"`
	SizeTotal        int64          `json:"size_total"`
	NumKeys          int64          `json:"num_keys"`
	NumLevels        int            `json:"num_levels"`
	NumTables        int            `json:"num_tables"`
	LevelSizes       []LevelMetrics `json:"level_sizes"`
	BlockCacheHits   int64          `json:"block_cache_hits"`
	BlockCacheMisses int64          `json:"block_cache_misses"`
	DiskTotal        uint64         `json:"disk_total"`
	DiskFree         uint64         `json:"disk_free"`
	DiskUsedPct      float64        `json:"disk_used_pct"`
	PutOps           int64          `json:"put_ops"`
	GetOps           int64          `json:"get_ops"`
	DeleteOps        int64          `json:"delete_ops"`
}

Metrics contains metadata about the store.

func NewMetrics

func NewMetrics(db *badger.DB) *Metrics

NewMetrics creates a basic Metrics with LSM and VLog sizes (backward compat).

type Options

type Options struct {
	Location       string `json:"location"`
	Application    string `json:"application"`
	Component      string `json:"component"`
	IndexCacheSize int64  `json:"index_cache_size"`
	BlockCacheSize int64  `json:"block_cache_size"`
	Logger         Logger `json:"-"`
}

Options configures a Store. Application and Component are only used to tag the default logger when Logger is nil.

type Query

type Query struct {
	Prefix    string    `json:"prefix"`
	Filters   []*Filter `json:"filters"`
	Limit     int       `json:"limit"`
	Offset    int       `json:"offset"`
	TimeRange TimeRange `json:"time_range"`
}

Query describes a prefix scan with optional filters and paging.

type Store

type Store struct {
	DB       *badger.DB
	Logger   Logger
	Location string
	// contains filtered or unexported fields
}

Store wraps a BadgerDB instance.

func NewStore

func NewStore(options *Options) (*Store, error)

NewStore opens (creating if needed) the BadgerDB at options.Location.

func (*Store) Close

func (store *Store) Close() error

Close closes the underlying database.

func (*Store) CollectMetrics

func (s *Store) CollectMetrics() *Metrics

CollectMetrics gathers comprehensive metrics from the store.

func (*Store) DeleteByMeta

func (store *Store) DeleteByMeta(meta byte) error

DeleteByMeta deletes every key whose user meta byte equals meta, then runs value-log garbage collection.

func (*Store) DeleteByPrefix

func (store *Store) DeleteByPrefix(prefix string) error

DeleteByPrefix deletes every key under prefix. Keys are collected in a read transaction first, then deleted, to avoid iterating while mutating.

func (*Store) DeleteKeys

func (store *Store) DeleteKeys(keysForDelete [][]byte) error

DeleteKeys deletes the given keys, committing and starting a new transaction whenever badger reports ErrTxnTooBig.

func (*Store) DeleteOneByKey

func (store *Store) DeleteOneByKey(key string) error

DeleteOneByKey deletes a single key.

func (*Store) DeleteScoped

func (store *Store) DeleteScoped(accountID, prefix, key string) error

DeleteScoped deletes a value by account-scoped key.

func (*Store) DeleteScopedByPrefix

func (store *Store) DeleteScopedByPrefix(accountID, prefix string) error

DeleteScopedByPrefix deletes all values for an account-scoped prefix.

func (*Store) GarbageCollection

func (store *Store) GarbageCollection() error

GarbageCollection runs one round of value-log GC. Badger's "nothing to rewrite" result is not an error and is not reported as one.

func (*Store) Get

func (store *Store) Get(prefix string, v interface{}) error

Get unmarshals every value under prefix into v, which must be a pointer to a slice.

func (*Store) GetByMeta

func (store *Store) GetByMeta(meta byte, v interface{}) error

GetByMeta unmarshals every value whose user meta byte equals meta into v.

func (*Store) GetCount

func (store *Store) GetCount(query *Query) (int, error)

GetCount returns the number of values under "prefix:" matching the filters.

func (*Store) GetKeyExists

func (store *Store) GetKeyExists(key string) (bool, error)

GetKeyExists reports whether the raw value stored at key is "true". A missing key reports false with no error; any other error (I/O failure, closed DB) is returned so callers can distinguish "absent" from "could not read".

func (*Store) GetMetrics

func (store *Store) GetMetrics() *Metrics

GetMetrics returns basic size metrics. Prefer CollectMetrics.

func (*Store) GetOne

func (store *Store) GetOne(key string, obj interface{}) error

GetOne unmarshals the value at key into obj.

func (*Store) GetOneByFilter

func (store *Store) GetOneByFilter(prefix string, filters *[]Filter, obj interface{}) error

GetOneByFilter unmarshals the first value under "prefix:" matching all filters into obj. obj is left untouched when nothing matches.

func (*Store) GetOneByFilterRaw

func (store *Store) GetOneByFilterRaw(prefix string, filter Filter) (string, error)

GetOneByFilterRaw returns the first raw value under "prefix:" matching filter.

func (*Store) GetOneRaw

func (store *Store) GetOneRaw(key string) (string, error)

GetOneRaw returns the value at key as a string.

func (*Store) GetScoped

func (store *Store) GetScoped(accountID, prefix, key string, obj interface{}) error

GetScoped retrieves a value by account-scoped key.

func (*Store) GetScopedByFilter

func (store *Store) GetScopedByFilter(accountID, prefix string, filters *[]Filter, obj interface{}) error

GetScopedByFilter retrieves a single value by account-scoped prefix and filters.

func (*Store) GetScopedCount

func (store *Store) GetScopedCount(accountID string, query *Query) (int, error)

GetScopedCount returns the count of items for an account-scoped prefix.

func (*Store) GetScopedStream

func (store *Store) GetScopedStream(w io.Writer, wg *sync.WaitGroup, accountID string, query *Query)

GetScopedStream streams values for an account-scoped prefix.

func (*Store) GetScopedStreamProjected

func (store *Store) GetScopedStreamProjected(w io.Writer, wg *sync.WaitGroup, accountID string, query *Query, transform func(value []byte) ([]byte, bool))

GetScopedStreamProjected streams an account-scoped prefix with a projection. See GetStreamProjected.

func (*Store) GetStream

func (store *Store) GetStream(w io.Writer, wg *sync.WaitGroup, query *Query)

GetStream writes a JSON array of matching values to w on a goroutine that is tracked by wg. Query.Prefix is scanned as "prefix:".

func (*Store) GetStreamProjected

func (store *Store) GetStreamProjected(w io.Writer, wg *sync.WaitGroup, query *Query, transform func(value []byte) ([]byte, bool))

GetStreamProjected streams a JSON array of transformed values under "query.Prefix:" to w on a goroutine tracked by wg, mirroring GetStream but applying transform to each stored value before writing. transform returns the bytes to emit and whether to include the item (false skips it) -- typically a projection to a few fields (for example map markers: id/lat/lon) so the wire payload is a fraction of the full documents. Emitted bytes must be valid JSON. Filters, Offset, and Limit are honored as in GetStream.

This uses a single iterator deliberately: benchmarks showed Badger's parallel Stream framework is slower and allocates ~19x more memory for prefix scans of our size (it targets full-DB backup across many SSTables), so the lean single-iterator scan wins here.

func (*Store) GetWithQuery

func (store *Store) GetWithQuery(query *Query, v interface{}) error

GetWithQuery unmarshals every value under query.Prefix that matches all of query.Filters (and the time range, if set) into v, honoring query.Offset and query.Limit for pagination.

func (*Store) ListScoped

func (store *Store) ListScoped(accountID, prefix string, v interface{}) error

ListScoped returns all values for an account-scoped prefix.

func (*Store) ListScopedWithQuery

func (store *Store) ListScopedWithQuery(accountID string, query *Query, v interface{}) error

ListScopedWithQuery returns values for an account-scoped prefix with query filters.

func (*Store) PutJSONPipe

func (store *Store) PutJSONPipe(prefix string, stream *io.PipeReader) error

PutJSONPipe reads a JSON array of objects from stream and stores each under "prefix:<uuid>". Objects without a uuid field are skipped and logged.

func (*Store) PutOne

func (store *Store) PutOne(key string, obj interface{}) error

PutOne stores obj as JSON under key.

func (*Store) PutOneRaw

func (store *Store) PutOneRaw(key string, obj string) error

PutOneRaw stores a string value verbatim.

func (*Store) PutOneRawWithTTL

func (store *Store) PutOneRawWithTTL(key string, obj string, ttl time.Duration) error

PutOneRawWithTTL stores a string value verbatim with an expiry.

func (*Store) PutOneRawWithTTLAndMeta

func (store *Store) PutOneRawWithTTLAndMeta(key string, obj string, ttl time.Duration, meta byte) error

PutOneRawWithTTLAndMeta stores a string value verbatim with an expiry and meta byte.

func (*Store) PutOneWithTTL

func (store *Store) PutOneWithTTL(key string, obj interface{}, ttl time.Duration) error

PutOneWithTTL stores obj as JSON under key with an expiry.

func (*Store) PutOneWithTTLAndMeta

func (store *Store) PutOneWithTTLAndMeta(key string, obj interface{}, ttl time.Duration, meta byte) error

PutOneWithTTLAndMeta stores obj with an expiry and a user meta byte.

func (*Store) PutScoped

func (store *Store) PutScoped(accountID, prefix, key string, obj interface{}) error

PutScoped stores a value with an account-scoped key.

func (*Store) PutScopedWithTTL

func (store *Store) PutScopedWithTTL(accountID, prefix, key string, obj interface{}, ttl time.Duration) error

PutScopedWithTTL stores a value with an account-scoped key and TTL.

func (*Store) PutSlice

func (store *Store) PutSlice(prefix string, id string, objs interface{}) error

PutSlice stores every element of objs under "prefix:<id>", where id is the gjson path of the element's identifier field.

func (*Store) PutStream

func (store *Store) PutStream(prefix string, r io.Reader) error

PutStream reads a JSON array of objects from r and stores each under "prefix:<uuid>". The reader is typically an http.Response.Body. A malformed stream returns an error; individual objects that fail to store are logged and skipped.

func (*Store) StartGC

func (store *Store) StartGC(ctx context.Context, interval time.Duration)

StartGC runs value-log garbage collection on the given interval until ctx is cancelled, returning immediately (GC runs on its own goroutine). Badger only reclaims value-log space when RunValueLogGC is called, so a long-running service should start this once after opening the store. Each tick runs GC repeatedly until badger reports nothing left to rewrite.

type TimeRange

type TimeRange struct {
	Start time.Time `json:"start"`
	End   time.Time `json:"end"`
}

TimeRange bounds a query by the start/end timestamps encoded in keys.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL