memdb

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 20, 2026 License: MPL-2.0 Imports: 20 Imported by: 0

README

go-maemmidb

A drop-in replacement for hashicorp/go-memdb that is faster on every operation we could think of measuring -- and a benchmark suite, compiled against both implementations from one set of sources, that fails the build if that ever stops being true.

import memdb "github.com/thevilledev/go-maemmidb"

Same package name, same API, same behaviour: an in-memory database with MVCC, atomic transactions across tables, rich indexing, and watch channels, built on immutable radix trees. No dependencies.

go-maemmidb is an independent project. It is not affiliated with, sponsored by, or endorsed by HashiCorp or IBM. It is derived from go-memdb under the terms of the Mozilla Public License 2.0; see Credits and license.

Performance

Measured against go-memdb v1.3.5 with 190 benchmarks from one shared source, on two machines. On an Apple M1 Max (Go 1.27.1): 186 are faster, 4 are statistically equal, none is slower, none allocates more or uses more memory -- geometric mean 2.74x. On an AMD Ryzen AI 9 HX PRO 370 (Zen 5, Linux): 186 faster, 4 equal, none slower -- geometric mean 2.94x.

100,000-row table, M1 Max go-memdb go-maemmidb
Insert + commit, then delete + commit (3 indexes) 45.2 µs 13.1 µs 3.4x
The same with 11 indexes 152 µs 64 µs 2.4x
Update + commit, keys unchanged (3 indexes) 29.1 µs 8.3 µs 3.5x
Bulk load, 3 indexes 0.79 s 0.22 s 3.6x
First by id: hit / miss 683 / 398 ns 398 / 127 ns 1.7x / 3.1x
Open a read transaction and First 939 ns 396 ns 2.4x
Open and close a read transaction 64.6 ns 24.4 ns 2.6x
Last on a non-unique index 624 ns 197 ns 3.2x
Iterate a 100-row group: forward / reverse 5.1 / 9.6 µs 2.2 / 2.1 µs 2.3x / 4.5x
Scan all 100,000 rows 4.7 ms 2.5 ms 1.9x
Read your own writes (10 inserts + lookups in one txn) 156 µs 44 µs 3.6x
DeletePrefix of 100 rows 544 µs 155 µs 3.5x
Track 1,000 changes and call Changes() 11.7 ms 4.1 ms 2.9x
Watch a row, update it, observe the notification 31.2 µs 8.7 µs 3.6x
Parallel readers on 8 procs, with a writer 154 ns 68 ns 2.3x
Upstream's own BenchmarkWatch (1024 channels, expired timeout) 94 µs 21 ns
Heap bytes per row (3 / 11 indexes) 2.4 / 6.5 kB 1.0 / 2.7 kB -59%
Heap objects per row (3 / 11 indexes) 32 / 86 12 / 32 -64%

The M1 was carrying ordinary desktop load, so read the ratios rather than the absolute times.

Every number comes from the same benchmark source file built twice: once against github.com/hashicorp/go-memdb (-tags upstream) and once against this package, via a file of type aliases, so there is no adapter in the measured path. Runs are interleaved (A/B/A/B), every round with differently laid out binaries, and compared with benchstat. make bench-gate then enforces the claim: it fails if any benchmark is slower (statistically significant and beyond a 2% noise tolerance), allocates more often, or uses more memory than upstream. Method, environment and the full tables are in BENCHMARKS.md.

Where the speed comes from, in one paragraph: go-immutable-radix allocates a channel for every node and leaf it writes, tracks writable nodes in an LRU, and finds children by binary search with a closure; go-memdb on top of it resolves struct fields by name through reflection on every call and builds every index key through string concatenation. Here, watch channels are created lazily by whoever watches and sealed when their node is replaced; node ownership is an integer comparison; children are found with a bitmap and a population count (the rank trick of Roaring bitmaps) in nodes that carry their child array inline; field offsets are resolved once; keys are appended to a reused buffer. DESIGN.md has the details, including why Roaring bitmaps proper do not belong on the ordered hot path and where they will go instead.

Compatibility

  • The complete upstream test suite runs here unmodified: the eight *_test.go files are byte-identical copies, checked against their upstream SHA-256 by make verify-upstream-tests.
  • An API lock compiled against both packages pins every exported signature and struct shape.
  • Differential fuzzers drive both implementations (and both radix trees) with the same random transactions and demand identical results, ordering, error messages, Changes(), snapshot isolation and fired watch channels.

The handful of deliberate differences -- two upstream defects that are fixed rather than reproduced, and a few places where upstream's behaviour is random -- are listed in COMPATIBILITY.md.

Migrating from go-memdb

The migration is the import path, and nothing else. The package is still named memdb, so every call site -- memdb.NewMemDB, memdb.StringFieldIndex, *memdb.Txn -- keeps its spelling, and even an unaliased import still binds to the name memdb:

go get github.com/thevilledev/go-maemmidb@v0.1.0
grep -rl 'github.com/hashicorp/go-memdb' --include='*.go' . \
  | xargs sed -i '' 's|"github.com/hashicorp/go-memdb"|"github.com/thevilledev/go-maemmidb"|g'
go mod tidy && go build ./... && go test ./...

Example

Unchanged from go-memdb:

// Create a sample struct
type Person struct {
	Email string
	Name  string
	Age   int
}

// Create the DB schema
schema := &memdb.DBSchema{
	Tables: map[string]*memdb.TableSchema{
		"person": {
			Name: "person",
			Indexes: map[string]*memdb.IndexSchema{
				"id": {
					Name:    "id",
					Unique:  true,
					Indexer: &memdb.StringFieldIndex{Field: "Email"},
				},
				"age": {
					Name:    "age",
					Unique:  false,
					Indexer: &memdb.IntFieldIndex{Field: "Age"},
				},
			},
		},
	},
}

// Create a new data base
db, err := memdb.NewMemDB(schema)
if err != nil {
	panic(err)
}

// Create a write transaction
txn := db.Txn(true)

// Insert some people
people := []*Person{
	{"joe@aol.com", "Joe", 30},
	{"lucy@aol.com", "Lucy", 35},
	{"tariq@aol.com", "Tariq", 21},
	{"dorothy@aol.com", "Dorothy", 53},
}
for _, p := range people {
	if err := txn.Insert("person", p); err != nil {
		panic(err)
	}
}

// Commit the transaction
txn.Commit()

// Create read-only transaction
txn = db.Txn(false)
defer txn.Abort()

// Lookup by email
raw, err := txn.First("person", "id", "joe@aol.com")
if err != nil {
	panic(err)
}
fmt.Printf("Hello %s!\n", raw.(*Person).Name)

// Range scan over people with ages between 25 and 35 inclusive
it, err := txn.LowerBound("person", "age", 25)
if err != nil {
	panic(err)
}
for obj := it.Next(); obj != nil; obj = it.Next() {
	p := obj.(*Person)
	if p.Age > 35 {
		break
	}
	fmt.Printf("  %s is aged %d\n", p.Name, p.Age)
}

Beyond go-memdb

Everything above is go-memdb's API. Three additions go further; none of them changes what the rest does, and a database that does not use them pays nothing for them. The benchmark gate holds them to that.

Bitmap indexes and set queries

Wrap an indexer in BitmapIndex and the index maps each value to the set of rows that have it -- a persistent, Roaring-style compressed bitmap over row ids -- instead of to an ordered list of rows:

"status": {Name: "status", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Status"}}},
"node":   {Name: "node",   Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "NodeID"}}},
"tags":   {Name: "tags",   Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringSliceFieldIndex{Field: "Tags"}}},

running, err := txn.Where("alloc", "status", "running")
onNode, _ := txn.Where("alloc", "node", nodeID)
batch, _ := txn.Where("alloc", "tags", "batch")

n := running.And(onNode).Len()                 // a count, without visiting a row
for obj := range running.And(onNode).AndNot(batch).All() {
	...
}

all, _ := txn.AllRows("alloc")
notRunning := all.AndNot(running)              // a complement; all.Len() is the table size

Sets are immutable values as transactional as everything else (snapshots, aborts, read-your-writes), and WhereWatch fires when a row enters or leaves a set. It suits columns with few distinct values relative to the number of rows: states, types, flags, owners, tags. Measured on a 100,000-row table against the same columns as ordinary indexes (BENCHMARKS.md):

ordinary indexes bitmap indexes
Count rows matching three columns 214 µs (index walk + filter) 18 µs
Count rows with one value 242 µs 39 ns
Visit the 10,000 rows matching two columns 657 µs 237 µs
Replace a row, indexed values unchanged 11.6 µs 4.5 µs
Insert a row, then delete it 24.6 µs 21.7 µs
Heap per row (id + three such indexes) 1213 B 453 B

What it gives up: results come in row id order (roughly insertion order), not index order; there are no range scans, and First/Get refuse such an index.

A typed API
var people = memdb.NewTable[Person]("person")
var personByEmail = people.StringKey("id")

err := people.Insert(txn, &Person{...})
p, err := personByEmail.First(txn, "joe@aol.com") // p is a *Person
rows, err := people.Get(txn, "age", 30)
for p := range rows.All() { ... }
  • StringKey, IntKey and UintKey take the key as a Go value. The untyped API has to box its arguments into interfaces -- an allocation per query for anything but small integers -- and resolve two names; a typed key does neither: a point lookup on a small table takes 51 ns instead of 92 ns (157 instead of 229 ns on 100,000 rows) and does not allocate.
  • StringIndex[T], IntIndex[T, N], UintIndex[T, N], BoolIndex[T] and StringSliceIndex[T] are indexers built from accessor functions (Get: func(p *Person) string { return p.Email }). They go where the *FieldIndex types go, produce exactly the same keys, and use no reflection anywhere.
  • Table[T] inserts *T and returns *T. Typed and untyped calls mix freely.
Range-over-func

memdb.All(it) and memdb.AllOf[*Person](it) turn any ResultIterator into an iter.Seq.

Build tags

By default, struct fields of indexed objects are read at a cached offset, and a tree node finds its children at a fixed offset behind its header, both through package unsafe. Build with -tags memdb_safe (or purego) for a pure-safe variant that uses reflection with a cached field index and an ordinary slice instead: same behaviour, still far faster than upstream, tested in CI in both modes.

Development

make check          # lint, headers, upstream-suite checksums, tests, -race, safe build, differential tests
make fuzz           # tree fuzzer + differential fuzzer against upstream
make bench-compare  # interleaved A/B run against upstream, then benchstat
make bench-gate     # fail unless universally faster
make bench-self BASE=origin/main && make bench-self-gate
                    # the same, against an older revision of this package

The benchmarks/ directory is a separate Go module, so that the original implementations are never a dependency of the main module.

Credits and license

go-maemmidb is a derivative work of go-memdb, Copyright (c) 2015 HashiCorp, Inc. / Copyright IBM Corp. 2015, 2026. Its public API, documentation, semantics and test suite are go-memdb's; the design of its storage engine follows the semantics of go-immutable-radix. All credit for the design of the database -- the table/index model, the indexers, the watch mechanism -- belongs to the go-memdb authors. See NOTICE for which files are verbatim, which are modified, and which are new.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. The full text is in LICENSE.

Documentation

Overview

Package memdb provides an in-memory database that supports transactions and MVCC.

Example

The example from the README (which is the example from go-memdb's README), compiled and checked on every test run. 

package main

import (
	"fmt"

	memdb "github.com/thevilledev/go-maemmidb"
)

func main() {
	// Create a sample struct
	type Person struct {
		Email string
		Name  string
		Age   int
	}

	// Create the DB schema
	schema := &memdb.DBSchema{
		Tables: map[string]*memdb.TableSchema{
			"person": {
				Name: "person",
				Indexes: map[string]*memdb.IndexSchema{
					"id": {
						Name:    "id",
						Unique:  true,
						Indexer: &memdb.StringFieldIndex{Field: "Email"},
					},
					"age": {
						Name:    "age",
						Unique:  false,
						Indexer: &memdb.IntFieldIndex{Field: "Age"},
					},
				},
			},
		},
	}

	// Create a new data base
	db, err := memdb.NewMemDB(schema)
	if err != nil {
		panic(err)
	}

	// Create a write transaction
	txn := db.Txn(true)

	// Insert some people
	people := []*Person{
		{"joe@aol.com", "Joe", 30},
		{"lucy@aol.com", "Lucy", 35},
		{"tariq@aol.com", "Tariq", 21},
		{"dorothy@aol.com", "Dorothy", 53},
	}
	for _, p := range people {
		if err := txn.Insert("person", p); err != nil {
			panic(err)
		}
	}

	// Commit the transaction
	txn.Commit()

	// Create read-only transaction
	txn = db.Txn(false)
	defer txn.Abort()

	// Lookup by email
	raw, err := txn.First("person", "id", "joe@aol.com")
	if err != nil {
		panic(err)
	}

	// Say hi!
	fmt.Printf("Hello %s!\n", raw.(*Person).Name)

	// List all the people
	it, err := txn.Get("person", "id")
	if err != nil {
		panic(err)
	}

	fmt.Println("All the people:")
	for obj := it.Next(); obj != nil; obj = it.Next() {
		p := obj.(*Person)
		fmt.Printf("  %s\n", p.Name)
	}

	// Range scan over people with ages between 25 and 35 inclusive
	it, err = txn.LowerBound("person", "age", 25)
	if err != nil {
		panic(err)
	}

	fmt.Println("People aged 25 - 35:")
	for obj := it.Next(); obj != nil; obj = it.Next() {
		p := obj.(*Person)
		if p.Age > 35 {
			break
		}
		fmt.Printf("  %s is aged %d\n", p.Name, p.Age)
	}

}

Output:
Hello Joe!
All the people:
  Dorothy
  Joe
  Lucy
  Tariq
People aged 25 - 35:
  Joe is aged 30
  Lucy is aged 35

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound is returned when the requested item is not found
	ErrNotFound = fmt.Errorf("not found")
)
View Source 
var MapType = reflect.MapOf(reflect.TypeOf(""), reflect.TypeOf("")).Kind()

Functions

func All 

func All(it ResultIterator) iter.Seq[interface{}]

All adapts a ResultIterator to a range-over-func iterator: 

it, err := txn.Get("person", "age", 30)
if err != nil {
	return err
}
for obj := range memdb.All(it) {
	p := obj.(*Person)
	...
}

It works with every ResultIterator, including a FilterIterator. The sequence consumes the iterator: it yields what Next has not returned yet and can be ranged over once. Breaking out of the loop leaves the rest in the iterator.

func AllOf 

func AllOf[T any](it ResultIterator) iter.Seq[T]

AllOf is All with every result asserted to T, which is normally the pointer type the table stores: 

for p := range memdb.AllOf[*Person](it) {
	...
}

Like a plain type assertion, it panics on an object of another type.

Example

Any ResultIterator can be ranged over. 

package main

import (
	"fmt"

	memdb "github.com/thevilledev/go-maemmidb"
)

// Alloc is the row type of the examples of go-maemmidb's extensions.
type Alloc struct {
	ID     string
	Node   string
	Status string
	Tags   []string
}

func allocSchema() *memdb.DBSchema {
	return &memdb.DBSchema{Tables: map[string]*memdb.TableSchema{
		"alloc": {Name: "alloc", Indexes: map[string]*memdb.IndexSchema{

			"id": {Name: "id", Unique: true, Indexer: &memdb.StringIndex[Alloc]{Get: func(a *Alloc) string { return a.ID }}},

			"node":   {Name: "node", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Node"}}},
			"status": {Name: "status", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Status"}}},
			"tags": {Name: "tags", AllowMissing: true,
				Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringSliceFieldIndex{Field: "Tags"}}},
		}},
	}}
}

func allocDB() *memdb.MemDB {
	db, err := memdb.NewMemDB(allocSchema())
	if err != nil {
		panic(err)
	}
	txn := db.Txn(true)
	for _, a := range []*Alloc{
		{ID: "a1", Node: "n1", Status: "running", Tags: []string{"batch"}},
		{ID: "a2", Node: "n1", Status: "running"},
		{ID: "a3", Node: "n1", Status: "failed", Tags: []string{"batch", "gpu"}},
		{ID: "a4", Node: "n2", Status: "running", Tags: []string{"gpu"}},
		{ID: "a5", Node: "n2", Status: "pending"},
	} {
		if err := txn.Insert("alloc", a); err != nil {
			panic(err)
		}
	}
	txn.Commit()
	return db
}

func main() {
	db := allocDB()
	it, err := db.Txn(false).Get("alloc", "id_prefix", "a")
	if err != nil {
		panic(err)
	}
	for a := range memdb.AllOf[*Alloc](it) {
		if a.Status != "running" {
			fmt.Println(a.ID, a.Status)
		}
	}

}

Output:
a3 failed
a5 pending

func IsIntType 

func IsIntType(k reflect.Kind) (size int, okay bool)

IsIntType returns whether the passed type is a type of int and the number of bytes needed to encode the type.

func IsUintType 

func IsUintType(k reflect.Kind) (size int, okay bool)

IsUintType returns whether the passed type is a type of uint and the number of bytes needed to encode the type.

Types

type BitmapIndex 

type BitmapIndex struct {
	Indexer Indexer
}

BitmapIndex wraps the Indexer of an IndexSchema to make the index a bitmap index: 

"status": {
	Name:    "status",
	Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Status"}},
},

Any SingleIndexer or MultiIndexer can be wrapped; with a MultiIndexer a row is in the set of each of its values. AllowMissing works as usual. A bitmap index cannot be unique and cannot be the "id" index. It is queried with Txn.Where and Txn.WhereWatch; First, Get and the other ordered queries refuse it.

func (*BitmapIndex) FromArgs 

func (b *BitmapIndex) FromArgs(args ...interface{}) ([]byte, error)

FromArgs is the wrapped indexer's.

func (*BitmapIndex) FromObject 

func (b *BitmapIndex) FromObject(obj interface{}) (bool, [][]byte, error)

FromObject makes BitmapIndex a MultiIndexer, which is what a schema requires of it. Transactions use the wrapped indexer directly.

func (*BitmapIndex) PrefixFromArgs 

func (b *BitmapIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

PrefixFromArgs is the wrapped indexer's, if it has one.

type BoolFieldIndex 

type BoolFieldIndex struct {
	Field string
}

BoolFieldIndex is used to extract an boolean field from an object using reflection and builds an index on that field.

func (*BoolFieldIndex) FromArgs 

func (i *BoolFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*BoolFieldIndex) FromObject 

func (i *BoolFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

type BoolIndex 

type BoolIndex[T any] struct {
	Get func(*T) bool
	// contains filtered or unexported fields
}

BoolIndex indexes a boolean that a function extracts from the object: the typed twin of BoolFieldIndex (and, with a suitable function, of FieldSetIndex and ConditionalIndex).

func (*BoolIndex[T]) FromArgs 

func (b *BoolIndex[T]) FromArgs(args ...interface{}) ([]byte, error)

func (*BoolIndex[T]) FromObject 

func (b *BoolIndex[T]) FromObject(obj interface{}) (bool, []byte, error)

type Change 

type Change struct {
	Table  string
	Before interface{}
	After  interface{}
	// contains filtered or unexported fields
}

Change describes a mutation to an object in a table.

func (*Change) Created 

func (m *Change) Created() bool

Created returns true if the mutation describes a new object being inserted.

func (*Change) Deleted 

func (m *Change) Deleted() bool

Deleted returns true if the mutation describes an existing object being deleted.

func (*Change) Updated 

func (m *Change) Updated() bool

Updated returns true if the mutation describes an existing object being updated.

type Changes 

type Changes []Change

Changes describes a set of mutations to memDB tables performed during a transaction.

type CompoundIndex 

type CompoundIndex struct {
	Indexes []Indexer

	// AllowMissing results in an index based on only the indexers
	// that return data. If true, you may end up with 2/3 columns
	// indexed which might be useful for an index scan. Otherwise,
	// the CompoundIndex requires all indexers to be satisfied.
	AllowMissing bool
}

CompoundIndex is used to build an index using multiple sub-indexes Prefix based iteration is supported as long as the appropriate prefix of indexers support it. All sub-indexers are only assumed to expect a single argument.

func (*CompoundIndex) FromArgs 

func (c *CompoundIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*CompoundIndex) FromObject 

func (c *CompoundIndex) FromObject(raw interface{}) (bool, []byte, error)

func (*CompoundIndex) PrefixFromArgs 

func (c *CompoundIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type CompoundMultiIndex 

type CompoundMultiIndex struct {
	Indexes []Indexer

	// AllowMissing results in an index based on only the indexers
	// that return data. If true, you may end up with 2/3 columns
	// indexed which might be useful for an index scan. Otherwise,
	// CompoundMultiIndex requires all indexers to be satisfied.
	AllowMissing bool
}

CompoundMultiIndex is used to build an index using multiple sub-indexes.

Unlike CompoundIndex, CompoundMultiIndex can have both SingleIndexer and MultiIndexer sub-indexers. However, each MultiIndexer adds considerable overhead/complexity in terms of the number of indexes created under-the-hood. It is not suggested to use more than one or two, if possible.

Another change from CompoundIndexer is that if AllowMissing is set, not only is it valid to have empty index fields, but it will still create index values up to the first empty index. This means that if you have a value with an empty field, rather than using a prefix for lookup, you can simply pass in less arguments. As an example, if {Foo, Bar} is indexed but Bar is missing for a value and AllowMissing is set, an index will still be created for {Foo} and it is valid to do a lookup passing in only Foo as an argument. Note that the ordering isn't guaranteed -- it's last-insert wins, but this is true if you have two objects that have the same indexes not using AllowMissing anyways.

Because StringMapFieldIndexers can take a varying number of args, it is currently a requirement that whenever it is used, two arguments must _always_ be provided for it. In theory we only need one, except a bug in that indexer means the single-argument version will never work. You can leave the second argument nil, but it will never produce a value. We support this for whenever that bug is fixed, likely in a next major version bump.

Prefix-based indexing is not currently supported.

func (*CompoundMultiIndex) FromArgs 

func (c *CompoundMultiIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*CompoundMultiIndex) FromObject 

func (c *CompoundMultiIndex) FromObject(raw interface{}) (bool, [][]byte, error)

type ConditionalIndex 

type ConditionalIndex struct {
	Conditional ConditionalIndexFunc
}

ConditionalIndex builds an index based on a condition specified by a passed user function. This function may examine the passed object and return a boolean to encapsulate an arbitrarily complex conditional.

func (*ConditionalIndex) FromArgs 

func (c *ConditionalIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*ConditionalIndex) FromObject 

func (c *ConditionalIndex) FromObject(obj interface{}) (bool, []byte, error)

type ConditionalIndexFunc 

type ConditionalIndexFunc func(obj interface{}) (bool, error)

ConditionalIndexFunc is the required function interface for a ConditionalIndex.

type DBSchema 

type DBSchema struct {
	// Tables is the set of tables within this database. The key is the
	// table name and must match the Name in TableSchema.
	Tables map[string]*TableSchema
}

DBSchema is the schema to use for the full database with a MemDB instance.

MemDB will require a valid schema. Schema validation can be tested using the Validate function. Calling this function is recommended in unit tests.

func (*DBSchema) Validate 

func (s *DBSchema) Validate() error

Validate validates the schema.

type FieldSetIndex 

type FieldSetIndex struct {
	Field string
}

FieldSetIndex is used to extract a field from an object using reflection and builds an index on whether the field is set by comparing it against its type's nil value.

func (*FieldSetIndex) FromArgs 

func (f *FieldSetIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*FieldSetIndex) FromObject 

func (f *FieldSetIndex) FromObject(obj interface{}) (bool, []byte, error)

type FilterFunc 

type FilterFunc func(interface{}) bool

FilterFunc is a function that takes the results of an iterator and returns whether the result should be filtered out.

type FilterIterator 

type FilterIterator struct {
	// contains filtered or unexported fields
}

FilterIterator is used to wrap a ResultIterator and apply a filter over it.

func NewFilterIterator 

func NewFilterIterator(iter ResultIterator, filter FilterFunc) *FilterIterator

NewFilterIterator wraps a ResultIterator. The filter function is applied to each value returned by a call to iter.Next.

See the documentation for ResultIterator to understand the behaviour of the returned FilterIterator.

func (*FilterIterator) Next 

func (f *FilterIterator) Next() interface{}

Next returns the next non-filtered result from the wrapped iterator.

func (*FilterIterator) WatchCh 

func (f *FilterIterator) WatchCh() <-chan struct{}

WatchCh returns the watch channel of the wrapped iterator.

type IndexSchema 

type IndexSchema struct {
	// Name of the index. This must be unique among a tables set of indexes.
	// This must match the key in the map of Indexes for a TableSchema.
	Name string

	// AllowMissing if true ignores this index if it doesn't produce a
	// value. For example, an index that extracts a field that doesn't
	// exist from a structure.
	AllowMissing bool

	Unique  bool
	Indexer Indexer
}

IndexSchema is the schema for an index. An index defines how a table is queried.

func (*IndexSchema) Validate 

func (s *IndexSchema) Validate() error

type Indexer 

type Indexer interface {
	// FromArgs is called to build the exact index key from a list of arguments.
	FromArgs(args ...interface{}) ([]byte, error)
}

Indexer is an interface used for defining indexes. Indexes are used for efficient lookup of objects in a MemDB table. An Indexer must also implement one of SingleIndexer or MultiIndexer.

Indexers are primarily responsible for returning the lookup key as a byte slice. The byte slice is the key data in the underlying data storage.

type IntFieldIndex 

type IntFieldIndex struct {
	Field string
}

IntFieldIndex is used to extract an int field from an object using reflection and builds an index on that field.

func (*IntFieldIndex) FromArgs 

func (i *IntFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*IntFieldIndex) FromObject 

func (i *IntFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

type IntIndex 

type IntIndex[T any, N signedInt] struct {
	Get func(*T) N
	// contains filtered or unexported fields
}

IntIndex indexes a signed integer that a function extracts from the object: the typed twin of IntFieldIndex. Keys are as wide as N and sort numerically.

Unlike IntFieldIndex, whose FromArgs encodes an argument at the width of the argument's own type, IntIndex accepts any signed integer type in queries and always encodes at the width of N, so an untyped constant works as expected.

func (*IntIndex[T, N]) FromArgs 

func (i *IntIndex[T, N]) FromArgs(args ...interface{}) ([]byte, error)

func (*IntIndex[T, N]) FromObject 

func (i *IntIndex[T, N]) FromObject(obj interface{}) (bool, []byte, error)

type IntKey 

type IntKey[T any, N signedInt] struct {
	// contains filtered or unexported fields
}

IntKey is a handle on one index of a table whose values are signed integers of type N: an index built from an IntFieldIndex on a field of that type, or from an IntIndex. See StringKey.

func IntKeyOf 

func IntKeyOf[N signedInt, T any](t Table[T], index string) *IntKey[T, N]

IntKeyOf returns a typed handle on the named integer index. (It is a function because a method cannot introduce the type parameter N.)

func (*IntKey[T, N]) First 

func (k *IntKey[T, N]) First(txn *Txn, key N) (*T, error)

First returns the first row whose index value is key, like Txn.First.

func (*IntKey[T, N]) Get 

func (k *IntKey[T, N]) Get(txn *Txn, key N) (Rows[T], error)

Get returns the rows whose index value is key, like Txn.Get.

func (*IntKey[T, N]) LowerBound 

func (k *IntKey[T, N]) LowerBound(txn *Txn, key N) (Rows[T], error)

LowerBound returns the rows whose index value is at least key, in ascending order, like Txn.LowerBound.

type MemDB 

type MemDB struct {
	// contains filtered or unexported fields
}

MemDB is an in-memory database providing Atomicity, Consistency, and Isolation from ACID. MemDB doesn't provide Durability since it is an in-memory database.

MemDB provides a table abstraction to store objects (rows) with multiple indexes based on inserted values. The database makes use of immutable radix trees to provide transactions and MVCC.

Objects inserted into MemDB are not copied. It is **extremely important** that objects are not modified in-place after they are inserted since they are stored directly in MemDB. It remains unsafe to modify inserted objects even after they've been deleted from MemDB since there may still be older snapshots of the DB being read from other goroutines.

func NewMemDB 

func NewMemDB(schema *DBSchema) (*MemDB, error)

NewMemDB creates a new MemDB with the given schema.

func (*MemDB) DBSchema 

func (db *MemDB) DBSchema() *DBSchema

DBSchema returns schema in use for introspection.

The method is intended for *read-only* debugging use cases, returned schema should *never be modified in-place*.

func (*MemDB) Snapshot 

func (db *MemDB) Snapshot() *MemDB

Snapshot is used to capture a point-in-time snapshot of the database that will not be affected by any write operations to the existing DB.

If MemDB is storing reference-based values (pointers, maps, slices, etc.), the Snapshot will not deep copy those values. Therefore, it is still unsafe to modify any inserted values in either DB.

func (*MemDB) Txn 

func (db *MemDB) Txn(write bool) *Txn

Txn is used to start a new transaction in either read or write mode. There can only be a single concurrent writer, but any number of readers.

type MultiIndexer 

type MultiIndexer interface {
	// FromObject extracts index values from an object. The return values
	// are the same as a SingleIndexer except there can be multiple index
	// values.
	FromObject(raw interface{}) (bool, [][]byte, error)
}

MultiIndexer is an interface used for defining indexes that generate multiple values per object. Each value is stored as a seperate index pointing to the same object.

For example, an index that extracts the first and last name of a person and allows lookup based on eitherd would be a MultiIndexer. The FromObject of this example would split the first and last name and return both as values.

type PrefixIndexer 

type PrefixIndexer interface {
	// PrefixFromArgs is the same as FromArgs for an Indexer except that
	// the index value returned should return all prefix-matched values.
	PrefixFromArgs(args ...interface{}) ([]byte, error)
}

PrefixIndexer is an optional interface on top of an Indexer that allows indexes to support prefix-based iteration.

type ResultIterator 

type ResultIterator interface {
	WatchCh() <-chan struct{}
	// Next returns the next result from the iterator. If there are no more results
	// nil is returned.
	Next() interface{}
}

ResultIterator is used to iterate over a list of results from a query on a table.

When a ResultIterator is created from a write transaction, the results from Next will reflect a snapshot of the table at the time the ResultIterator is created. This means that calling Insert or Delete on a transaction while iterating is allowed, but the changes made by Insert or Delete will not be observed in the results returned from subsequent calls to Next. For example if an item is deleted from the index used by the iterator it will still be returned by Next. If an item is inserted into the index used by the iterator, it will not be returned by Next. However, an iterator created after a call to Insert or Delete will reflect the modifications.

When a ResultIterator is created from a write transaction, and there are already modifications to the index used by the iterator, the modification cache of the index will be invalidated. This may result in some additional allocations if the same node in the index is modified again.

type RowSet 

type RowSet struct {
	// contains filtered or unexported fields
}

RowSet is a set of rows of one table, as one transaction sees them: the result of Txn.Where and Txn.AllRows and of combining such sets. It is an immutable value, cheap to copy, and safe for concurrent use. It stays valid for as long as it is kept, like a read transaction, whatever happens to the transaction it came from.

Sets can only be combined if they are from the same table and the same state of the database: from one read transaction, or from one write transaction with no write in between. Anything else is a programming error and panics.

func (RowSet) All 

func (s RowSet) All() iter.Seq[interface{}]

All returns the rows of the set as a sequence, in Iterator's order.

func (RowSet) And 

func (s RowSet) And(others ...RowSet) RowSet

And returns the rows that are in s and in every one of the others.

func (RowSet) AndNot 

func (s RowSet) AndNot(others ...RowSet) RowSet

AndNot returns the rows of s that are in none of the others.

func (RowSet) Iterator 

func (s RowSet) Iterator() ResultIterator

Iterator returns the rows of the set in ascending row id order, which is the order of insertion as long as no row has been deleted (the id of a deleted row is given to a later one). Its watch channel is nil: watch the queries the set was built from, see Txn.WhereWatch.

func (RowSet) Len 

func (s RowSet) Len() int

Len returns the number of rows in the set. It does not visit them.

func (RowSet) Or 

func (s RowSet) Or(others ...RowSet) RowSet

Or returns the rows that are in s or in any of the others.

type Rows 

type Rows[T any] struct {
	// contains filtered or unexported fields
}

Rows is the result of a typed query: All ranges over it, WatchCh is the query's watch channel (see ResultIterator).

func (Rows[T]) All 

func (r Rows[T]) All() iter.Seq[*T]

All returns the rows as a single-use sequence.

func (Rows[T]) Next 

func (r Rows[T]) Next() *T

Next returns the next row, or nil when there are no more.

func (Rows[T]) WatchCh 

func (r Rows[T]) WatchCh() <-chan struct{}

WatchCh is ResultIterator.WatchCh.

type SingleIndexer 

type SingleIndexer interface {
	// FromObject extracts the index value from an object. The return values
	// are whether the index value was found, the index value, and any error
	// while extracting the index value, respectively.
	FromObject(raw interface{}) (bool, []byte, error)
}

SingleIndexer is an interface used for defining indexes that generate a single value per object

type StringFieldIndex 

type StringFieldIndex struct {
	Field     string
	Lowercase bool
}

StringFieldIndex is used to extract a field from an object using reflection and builds an index on that field.

func (*StringFieldIndex) FromArgs 

func (s *StringFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*StringFieldIndex) FromObject 

func (s *StringFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

func (*StringFieldIndex) PrefixFromArgs 

func (s *StringFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type StringIndex 

type StringIndex[T any] struct {
	Get       func(*T) string
	Lowercase bool
	// contains filtered or unexported fields
}

StringIndex indexes a string that a function extracts from the object. It is the typed twin of StringFieldIndex: same keys, same treatment of the empty string as a missing value, same prefix queries.

func (*StringIndex[T]) FromArgs 

func (s *StringIndex[T]) FromArgs(args ...interface{}) ([]byte, error)

func (*StringIndex[T]) FromObject 

func (s *StringIndex[T]) FromObject(obj interface{}) (bool, []byte, error)

func (*StringIndex[T]) PrefixFromArgs 

func (s *StringIndex[T]) PrefixFromArgs(args ...interface{}) ([]byte, error)

type StringKey 

type StringKey[T any] struct {
	// contains filtered or unexported fields
}

StringKey is a handle on one index of a table whose values are strings: an index built from a StringFieldIndex, StringSliceFieldIndex, UUIDFieldIndex, StringIndex or StringSliceIndex. (With any other indexer it still works, by going through the untyped path.) Declare it once and share it; it is safe for concurrent use. 

var personByEmail = people.StringKey("id")

p, err := personByEmail.First(txn, "joe@aol.com")

func (*StringKey[T]) First 

func (k *StringKey[T]) First(txn *Txn, key string) (*T, error)

First returns the first row whose index value is key, like Txn.First.

func (*StringKey[T]) FirstPrefix 

func (k *StringKey[T]) FirstPrefix(txn *Txn, prefix string) (*T, error)

FirstPrefix returns the first row whose index value starts with prefix, like Txn.First on the index's "_prefix" name.

func (*StringKey[T]) Get 

func (k *StringKey[T]) Get(txn *Txn, key string) (Rows[T], error)

Get returns the rows whose index value is key, like Txn.Get.

func (*StringKey[T]) GetPrefix 

func (k *StringKey[T]) GetPrefix(txn *Txn, prefix string) (Rows[T], error)

GetPrefix returns the rows whose index value starts with prefix, like Txn.Get on the index's "_prefix" name.

type StringMapFieldIndex 

type StringMapFieldIndex struct {
	Field     string
	Lowercase bool
}

StringMapFieldIndex is used to extract a field of type map[string]string from an object using reflection and builds an index on that field.

Note that although FromArgs in theory supports using either one or two arguments, there is a bug: FromObject only creates an index using key/value, and does not also create an index using key. This means a lookup using one argument will never actually work.

It is currently left as-is to prevent backwards compatibility issues.

TODO: Fix this in the next major bump.

func (*StringMapFieldIndex) FromArgs 

func (s *StringMapFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*StringMapFieldIndex) FromObject 

func (s *StringMapFieldIndex) FromObject(obj interface{}) (bool, [][]byte, error)

type StringSliceFieldIndex 

type StringSliceFieldIndex struct {
	Field     string
	Lowercase bool
}

StringSliceFieldIndex builds an index from a field on an object that is a string slice ([]string). Each value within the string slice can be used for lookup.

func (*StringSliceFieldIndex) FromArgs 

func (s *StringSliceFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*StringSliceFieldIndex) FromObject 

func (s *StringSliceFieldIndex) FromObject(obj interface{}) (bool, [][]byte, error)

func (*StringSliceFieldIndex) PrefixFromArgs 

func (s *StringSliceFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type StringSliceIndex 

type StringSliceIndex[T any] struct {
	Get       func(*T) []string
	Lowercase bool
	// contains filtered or unexported fields
}

StringSliceIndex indexes every string of a slice that a function extracts from the object: the typed twin of StringSliceFieldIndex.

func (*StringSliceIndex[T]) FromArgs 

func (s *StringSliceIndex[T]) FromArgs(args ...interface{}) ([]byte, error)

func (*StringSliceIndex[T]) FromObject 

func (s *StringSliceIndex[T]) FromObject(obj interface{}) (bool, [][]byte, error)

func (*StringSliceIndex[T]) PrefixFromArgs 

func (s *StringSliceIndex[T]) PrefixFromArgs(args ...interface{}) ([]byte, error)

type Table 

type Table[T any] struct {
	// contains filtered or unexported fields
}

Table is a typed handle on a table that stores *T. It carries no state but the table's name: declare it once, next to the schema, and use it with any database that has the table. 

var people = memdb.NewTable[Person]("person")

err := people.Insert(txn, &Person{...})
p, err := people.First(txn, "id", "joe@aol.com") // p is a *Person
Example

A typed table returns *Alloc instead of interface{}; a typed key looks a row up without boxing the key or resolving names. 

package main

import (
	"fmt"

	memdb "github.com/thevilledev/go-maemmidb"
)

// Alloc is the row type of the examples of go-maemmidb's extensions.
type Alloc struct {
	ID     string
	Node   string
	Status string
	Tags   []string
}

func allocSchema() *memdb.DBSchema {
	return &memdb.DBSchema{Tables: map[string]*memdb.TableSchema{
		"alloc": {Name: "alloc", Indexes: map[string]*memdb.IndexSchema{

			"id": {Name: "id", Unique: true, Indexer: &memdb.StringIndex[Alloc]{Get: func(a *Alloc) string { return a.ID }}},

			"node":   {Name: "node", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Node"}}},
			"status": {Name: "status", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Status"}}},
			"tags": {Name: "tags", AllowMissing: true,
				Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringSliceFieldIndex{Field: "Tags"}}},
		}},
	}}
}

func allocDB() *memdb.MemDB {
	db, err := memdb.NewMemDB(allocSchema())
	if err != nil {
		panic(err)
	}
	txn := db.Txn(true)
	for _, a := range []*Alloc{
		{ID: "a1", Node: "n1", Status: "running", Tags: []string{"batch"}},
		{ID: "a2", Node: "n1", Status: "running"},
		{ID: "a3", Node: "n1", Status: "failed", Tags: []string{"batch", "gpu"}},
		{ID: "a4", Node: "n2", Status: "running", Tags: []string{"gpu"}},
		{ID: "a5", Node: "n2", Status: "pending"},
	} {
		if err := txn.Insert("alloc", a); err != nil {
			panic(err)
		}
	}
	txn.Commit()
	return db
}

func main() {
	db := allocDB()
	allocs := memdb.NewTable[Alloc]("alloc")
	byID := allocs.StringKey("id")

	txn := db.Txn(true)
	if err := allocs.Insert(txn, &Alloc{ID: "a6", Node: "n3", Status: "pending"}); err != nil {
		panic(err)
	}
	txn.Commit()

	txn = db.Txn(false)
	a, err := byID.First(txn, "a6")
	if err != nil {
		panic(err)
	}
	fmt.Println(a.ID, a.Node, a.Status)

	missing, _ := byID.First(txn, "nope")
	fmt.Println(missing == nil)

	rows, err := byID.GetPrefix(txn, "a")
	if err != nil {
		panic(err)
	}
	n := 0
	for range rows.All() {
		n++
	}
	fmt.Println(n, "allocations")

}

Output:
a6 n3 pending
true
6 allocations

func NewTable 

func NewTable[T any](name string) Table[T]

NewTable returns the typed handle of the named table.

func (Table[T]) Delete 

func (t Table[T]) Delete(txn *Txn, obj *T) error

Delete is Txn.Delete.

func (Table[T]) First 

func (t Table[T]) First(txn *Txn, index string, args ...interface{}) (*T, error)

First is Txn.First.

func (Table[T]) FirstWatch 

func (t Table[T]) FirstWatch(txn *Txn, index string, args ...interface{}) (<-chan struct{}, *T, error)

FirstWatch is Txn.FirstWatch.

func (Table[T]) Get 

func (t Table[T]) Get(txn *Txn, index string, args ...interface{}) (Rows[T], error)

Get is Txn.Get.

func (Table[T]) GetReverse 

func (t Table[T]) GetReverse(txn *Txn, index string, args ...interface{}) (Rows[T], error)

GetReverse is Txn.GetReverse.

func (Table[T]) Insert 

func (t Table[T]) Insert(txn *Txn, obj *T) error

Insert is Txn.Insert.

func (Table[T]) Last 

func (t Table[T]) Last(txn *Txn, index string, args ...interface{}) (*T, error)

Last is Txn.Last.

func (Table[T]) LowerBound 

func (t Table[T]) LowerBound(txn *Txn, index string, args ...interface{}) (Rows[T], error)

LowerBound is Txn.LowerBound.

func (Table[T]) Name 

func (t Table[T]) Name() string

Name returns the table's name.

func (Table[T]) ReverseLowerBound 

func (t Table[T]) ReverseLowerBound(txn *Txn, index string, args ...interface{}) (Rows[T], error)

ReverseLowerBound is Txn.ReverseLowerBound.

func (Table[T]) StringKey 

func (t Table[T]) StringKey(index string) *StringKey[T]

StringKey returns a typed handle on the named index. The name is the plain index name: prefix queries have methods of their own.

type TableSchema 

type TableSchema struct {
	// Name of the table. This must match the key in the Tables map in DBSchema.
	Name string

	// Indexes is the set of indexes for querying this table. The key
	// is a unique name for the index and must match the Name in the
	// IndexSchema.
	Indexes map[string]*IndexSchema
}

TableSchema is the schema for a single table.

func (*TableSchema) Validate 

func (s *TableSchema) Validate() error

Validate is used to validate the table schema

type Txn 

type Txn struct {
	// contains filtered or unexported fields
}

Txn is a transaction against a MemDB. This can be a read or write transaction.

func (*Txn) Abort 

func (txn *Txn) Abort()

Abort is used to cancel this transaction. This is a noop for read transactions, already aborted or commited transactions.

func (*Txn) AllRows 

func (txn *Txn) AllRows(table string) (RowSet, error)

AllRows returns the set of all rows of a table that has at least one bitmap index. Its Len is the size of the table; s.AndNot(...) of it is a complement.

func (*Txn) Changes 

func (txn *Txn) Changes() Changes

Changes returns the set of object changes that have been made in the transaction so far. If change tracking is not enabled it wil always return nil. It can be called before or after Commit. If it is before Commit it will return all changes made so far which may not be the same as the final Changes. After abort it will always return nil. As with other Txn methods it's not safe to call this from a different goroutine than the one making mutations or committing the transaction. Mutations will appear in the order they were performed in the transaction but multiple operations to the same object will be collapsed so only the effective overall change to that object is present. If transaction operations are dependent (e.g. copy object X to Y then delete X) this might mean the set of mutations is incomplete to verify history, but it is complete in that the net effect is preserved (Y got a new value, X got removed).

func (*Txn) Commit 

func (txn *Txn) Commit()

Commit is used to finalize this transaction. This is a noop for read transactions, already aborted or committed transactions.

func (*Txn) Defer 

func (txn *Txn) Defer(fn func())

Defer is used to push a new arbitrary function onto a stack which gets called when a transaction is committed and finished. Deferred functions are called in LIFO order, and only invoked at the end of write transactions.

func (*Txn) Delete 

func (txn *Txn) Delete(table string, obj interface{}) error

Delete is used to delete a single object from the given table. This object must already exist in the table.

func (*Txn) DeleteAll 

func (txn *Txn) DeleteAll(table, index string, args ...interface{}) (int, error)

DeleteAll is used to delete all the objects in a given table matching the constraints on the index

func (*Txn) DeletePrefix 

func (txn *Txn) DeletePrefix(table string, prefix_index string, prefix string) (bool, error)

DeletePrefix is used to delete an entire subtree based on a prefix. The given index must be a prefix index, and will be used to perform a scan and enumerate the set of objects to delete. These will be removed from all other indexes, and then a special prefix operation will delete the objects from the given index in an efficient subtree delete operation. This is useful when you have a very large number of objects indexed by the given index, along with a much smaller number of entries in the other indexes for those objects.

func (*Txn) First 

func (txn *Txn) First(table, index string, args ...interface{}) (interface{}, error)

First is used to return the first matching object for the given constraints on the index.

Note that all values read in the transaction form a consistent snapshot from the time when the transaction was created.

func (*Txn) FirstWatch 

func (txn *Txn) FirstWatch(table, index string, args ...interface{}) (<-chan struct{}, interface{}, error)

FirstWatch is used to return the first matching object for the given constraints on the index along with the watch channel.

Note that all values read in the transaction form a consistent snapshot from the time when the transaction was created.

The watch channel is closed when a subsequent write transaction has updated the result of the query. Since each read transaction operates on an isolated snapshot, a new read transaction must be started to observe the changes that have been made.

If the value of index ends with "_prefix", FirstWatch will perform a prefix match instead of full match on the index. The registered indexer must implement PrefixIndexer, otherwise an error is returned.

Example

A watch fires when a later write transaction changes the result of the query it came from. 

package main

import (
	"fmt"

	memdb "github.com/thevilledev/go-maemmidb"
)

func main() {
	type Node struct {
		ID     string
		Status string
	}
	db, err := memdb.NewMemDB(&memdb.DBSchema{Tables: map[string]*memdb.TableSchema{
		"nodes": {Name: "nodes", Indexes: map[string]*memdb.IndexSchema{
			"id": {Name: "id", Unique: true, Indexer: &memdb.StringFieldIndex{Field: "ID"}},
		}},
	}})
	if err != nil {
		panic(err)
	}

	txn := db.Txn(true)
	if err := txn.Insert("nodes", &Node{ID: "n1", Status: "starting"}); err != nil {
		panic(err)
	}
	txn.Commit()

	// Read the node and ask to be told when it changes.
	watch, obj, err := db.Txn(false).FirstWatch("nodes", "id", "n1")
	if err != nil {
		panic(err)
	}
	fmt.Println("status:", obj.(*Node).Status)

	// Objects are never modified in place: write a new version.
	txn = db.Txn(true)
	if err := txn.Insert("nodes", &Node{ID: "n1", Status: "ready"}); err != nil {
		panic(err)
	}
	txn.Commit()

	<-watch // closed by the commit above
	obj, _ = db.Txn(false).First("nodes", "id", "n1")
	fmt.Println("status:", obj.(*Node).Status)

}

Output:
status: starting
status: ready

func (*Txn) Get 

func (txn *Txn) Get(table, index string, args ...interface{}) (ResultIterator, error)

Get is used to construct a ResultIterator over all the rows that match the given constraints of an index. The index values must match exactly (this is not a range-based or prefix-based lookup) by default.

Prefix lookups: if the named index implements PrefixIndexer, you may perform prefix-based lookups by appending "_prefix" to the index name. In this scenario, the index values given in args are treated as prefix lookups. For example, a StringFieldIndex will match any string with the given value as a prefix: "mem" matches "memdb".

See the documentation for ResultIterator to understand the behaviour of the returned ResultIterator.

func (*Txn) GetReverse 

func (txn *Txn) GetReverse(table, index string, args ...interface{}) (ResultIterator, error)

GetReverse is used to construct a Reverse ResultIterator over all the rows that match the given constraints of an index. The returned ResultIterator's Next() will return the next Previous value.

See the documentation on Get for details on arguments.

See the documentation for ResultIterator to understand the behaviour of the returned ResultIterator.

func (*Txn) Insert 

func (txn *Txn) Insert(table string, obj interface{}) error

Insert is used to add or update an object into the given table.

When updating an object, the obj provided should be a copy rather than a value updated in-place. Modifying values in-place that are already inserted into MemDB is not supported behavior.

func (*Txn) Last 

func (txn *Txn) Last(table, index string, args ...interface{}) (interface{}, error)

Last is used to return the last matching object for the given constraints on the index.

Note that all values read in the transaction form a consistent snapshot from the time when the transaction was created.

func (*Txn) LastWatch 

func (txn *Txn) LastWatch(table, index string, args ...interface{}) (<-chan struct{}, interface{}, error)

LastWatch is used to return the last matching object for the given constraints on the index along with the watch channel.

Note that all values read in the transaction form a consistent snapshot from the time when the transaction was created.

The watch channel is closed when a subsequent write transaction has updated the result of the query. Since each read transaction operates on an isolated snapshot, a new read transaction must be started to observe the changes that have been made.

If the value of index ends with "_prefix", LastWatch will perform a prefix match instead of full match on the index. The registered indexer must implement PrefixIndexer, otherwise an error is returned.

func (*Txn) LongestPrefix 

func (txn *Txn) LongestPrefix(table, index string, args ...interface{}) (interface{}, error)

LongestPrefix is used to fetch the longest prefix match for the given constraints on the index. Note that this will not work with the memdb StringFieldIndex because it adds null terminators which prevent the algorithm from correctly finding a match (it will get to right before the null and fail to find a leaf node). This should only be used where the prefix given is capable of matching indexed entries directly, which typically only applies to a custom indexer. See the unit test for an example.

Note that all values read in the transaction form a consistent snapshot from the time when the transaction was created.

func (*Txn) LowerBound 

func (txn *Txn) LowerBound(table, index string, args ...interface{}) (ResultIterator, error)

LowerBound is used to construct a ResultIterator over all the the range of rows that have an index value greater than or equal to the provide args. Calling this then iterating until the rows are larger than required allows range scans within an index. It is not possible to watch the resulting iterator since the radix tree doesn't efficiently allow watching on lower bound changes. The WatchCh returned will be nill and so will block forever.

If the value of index ends with "_prefix", LowerBound will perform a prefix match instead of a full match on the index. The registered index must implement PrefixIndexer, otherwise an error is returned.

See the documentation for ResultIterator to understand the behaviour of the returned ResultIterator.

func (*Txn) ReverseLowerBound 

func (txn *Txn) ReverseLowerBound(table, index string, args ...interface{}) (ResultIterator, error)

ReverseLowerBound is used to construct a Reverse ResultIterator over all the the range of rows that have an index value less than or equal to the provide args. Calling this then iterating until the rows are lower than required allows range scans within an index. It is not possible to watch the resulting iterator since the radix tree doesn't efficiently allow watching on lower bound changes. The WatchCh returned will be nill and so will block forever.

See the documentation for ResultIterator to understand the behaviour of the returned ResultIterator.

func (*Txn) Snapshot 

func (txn *Txn) Snapshot() *Txn

Snapshot creates a snapshot of the current state of the transaction. Returns a new read-only transaction or nil if the transaction is already aborted or committed.

func (*Txn) TrackChanges 

func (txn *Txn) TrackChanges()

TrackChanges enables change tracking for the transaction. If called at any point before commit, subsequent mutations will be recorded and can be retrieved using ChangeSet. Once this has been called on a transaction it can't be unset. As with other Txn methods it's not safe to call this from a different goroutine than the one making mutations or committing the transaction.

func (*Txn) Where 

func (txn *Txn) Where(table, index string, args ...interface{}) (RowSet, error)

Where returns the set of rows whose value in a bitmap index matches the arguments: the counterpart of Get for a BitmapIndex. As with Get, appending "_prefix" to the index name matches every value that starts with the argument, if the wrapped indexer supports prefixes.

Example

Sets of rows from bitmap indexes are combined with And, Or and AndNot, and know their size without visiting a row. 

package main

import (
	"fmt"

	memdb "github.com/thevilledev/go-maemmidb"
)

// Alloc is the row type of the examples of go-maemmidb's extensions.
type Alloc struct {
	ID     string
	Node   string
	Status string
	Tags   []string
}

func allocSchema() *memdb.DBSchema {
	return &memdb.DBSchema{Tables: map[string]*memdb.TableSchema{
		"alloc": {Name: "alloc", Indexes: map[string]*memdb.IndexSchema{

			"id": {Name: "id", Unique: true, Indexer: &memdb.StringIndex[Alloc]{Get: func(a *Alloc) string { return a.ID }}},

			"node":   {Name: "node", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Node"}}},
			"status": {Name: "status", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Status"}}},
			"tags": {Name: "tags", AllowMissing: true,
				Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringSliceFieldIndex{Field: "Tags"}}},
		}},
	}}
}

func allocDB() *memdb.MemDB {
	db, err := memdb.NewMemDB(allocSchema())
	if err != nil {
		panic(err)
	}
	txn := db.Txn(true)
	for _, a := range []*Alloc{
		{ID: "a1", Node: "n1", Status: "running", Tags: []string{"batch"}},
		{ID: "a2", Node: "n1", Status: "running"},
		{ID: "a3", Node: "n1", Status: "failed", Tags: []string{"batch", "gpu"}},
		{ID: "a4", Node: "n2", Status: "running", Tags: []string{"gpu"}},
		{ID: "a5", Node: "n2", Status: "pending"},
	} {
		if err := txn.Insert("alloc", a); err != nil {
			panic(err)
		}
	}
	txn.Commit()
	return db
}

func main() {
	db := allocDB()
	txn := db.Txn(false)

	running, err := txn.Where("alloc", "status", "running")
	if err != nil {
		panic(err)
	}
	onN1, _ := txn.Where("alloc", "node", "n1")
	batch, _ := txn.Where("alloc", "tags", "batch")
	all, _ := txn.AllRows("alloc")

	fmt.Println("allocations:", all.Len())
	fmt.Println("running:", running.Len())
	fmt.Println("running on n1:", running.And(onN1).Len())
	fmt.Println("not running:", all.AndNot(running).Len())

	fmt.Print("running on n1, not batch:")
	for obj := range running.And(onN1).AndNot(batch).All() {
		fmt.Print(" ", obj.(*Alloc).ID)
	}
	fmt.Println()

}

Output:
allocations: 5
running: 3
running on n1: 2
not running: 2
running on n1, not batch: a2

func (*Txn) WhereWatch 

func (txn *Txn) WhereWatch(table, index string, args ...interface{}) (<-chan struct{}, RowSet, error)

WhereWatch is Where plus a channel that is closed when a later transaction changes the result: when a row enters or leaves the set. (A prefix query is watched like any prefix: the channel may also fire for a change to a neighbouring value.)

Example

A watched set fires when a row enters or leaves it -- and not when a row in it is merely replaced. 

package main

import (
	"fmt"

	memdb "github.com/thevilledev/go-maemmidb"
)

// Alloc is the row type of the examples of go-maemmidb's extensions.
type Alloc struct {
	ID     string
	Node   string
	Status string
	Tags   []string
}

func allocSchema() *memdb.DBSchema {
	return &memdb.DBSchema{Tables: map[string]*memdb.TableSchema{
		"alloc": {Name: "alloc", Indexes: map[string]*memdb.IndexSchema{

			"id": {Name: "id", Unique: true, Indexer: &memdb.StringIndex[Alloc]{Get: func(a *Alloc) string { return a.ID }}},

			"node":   {Name: "node", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Node"}}},
			"status": {Name: "status", Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringFieldIndex{Field: "Status"}}},
			"tags": {Name: "tags", AllowMissing: true,
				Indexer: &memdb.BitmapIndex{Indexer: &memdb.StringSliceFieldIndex{Field: "Tags"}}},
		}},
	}}
}

func allocDB() *memdb.MemDB {
	db, err := memdb.NewMemDB(allocSchema())
	if err != nil {
		panic(err)
	}
	txn := db.Txn(true)
	for _, a := range []*Alloc{
		{ID: "a1", Node: "n1", Status: "running", Tags: []string{"batch"}},
		{ID: "a2", Node: "n1", Status: "running"},
		{ID: "a3", Node: "n1", Status: "failed", Tags: []string{"batch", "gpu"}},
		{ID: "a4", Node: "n2", Status: "running", Tags: []string{"gpu"}},
		{ID: "a5", Node: "n2", Status: "pending"},
	} {
		if err := txn.Insert("alloc", a); err != nil {
			panic(err)
		}
	}
	txn.Commit()
	return db
}

func main() {
	db := allocDB()
	watch, failed, err := db.Txn(false).WhereWatch("alloc", "status", "failed")
	if err != nil {
		panic(err)
	}
	fired := func() bool {
		select {
		case <-watch:
			return true
		default:
			return false
		}
	}
	fmt.Println("failed:", failed.Len())

	txn := db.Txn(true)
	_ = txn.Insert("alloc", &Alloc{ID: "a3", Node: "n1", Status: "failed", Tags: []string{"retried"}})
	txn.Commit()
	fmt.Println("after replacing a failed allocation:", fired())

	txn = db.Txn(true)
	_ = txn.Insert("alloc", &Alloc{ID: "a5", Node: "n2", Status: "failed"})
	txn.Commit()
	fmt.Println("after another allocation failed:", fired())

}

Output:
failed: 1
after replacing a failed allocation: false
after another allocation failed: true

type UUIDFieldIndex 

type UUIDFieldIndex struct {
	Field string
}

UUIDFieldIndex is used to extract a field from an object using reflection and builds an index on that field by treating it as a UUID. This is an optimization to using a StringFieldIndex as the UUID can be more compactly represented in byte form.

func (*UUIDFieldIndex) FromArgs 

func (u *UUIDFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*UUIDFieldIndex) FromObject 

func (u *UUIDFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

func (*UUIDFieldIndex) PrefixFromArgs 

func (u *UUIDFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type UintFieldIndex 

type UintFieldIndex struct {
	Field string
}

UintFieldIndex is used to extract a uint field from an object using reflection and builds an index on that field.

func (*UintFieldIndex) FromArgs 

func (u *UintFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*UintFieldIndex) FromObject 

func (u *UintFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

type UintIndex 

type UintIndex[T any, N unsignedInt] struct {
	Get func(*T) N
	// contains filtered or unexported fields
}

UintIndex indexes an unsigned integer that a function extracts from the object: the typed twin of UintFieldIndex, with IntIndex's handling of query arguments.

func (*UintIndex[T, N]) FromArgs 

func (u *UintIndex[T, N]) FromArgs(args ...interface{}) ([]byte, error)

func (*UintIndex[T, N]) FromObject 

func (u *UintIndex[T, N]) FromObject(obj interface{}) (bool, []byte, error)

type UintKey 

type UintKey[T any, N unsignedInt] struct {
	// contains filtered or unexported fields
}

UintKey is IntKey for unsigned integers: an index built from a UintFieldIndex on a field of type N, or from a UintIndex.

func UintKeyOf 

func UintKeyOf[N unsignedInt, T any](t Table[T], index string) *UintKey[T, N]

UintKeyOf returns a typed handle on the named unsigned integer index.

func (*UintKey[T, N]) First 

func (k *UintKey[T, N]) First(txn *Txn, key N) (*T, error)

First returns the first row whose index value is key, like Txn.First.

func (*UintKey[T, N]) Get 

func (k *UintKey[T, N]) Get(txn *Txn, key N) (Rows[T], error)

Get returns the rows whose index value is key, like Txn.Get.

func (*UintKey[T, N]) LowerBound 

func (k *UintKey[T, N]) LowerBound(txn *Txn, key N) (Rows[T], error)

LowerBound returns the rows whose index value is at least key, in ascending order, like Txn.LowerBound.

type WatchSet 

type WatchSet map[<-chan struct{}]struct{}

WatchSet is a collection of watch channels. The zero value is not usable. Use NewWatchSet to create a WatchSet.

func NewWatchSet 

func NewWatchSet() WatchSet

NewWatchSet constructs a new watch set.

func (WatchSet) Add 

func (w WatchSet) Add(watchCh <-chan struct{})

Add appends a watchCh to the WatchSet if non-nil.

func (WatchSet) AddWithLimit 

func (w WatchSet) AddWithLimit(softLimit int, watchCh <-chan struct{}, altCh <-chan struct{})

AddWithLimit appends a watchCh to the WatchSet if non-nil, and if the given softLimit hasn't been exceeded. Otherwise, it will watch the given alternate channel. It's expected that the altCh will be the same on many calls to this function, so you will exceed the soft limit a little bit if you hit this, but not by much.

This is useful if you want to track individual items up to some limit, after which you watch a higher-level channel (usually a channel from start of an iterator higher up in the radix tree) that will watch a superset of items.

func (WatchSet) Watch 

func (w WatchSet) Watch(timeoutCh <-chan time.Time) bool

Watch blocks until one of the channels in the watch set is closed, or timeoutCh sends a value. Returns true if timeoutCh is what caused Watch to unblock.

func (WatchSet) WatchCh 

func (w WatchSet) WatchCh(ctx context.Context) <-chan error

WatchCh returns a channel that is used to wait for any channel of the watch set to trigger or for the context to be cancelled. WatchCh creates a new goroutine each call, so callers may need to cache the returned channel to avoid creating extra goroutines.

func (WatchSet) WatchCtx 

func (w WatchSet) WatchCtx(ctx context.Context) error

WatchCtx blocks until one of the channels in the watch set is closed, or ctx is done (cancelled or exceeds the deadline). WatchCtx returns an error if the ctx causes it to unblock, otherwise returns nil.

WatchCtx should be preferred over Watch.

Source Files

View all Source files

Directories

Path Synopsis
internal
bitmap
Package bitmap implements the sets of row ids behind go-maemmidb's bitmap indexes: a persistent (immutable, structurally shared) compressed bitmap over uint32.
 Package bitmap implements the sets of row ids behind go-maemmidb's bitmap indexes: a persistent (immutable, structurally shared) compressed bitmap over uint32.
pvec
Package pvec implements the row table behind go-maemmidb's bitmap indexes: a persistent (immutable, structurally shared) vector that maps a row id to the row's object.
 Package pvec implements the row table behind go-maemmidb's bitmap indexes: a persistent (immutable, structurally shared) vector that maps a row id to the row's object.
radix
Package radix implements the storage engine of go-maemmidb: a persistent (immutable, structurally shared), path-compressed radix tree.
 Package radix implements the storage engine of go-maemmidb: a persistent (immutable, structurally shared), path-compressed radix tree.
This tool generates the special-case code for a small number of watchers which runs all the watches in a single select vs.
 This tool generates the special-case code for a small number of watchers which runs all the watches in a single select vs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.