memdb

package module
v0.0.0-...-032f93b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 8, 2018 License: MPL-2.0 Imports: 12 Imported by: 0

README

go-memdb

Provides the memdb package that implements a simple in-memory database built on immutable radix trees. The database provides Atomicity, Consistency and Isolation from ACID. Being that it is in-memory, it does not provide durability. The database is instantiated with a schema that specifies the tables and indices that exist and allows transactions to be executed.

The database provides the following:

  • Multi-Version Concurrency Control (MVCC) - By leveraging immutable radix trees the database is able to support any number of concurrent readers without locking, and allows a writer to make progress.

  • Transaction Support - The database allows for rich transactions, in which multiple objects are inserted, updated or deleted. The transactions can span multiple tables, and are applied atomically. The database provides atomicity and isolation in ACID terminology, such that until commit the updates are not visible.

  • Rich Indexing - Tables can support any number of indexes, which can be simple like a single field index, or more advanced compound field indexes. Certain types like UUID can be efficiently compressed from strings into byte indexes for reduced storage requirements.

  • Watches - Callers can populate a watch set as part of a query, which can be used to detect when a modification has been made to the database which affects the query results. This lets callers easily watch for changes in the database in a very general way.

For the underlying immutable radix trees, see go-immutable-radix.

Documentation

The full documentation is available on Godoc.

Example

Below is a simple example of usage

// Create a sample struct
type Person struct {
    Email string
    Name  string
    Age   int
}

// Create the DB schema
schema := &memdb.DBSchema{
    Tables: map[string]*memdb.TableSchema{
        "person": &memdb.TableSchema{
            Name: "person",
            Indexes: map[string]*memdb.IndexSchema{
                "id": &memdb.IndexSchema{
                    Name:    "id",
                    Unique:  true,
                    Indexer: &memdb.StringFieldIndex{Field: "Email"},
                },
            },
        },
    },
}

// Create a new data base
db, err := memdb.NewMemDB(schema)
if err != nil {
    panic(err)
}

// Create a write transaction
txn := db.Txn(true)

// Insert a new person
p := &Person{"joe@aol.com", "Joe", 30}
if err := txn.Insert("person", p); err != nil {
    panic(err)
}

// Commit the transaction
txn.Commit()

// Create read-only transaction
txn = db.Txn(false)
defer txn.Abort()

// Lookup by email
raw, err := txn.First("person", "id", "joe@aol.com")
if err != nil {
    panic(err)
}

// Say hi!
fmt.Printf("Hello %s!", raw.(*Person).Name)

Documentation

Overview

Package memdb provides an in-memory database that supports transactions and MVCC.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound is returned when the requested item is not found
	ErrNotFound = fmt.Errorf("not found")
)
View Source 
var MapType = reflect.MapOf(reflect.TypeOf(""), reflect.TypeOf("")).Kind()

Functions

func IsUintType 

func IsUintType(k reflect.Kind) (size int, okay bool)

IsUintType returns whether the passed type is a type of uint and the number of bytes needed to encode the type.

Types

type CompoundIndex 

type CompoundIndex struct {
	Indexes []Indexer

	// AllowMissing results in an index based on only the indexers
	// that return data. If true, you may end up with 2/3 columns
	// indexed which might be useful for an index scan. Otherwise,
	// the CompoundIndex requires all indexers to be satisfied.
	AllowMissing bool
}

CompoundIndex is used to build an index using multiple sub-indexes Prefix based iteration is supported as long as the appropriate prefix of indexers support it. All sub-indexers are only assumed to expect a single argument.

func (*CompoundIndex) FromArgs 

func (c *CompoundIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*CompoundIndex) FromObject 

func (c *CompoundIndex) FromObject(raw interface{}) (bool, []byte, error)

func (*CompoundIndex) PrefixFromArgs 

func (c *CompoundIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type ConditionalIndex 

type ConditionalIndex struct {
	Conditional ConditionalIndexFunc
}

ConditionalIndex builds an index based on a condition specified by a passed user function. This function may examine the passed object and return a boolean to encapsulate an arbitrarily complex conditional.

func (*ConditionalIndex) FromArgs 

func (c *ConditionalIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*ConditionalIndex) FromObject 

func (c *ConditionalIndex) FromObject(obj interface{}) (bool, []byte, error)

type ConditionalIndexFunc 

type ConditionalIndexFunc func(obj interface{}) (bool, error)

ConditionalIndexFunc is the required function interface for a ConditionalIndex.

type DBSchema 

type DBSchema struct {
	// Tables is the set of tables within this database. The key is the
	// table name and must match the Name in TableSchema.
	Tables map[string]*TableSchema
}

DBSchema is the schema to use for the full database with a MemDB instance.

MemDB will require a valid schema. Schema validation can be tested using the Validate function. Calling this function is recommended in unit tests.

func (*DBSchema) Validate 

func (s *DBSchema) Validate() error

Validate validates the schema.

type FieldSetIndex 

type FieldSetIndex struct {
	Field string
}

FieldSetIndex is used to extract a field from an object using reflection and builds an index on whether the field is set by comparing it against its type's nil value.

func (*FieldSetIndex) FromArgs 

func (f *FieldSetIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*FieldSetIndex) FromObject 

func (f *FieldSetIndex) FromObject(obj interface{}) (bool, []byte, error)

type FilterFunc 

type FilterFunc func(interface{}) bool

FilterFunc is a function that takes the results of an iterator and returns whether the result should be filtered out.

type FilterIterator 

type FilterIterator struct {
	// contains filtered or unexported fields
}

FilterIterator is used to wrap a ResultIterator and apply a filter over it.

func NewFilterIterator 

func NewFilterIterator(wrap ResultIterator, filter FilterFunc) *FilterIterator

func (*FilterIterator) Next 

func (f *FilterIterator) Next() interface{}

Next returns the next non-filtered result from the wrapped iterator

func (*FilterIterator) WatchCh 

func (f *FilterIterator) WatchCh() <-chan struct{}

WatchCh returns the watch channel of the wrapped iterator.

type IndexSchema 

type IndexSchema struct {
	// Name of the index. This must be unique among a tables set of indexes.
	// This must match the key in the map of Indexes for a TableSchema.
	Name string

	// AllowMissing if true ignores this index if it doesn't produce a
	// value. For example, an index that extracts a field that doesn't
	// exist from a structure.
	AllowMissing bool

	Unique  bool
	Indexer Indexer
}

IndexSchema is the schema for an index. An index defines how a table is queried.

func (*IndexSchema) Validate 

func (s *IndexSchema) Validate() error

type Indexer 

type Indexer interface {
	// FromArgs is called to build the exact index key from a list of arguments.
	FromArgs(args ...interface{}) ([]byte, error)
}

Indexer is an interface used for defining indexes. Indexes are used for efficient lookup of objects in a MemDB table. An Indexer must also implement one of SingleIndexer or MultiIndexer.

Indexers are primarily responsible for returning the lookup key as a byte slice. The byte slice is the key data in the underlying data storage.

type MemDB 

type MemDB struct {
	// contains filtered or unexported fields
}

MemDB is an in-memory database.

MemDB provides a table abstraction to store objects (rows) with multiple indexes based on inserted values. The database makes use of immutable radix trees to provide transactions and MVCC.

func NewMemDB 

func NewMemDB(schema *DBSchema) (*MemDB, error)

NewMemDB creates a new MemDB with the given schema

func (*MemDB) Snapshot 

func (db *MemDB) Snapshot() *MemDB

Snapshot is used to capture a point-in-time snapshot of the database that will not be affected by any write operations to the existing DB.

func (*MemDB) Txn 

func (db *MemDB) Txn(write bool) *Txn

Txn is used to start a new transaction, in either read or write mode. There can only be a single concurrent writer, but any number of readers.

type MultiIndexer 

type MultiIndexer interface {
	// FromObject extracts index values from an object. The return values
	// are the same as a SingleIndexer except there can be multiple index
	// values.
	FromObject(raw interface{}) (bool, [][]byte, error)
}

MultiIndexer is an interface used for defining indexes that generate multiple values per object. Each value is stored as a seperate index pointing to the same object.

For example, an index that extracts the first and last name of a person and allows lookup based on eitherd would be a MultiIndexer. The FromObject of this example would split the first and last name and return both as values.

type PrefixIndexer 

type PrefixIndexer interface {
	// PrefixFromArgs is the same as FromArgs for an Indexer except that
	// the index value returned should return all prefix-matched values.
	PrefixFromArgs(args ...interface{}) ([]byte, error)
}

PrefixIndexer is an optional interface on top of an Indexer that allows indexes to support prefix-based iteration.

type ResultIterator 

type ResultIterator interface {
	WatchCh() <-chan struct{}
	Next() interface{}
}

ResultIterator is used to iterate over a list of results from a Get query on a table.

type SingleIndexer 

type SingleIndexer interface {
	// FromObject extracts the index value from an object. The return values
	// are whether the index value was found, the index value, and any error
	// while extracting the index value, respectively.
	FromObject(raw interface{}) (bool, []byte, error)
}

SingleIndexer is an interface used for defining indexes that generate a single value per object

type StringFieldIndex 

type StringFieldIndex struct {
	Field     string
	Lowercase bool
}

StringFieldIndex is used to extract a field from an object using reflection and builds an index on that field.

func (*StringFieldIndex) FromArgs 

func (s *StringFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*StringFieldIndex) FromObject 

func (s *StringFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

func (*StringFieldIndex) PrefixFromArgs 

func (s *StringFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type StringMapFieldIndex 

type StringMapFieldIndex struct {
	Field     string
	Lowercase bool
}

StringMapFieldIndex is used to extract a field of type map[string]string from an object using reflection and builds an index on that field.

func (*StringMapFieldIndex) FromArgs 

func (s *StringMapFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*StringMapFieldIndex) FromObject 

func (s *StringMapFieldIndex) FromObject(obj interface{}) (bool, [][]byte, error)

type StringSliceFieldIndex 

type StringSliceFieldIndex struct {
	Field     string
	Lowercase bool
}

StringSliceFieldIndex builds an index from a field on an object that is a string slice ([]string). Each value within the string slice can be used for lookup.

func (*StringSliceFieldIndex) FromArgs 

func (s *StringSliceFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*StringSliceFieldIndex) FromObject 

func (s *StringSliceFieldIndex) FromObject(obj interface{}) (bool, [][]byte, error)

func (*StringSliceFieldIndex) PrefixFromArgs 

func (s *StringSliceFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type TableSchema 

type TableSchema struct {
	// Name of the table. This must match the key in the Tables map in DBSchema.
	Name string

	// Indexes is the set of indexes for querying this table. The key
	// is a unique name for the index and must match the Name in the
	// IndexSchema.
	Indexes map[string]*IndexSchema
}

TableSchema is the schema for a single table.

func (*TableSchema) Validate 

func (s *TableSchema) Validate() error

Validate is used to validate the table schema

type Txn 

type Txn struct {
	// contains filtered or unexported fields
}

Txn is a transaction against a MemDB. This can be a read or write transaction.

func (*Txn) Abort 

func (txn *Txn) Abort()

Abort is used to cancel this transaction. This is a noop for read transactions.

func (*Txn) Commit 

func (txn *Txn) Commit()

Commit is used to finalize this transaction. This is a noop for read transactions.

func (*Txn) Defer 

func (txn *Txn) Defer(fn func())

Defer is used to push a new arbitrary function onto a stack which gets called when a transaction is committed and finished. Deferred functions are called in LIFO order, and only invoked at the end of write transactions.

func (*Txn) Delete 

func (txn *Txn) Delete(table string, obj interface{}) error

Delete is used to delete a single object from the given table This object must already exist in the table

func (*Txn) DeleteAll 

func (txn *Txn) DeleteAll(table, index string, args ...interface{}) (int, error)

DeleteAll is used to delete all the objects in a given table matching the constraints on the index

func (*Txn) DeletePrefix 

func (txn *Txn) DeletePrefix(table string, prefix_index string, prefix string) (bool, error)

DeletePrefix is used to delete an entire subtree based on a prefix. The given index must be a prefix index, and will be used to perform a scan and enumerate the set of objects to delete. These will be removed from all other indexes, and then a special prefix operation will delete the objects from the given index in an efficient subtree delete operation. This is useful when you have a very large number of objects indexed by the given index, along with a much smaller number of entries in the other indexes for those objects.

func (*Txn) First 

func (txn *Txn) First(table, index string, args ...interface{}) (interface{}, error)

First is used to return the first matching object for the given constraints on the index

func (*Txn) FirstWatch 

func (txn *Txn) FirstWatch(table, index string, args ...interface{}) (<-chan struct{}, interface{}, error)

FirstWatch is used to return the first matching object for the given constraints on the index along with the watch channel

func (*Txn) Get 

func (txn *Txn) Get(table, index string, args ...interface{}) (ResultIterator, error)

Get is used to construct a ResultIterator over all the rows that match the given constraints of an index.

func (*Txn) Insert 

func (txn *Txn) Insert(table string, obj interface{}) error

Insert is used to add or update an object into the given table

func (*Txn) LongestPrefix 

func (txn *Txn) LongestPrefix(table, index string, args ...interface{}) (interface{}, error)

LongestPrefix is used to fetch the longest prefix match for the given constraints on the index. Note that this will not work with the memdb StringFieldIndex because it adds null terminators which prevent the algorithm from correctly finding a match (it will get to right before the null and fail to find a leaf node). This should only be used where the prefix given is capable of matching indexed entries directly, which typically only applies to a custom indexer. See the unit test for an example.

type UUIDFieldIndex 

type UUIDFieldIndex struct {
	Field string
}

UUIDFieldIndex is used to extract a field from an object using reflection and builds an index on that field by treating it as a UUID. This is an optimization to using a StringFieldIndex as the UUID can be more compactly represented in byte form.

func (*UUIDFieldIndex) FromArgs 

func (u *UUIDFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*UUIDFieldIndex) FromObject 

func (u *UUIDFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

func (*UUIDFieldIndex) PrefixFromArgs 

func (u *UUIDFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error)

type UintFieldIndex 

type UintFieldIndex struct {
	Field string
}

UintFieldIndex is used to extract a uint field from an object using reflection and builds an index on that field.

func (*UintFieldIndex) FromArgs 

func (u *UintFieldIndex) FromArgs(args ...interface{}) ([]byte, error)

func (*UintFieldIndex) FromObject 

func (u *UintFieldIndex) FromObject(obj interface{}) (bool, []byte, error)

type WatchSet 

type WatchSet map[<-chan struct{}]struct{}

WatchSet is a collection of watch channels.

func NewWatchSet 

func NewWatchSet() WatchSet

NewWatchSet constructs a new watch set.

func (WatchSet) Add 

func (w WatchSet) Add(watchCh <-chan struct{})

Add appends a watchCh to the WatchSet if non-nil.

func (WatchSet) AddWithLimit 

func (w WatchSet) AddWithLimit(softLimit int, watchCh <-chan struct{}, altCh <-chan struct{})

AddWithLimit appends a watchCh to the WatchSet if non-nil, and if the given softLimit hasn't been exceeded. Otherwise, it will watch the given alternate channel. It's expected that the altCh will be the same on many calls to this function, so you will exceed the soft limit a little bit if you hit this, but not by much.

This is useful if you want to track individual items up to some limit, after which you watch a higher-level channel (usually a channel from start start of an iterator higher up in the radix tree) that will watch a superset of items.

func (WatchSet) Watch 

func (w WatchSet) Watch(timeoutCh <-chan time.Time) bool

Watch is used to wait for either the watch set to trigger or a timeout. Returns true on timeout.

func (WatchSet) WatchCtx 

func (w WatchSet) WatchCtx(ctx context.Context) error

WatchCtx is used to wait for either the watch set to trigger or for the context to be cancelled. Watch with a timeout channel can be mimicked by creating a context with a deadline. WatchCtx should be preferred over Watch.

Source Files

View all Source files

Directories

Path Synopsis
This tool generates the special-case code for a small number of watchers which runs all the watches in a single select vs.
 This tool generates the special-case code for a small number of watchers which runs all the watches in a single select vs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.