lmdbstore

README

Introduction

This package (lmdbstore) wraps the github.com/bmatsuo/lmdb-go/lmdb with convenient methods and defaults:

• Multiple databases within one lmdb.Env
• One goroutine that handles updates
• Convenient per database Get, Put, Del, and Drop methods
• Customizable (per lmdb.Env or database) Marshal and Unmarshal methods
• Defaults to the performant github.com/shamaton/msgpack/v2 for Marshal and Unmarshal

Usage

For a single database use case on the directory where the go program is in,

package main

import (
    "gitlab.com/benedictjohannes/lmbdbstore"
)
var db *lmdbstore.Db

func initLmdbStore() error {
    // initialize LmdbEnv
	lmdbEnv, err := lmdbstore.NewLmdb(lmdbstore.DefaultLmdbConfig)
	if err != nil {
        return err
	}
	// get the default database
	db, err = lmdbEnv.GetSingleDatabase()
	return err
}
type example struct {
    Id          int
    Description string
}
func main() {
	err := initLmdbStore()
	if err != nil {
		fmt.Println("Error initiating LMDB database: ", err)
		return
	}
	err = db.Put([]byte("your key"), example{888, "Fortune Cookies"})
	if err != nil {
		fmt.Println("Failed to save into database: ", err)
		return
	}
	var exampleVar example
	err = db.GetAndMarshal([]byte("your key"), &exampleVar)
	if err != nil {
		fmt.Println("Failed to get saved avlue from database: ", err)
		return
	}
	fmt.Println(exampleVar)
}

Another wrapper?

Using lmdb directly almost requires either writing a lot of wrapper methods or duplicating code across users of the database. The defaults this package initializes for its callers should be adequate for most use cases, while the convenience methods can make interacting with lmdb easier in Go programs.

TODOs

This package should not change the exported API, but these two should be taken care of before marking the package as stable (semver >=v1.0.0).

• Export error variables to identify type of error (like key not existing when running Db.Get())
• Tests
• Update of the underlying lmdb-go package

Documentation

Index

• Variables
• type Db
  • func (s *Db) Del(key []byte) error
  • func (s *Db) Drop() error
  • func (s *Db) Get(key []byte) (b []byte, err error)
  • func (s *Db) GetAndMarshal(key []byte, dest interface{}) (err error)
  • func (s *Db) Put(key []byte, value interface{}) error
  • func (s *Db) UpdateTxn(op lmdb.TxnOp) error
• type DbConfig
• type LmdbEnv
  • func NewLmdb(config LmdbEnvConfig) (*LmdbEnv, error)
  • func (e *LmdbEnv) Close()
  • func (l *LmdbEnv) GetDatabase(dbName string) *Db
  • func (l *LmdbEnv) GetSingleDatabase() (*Db, error)
• type LmdbEnvConfig

Variables

var DefaultLmdbConfig = LmdbEnvConfig{
	OpenPath:   ".",
	OpenFSMode: 0644,
	MapSize:    1 << 30,
	MaxReaders: 1,
	Databases:  []DbConfig{{DbName: "default"}},
	Marshal:    msgpack.MarshalAsArray,
	Unmarshal:  msgpack.UnmarshalAsArray,
}

Types

type Db

type Db struct {
	// contains filtered or unexported fields
}

Db reperesents a single Database inside LmdbEnv

Db should always be accessed through LmdbEnv.GetDatabase(dbName) or LmdbEnv.GetSingleDatabase()

Do not create Db struct directly

func (*Db) Del

func (s *Db) Del(key []byte) error

Del a value with key inside the database

The call will block until the transaction is finished

func (*Db) Drop

func (s *Db) Drop() error

Drop empties a database

The call will block until the transaction is finished

func (*Db) Get

func (s *Db) Get(key []byte) (b []byte, err error)

Get returns the binary value at key inside the database

If the key does not exist, an error is returned

The returned value is copied for safe use outside the lmdb.TxnOp

func (*Db) GetAndMarshal

func (s *Db) GetAndMarshal(key []byte, dest interface{}) (err error)

GetAndMarshal marshals value at key into &dest

If the key does not exist, an error is returned

The value is first copied for safe use outside the lmdb.TxnOp

Returned value is safe to use across goroutines

func (*Db) Put

func (s *Db) Put(key []byte, value interface{}) error

Put a value with key inside the database

The call will block until the transaction is finished

func (*Db) UpdateTxn

func (s *Db) UpdateTxn(op lmdb.TxnOp) error

UpdateTxn runs a lmdb.TxnOp inside the updater goroutine

The call will block until the transaction is finished

type DbConfig

type DbConfig struct {
	DbName    string
	Marshal   func(v interface{}) ([]byte, error)
	Unmarshal func(data []byte, v interface{}) error
}

DbConfig is configuration that will be created as entries in LmdbEnv.Databases

Marshal and Unmarshal are optional and defaults to the parent LmdbEnv's methods.

Different Marshal and Unmarshal per database is possible, but should never change for the lifetime of the database.

type LmdbEnv

type LmdbEnv struct {
	// Direct access to *lmdb.Env
	LmdbEnv *lmdb.Env
	// contains filtered or unexported fields
}

LmdbEnv wraps LmdbEnv with tested configurations

LmdbEnv should always be created by calling NewLmdb(LmdbEnvConfig)

Do not create the struct directly

func NewLmdb

func NewLmdb(config LmdbEnvConfig) (*LmdbEnv, error)

NewLmdb initialize a single LmdbEnv

Initialized LmdbEnv would spawn a single "updater" goroutine for Update transactions

The methods should be safe to use across multiple goroutines

func (*LmdbEnv) Close

func (e *LmdbEnv) Close()

Close flushes the Lmdb databases to disk and stop the updater goroutine

Note that closed LmdbEnv should not be used for any transactions

func (*LmdbEnv) GetDatabase

func (l *LmdbEnv) GetDatabase(dbName string) *Db

GetDatabase returns a database with name dbName

Non-existing entries will return nil

func (*LmdbEnv) GetSingleDatabase

func (l *LmdbEnv) GetSingleDatabase() (*Db, error)

GetSingleDatabase returns a single database

if Databases contains more than a single entry, GetSingleDatabase returns an error

type LmdbEnvConfig

type LmdbEnvConfig struct {
	OpenPath   string
	OpenFlag   uint
	OpenFSMode fs.FileMode
	MapSize    int64
	MaxReaders int
	// minimum 1 entry
	Databases []DbConfig
	// optional
	Marshal func(v interface{}) ([]byte, error)
	// optional
	Unmarshal func(data []byte, v interface{}) error
}

LmdbEnvConfig is configuration for LmdbEnv

There should be minimum 1 Databases []DbConfig entry

Marshal and Unmarshal are optional and defaults to github.com/shamaton/msgpack

All other fields should be set