gokv

README

gokv

Simple key-value store abstraction and implementations for Go

Features

Simple interface:

Note: The interface is not final yet! See Project status for details.

// Store is an abstraction for different key-value store implementations.
// A store must be able to store and retrieve key-value pairs,
// with the key being a string and the value being any Go interface{}.
type Store interface {
	// Set stores the given value for the given key.
	// The implementation automatically marshalls the value if required.
	// The marshalling target depends on the implementation. It can be JSON, gob etc.
	// Implementations should offer a configuration for this.
	Set(string, interface{}) error
	// Get retrieves the value for the given key.
	// The implementation automatically unmarshalls the value if required.
	// The unmarshalling source depends on the implementation. It can be JSON, gob etc.
	// The automatic unmarshalling requires a pointer to a proper type being passed as parameter.
	// The Get method will populate the fields of the object that the passed pointer
	// points to with the values of the retrieved object's values.
	// If no object is found it returns (false, nil).
	Get(string, interface{}) (bool, error)
}

Implementations ():

• Local in-memory
  • Go map (sync.Map)
    • Faster then a regular map when there are very few writes but lots of reads
  • Go map (map[string]interface{} with sync.RWMutex)
• Embedded
  • bbolt (formerly known as Bolt / Bolt DB)
    • bbolt is a fork of Bolt which was maintained by CoreOS, and now by RedHat (since CoreOS was acquired by them)
    • It's used for example in etcd as underlying persistent store
  • BadgerDB
    • Very similar to bbolt / Bolt, where bbolt is generally faster for reads, and Badger is generally faster for writes
  • LevelDB / goleveldb
• Distributed
  • Redis
  • Consul
  • etcd
    • Not advertised as general KV store, but can be used as one(https://www.cncf.io/blog/2018/08/28/cncf-to-host-tikv-in-the-sandbox/)
  • Memcached
  • Hazelcast
  • LedisDB
    • Similar to Redis, with several backing stores
  • TiKV
    • Originally created to complement TiDB, but recently [became a project in the CNCF]
• Cloud
  • Amazon DynamoDB
  • Amazon SimpleDB
  • Azure Cosmos DB
  • Azure Table Storage
  • Google Cloud Datastore

Some other databases aren't specifically engineered for storing key-value pairs, but if someone's running them already for other purposes and doesn't want to set up one of the proper key-value stores due to administrative overhead etc., they can of course be used as well. Let's focus on a few of the most popular though:

• SQL
  • MySQL
  • PostgreSQL
• NoSQL
  • MongoDB
• NewSQL
  • CockroachDB
  • TiDB

Project status

Note: gokv's API is not stable yet and is under active development. Upcoming releases are likely to contain breaking changes as long as the version is v0.x.y. You should use vendoring to prevent bad surprises. This project adheres to Semantic Versioning and all notable changes to this project are documented in RELEASES.md.

Planned interface methods until v1.0.0:

• Delete(string) error or similar
• List(interface{}) error / GetAll(interface{}) error or similar

Motivation

Initially developed as storage package within the project ln-paywall to provide the users of ln-paywall with multiple storage options, at some point it made sense to turn it into a repository of its own.

Before doing so I examined existing Go packages with a similar purpose (see Related projects, but none of them fit my needs. They either had too few implementations, or they didn't automatically marshal / unmarshal passed structs, or the interface had too many methods, making the project seem too complex to maintain and extend, proven by some that were abandoned or forked (splitting the community with it).

Related projects

• libkv
  • Uses []byte as value, no automatic (un-)marshalling of structs
  • No support for Redis
  • Not actively maintained anymore (3 direct commits + 1 merged PR in the last 10+ months, as of 2018-10-13)
• valkeyrie
  • Fork of libkv
  • Same disadvantage: Uses []byte as value, no automatic (un-)marshalling of structs
• gokvstores
  • Only supports Redis and local in-memory cache
  • Not actively maintained anymore (4 direct commits + 1 merged PR in the last 10+ months, as of 2018-10-13)
  • Only 13 stars (as of 2018-10-13)
• gokv
  • Requires a json.Marshaler / json.Unmarshaler as parameter, so you always need to explicitly implement their methods for your structs
  • Separate repo for each implementation, which has advantages and disadvantages
  • Single contributor
  • No releases (makes it harder to use with package managers like dep)
  • Only 2-7 stars (depending on the repository, as of 2018-10-13)
  • No support for Bolt DB / bbolt

Documentation

Overview

Package gokv contains a simple key-value store abstraction and multiple implementations.

Index

• Variables
• type BoltClient
  • func NewBoltClient(boltOptions BoltOptions) (BoltClient, error)
  • func (c BoltClient) Get(k string, v interface{}) (bool, error)
  • func (c BoltClient) Set(k string, v interface{}) error
• type BoltOptions
• type GoMap
  • func NewGoMap() GoMap
  • func (m GoMap) Get(k string, v interface{}) (bool, error)
  • func (m GoMap) Set(k string, v interface{}) error
• type RedisClient
  • func NewRedisClient(redisOptions RedisOptions) RedisClient
  • func (c RedisClient) Get(k string, v interface{}) (bool, error)
  • func (c RedisClient) Set(k string, v interface{}) error
• type RedisOptions
• type Store

Variables

var DefaultBoltOptions = BoltOptions{
	BucketName: "default",
	Path:       "bolt.db",
}

DefaultBoltOptions is a BoltOptions object with default values. BucketName: "default", Path: "bolt.db"

var DefaultRedisOptions = RedisOptions{
	Address: "localhost:6379",
}

DefaultRedisOptions is a RedisOptions object with default values. Address: "localhost:6379", Password: "", DB: 0

Types

type BoltClient

type BoltClient struct {
	// contains filtered or unexported fields
}

BoltClient is a gokv.Store implementation for bbolt (formerly known as Bolt / Bolt DB).

func NewBoltClient

func NewBoltClient(boltOptions BoltOptions) (BoltClient, error)

NewBoltClient creates a new BoltClient. Note: Bolt uses an exclusive write lock on the database file so it cannot be shared by multiple processes. So when creating multiple Bolt clients you should always use a new database file (by setting a different Path in the BoltOptions).

Don't worry about closing the Bolt DB as long as you don't need to close the DB while the process that opened it runs.

func (BoltClient) Get

func (c BoltClient) Get(k string, v interface{}) (bool, error)

Get retrieves the stored object for the given key and populates the fields of the object that v points to with the values of the retrieved object's values.

func (BoltClient) Set

func (c BoltClient) Set(k string, v interface{}) error

Set stores the given object for the given key.

type BoltOptions

type BoltOptions struct {
	// Bucket name for storing the key-value pairs.
	// Optional ("default" by default).
	BucketName string
	// Path of the DB file.
	// Optional ("bolt.db" by default).
	Path string
}

BoltOptions are the options for the BoltClient.

type GoMap

type GoMap struct {
	// contains filtered or unexported fields
}

GoMap is a gokv.Store implementation for a simple Go sync.Map.

func NewGoMap

func NewGoMap() GoMap

NewGoMap creates a new GoMap.

func (GoMap) Get

func (m GoMap) Get(k string, v interface{}) (bool, error)

Get retrieves the stored object for the given key and populates the fields of the object that v points to with the values of the retrieved object's values.

func (GoMap) Set

func (m GoMap) Set(k string, v interface{}) error

Set stores the given object for the given key.

type RedisClient

type RedisClient struct {
	// contains filtered or unexported fields
}

RedisClient is a gokv.Store implementation for Redis.

func NewRedisClient

func NewRedisClient(redisOptions RedisOptions) RedisClient

NewRedisClient creates a new RedisClient.

func (RedisClient) Get

func (c RedisClient) Get(k string, v interface{}) (bool, error)

Get retrieves the object for the given key and points the passed pointer to it.

func (RedisClient) Set

func (c RedisClient) Set(k string, v interface{}) error

Set stores the given object for the given key.

type RedisOptions

type RedisOptions struct {
	// Address of the Redis server, including the port.
	// Optional ("localhost:6379" by default).
	Address string
	// Password for the Redis server.
	// Optional ("" by default).
	Password string
	// DB to use.
	// Optional (0 by default).
	DB int
}

RedisOptions are the options for the Redis DB.

type Store

type Store interface {
	// Set stores the given value for the given key.
	// The implementation automatically marshalls the value if required.
	// The marshalling target depends on the implementation. It can be JSON, gob etc.
	// Implementations should offer a configuration for this.
	Set(string, interface{}) error
	// Get retrieves the value for the given key.
	// The implementation automatically unmarshalls the value if required.
	// The unmarshalling source depends on the implementation. It can be JSON, gob etc.
	// The automatic unmarshalling requires a pointer to a proper type being passed as parameter.
	// The Get method will populate the fields of the object that the passed pointer
	// points to with the values of the retrieved object's values.
	// If no object is found it returns (false, nil).
	Get(string, interface{}) (bool, error)
}

Store is an abstraction for different key-value store implementations. A store must be able to store and retrieve key-value pairs, with the key being a string and the value being any Go interface{}.