kvpack

README

kvpack

Performant marshaler and unmarshaler library for transactional key-value databases.

Design

kvpack is meant to be simple in design, both for the end-user and the library developer. Even though lots of care will be taken to optimize for performance internally, the API surface should always be kept simple.

Because of this, kvpack will not read any struct tags. The API surface will always be kept to a bare minimum.

Examples

See the reference documentation, section Examples.

Supported Types

The following kinds of types are supported:

• Slices
• Strings
• Booleans
• All numbers (uint?, int?, float?, complex?, etc.)
• Structs

The following kinds of types are not supported:

• Maps, because they're too complicated
• Arrays (TODO)

Performance

Note that these benchmarks should only be used to relatively compare how much performance is compromised by having the ability to arbitrarily get and set keys inside a struct. If your application does not benefit from having this ability, then it most likely doesn't need kvpack.

Performance of these benchmarks heavily depend on the underlying database driver, the complexity of the data structure and the size of the value of that structure. The benchmarks below use a fairly simple struct to compare between kvpack and encoding/json. The benchmarks are taken from the GitLab CI results.

pkg: github.com/diamondburned/kvpack/driver/bboltpack
cpu: Intel(R) Xeon(R) CPU @ 2.30GHz
BenchmarkSuite/PutKVPack         	   55287	     20924 ns/op	    9753 B/op	      62 allocs/op
BenchmarkSuite/PutJSON           	   54154	     21949 ns/op	    7513 B/op	      53 allocs/op
BenchmarkSuite/GetKVPack         	  312518	      3870 ns/op	    1039 B/op	      12 allocs/op
BenchmarkSuite/GetJSON           	  271411	      4639 ns/op	     404 B/op	      14 allocs/op

The data structure and value of the benchmark data

// CharacterData is a struct used for benchmarking.
type CharacterData struct {
	BestCharacter   string
	CharacterScore  int32
	OtherCharacters []CharacterData
}

// NewCharacterData creates a character data value with dummy values.
func NewCharacterData() CharacterData {
	return CharacterData{
		BestCharacter:  "Astolfo",
		CharacterScore: 100,
		OtherCharacters: []CharacterData{
			{"Felix Argyle", 100, nil},
			{"Hime Arikawa", 100, nil},
		},
	}
}

Documentation

Index

• Constants
• Variables
• type Database
  • func NewDatabase(db driver.Database) *Database
  • func (db *Database) Begin(readOnly bool) (*Transaction, error)
  • func (db *Database) Close() error
  • func (db *Database) Delete(prefix []byte) error
  • func (db *Database) DeleteFields(fields string) error
  • func (db *Database) Descend(namespaces ...string) *Database
  • func (db *Database) Each(fields string, v interface{}, eachFn func(k []byte) error) error
  • func (db *Database) Get(k []byte, v interface{}) error
  • func (db *Database) GetFields(fields string, v interface{}) error
  • func (db *Database) Namespace() string
  • func (db *Database) Put(k []byte, v interface{}) error
  • func (db *Database) PutFields(fields string, v interface{}) error
  • func (db *Database) Update(f func(*Transaction) error) error
  • func (db *Database) View(f func(*Transaction) error) error
  • func (db *Database) WithNamespace(namespaces ...string) *Database
• type Transaction
  • func NewTransaction(tx driver.Transaction, fullNamespace []byte, ro bool) *Transaction
  • func (tx *Transaction) Commit() error
  • func (tx *Transaction) Delete(k []byte) error
  • func (tx *Transaction) DeleteFields(fields string) error
  • func (tx *Transaction) Each(fields string, v interface{}, eachFn func(k []byte) error) error
  • func (tx *Transaction) Get(k []byte, v interface{}) error
  • func (tx *Transaction) GetFields(fields string, v interface{}) error
  • func (tx *Transaction) Put(k []byte, v interface{}) error
  • func (tx *Transaction) PutFields(fields string, v interface{}) error
  • func (tx *Transaction) Rollback() error

Examples

• Database.Descend
• Database.Each
• Database.GetFields
• Database.PutFields
• Database.WithNamespace
• Transaction.Each
• Transaction.GetFields
• Transaction.PutFields

Constants

const (
	// Namespace is the prefix of all keys managed by kvpack.
	Namespace = "__kvpack"
	// Separator is the delimiter of all key fields that are inserted by kvpack.
	Separator = "\x00"
)

Variables

var Break = errors.New("break each")

Break is returned from Each's callback when the callback is done and can break gracefully.

var ErrKeyNotFound = errors.New("key not found")

ErrKeyNotFound is returned if the given key is not found. If the key is found, but a child key is not found, then this error is not returned.

var ErrReadOnly = errors.New("transaction is read-only")

ErrReadOnly is returned if the transaction is read-only but a write action is being performed.

var ErrTooRecursed = errors.New("kvpack recursed too deep (> 1024)")

ErrTooRecursed is returned when kvpack functions are being recursed too deeply. When this happens, an error occurs to prevent running out of memory.

var ErrValueNeedsPtr = errors.New("given value must be a non-nil pointer")

ErrValueNeedsPtr is returned if the given value is not a pointer. This is required to handle pointers around in a sane way internally, so it is required of both Get and Put.

Types

type Database

type Database struct {
	driver.Database
	// contains filtered or unexported fields
}

Database describes a database that's managed by kvpack. A database is safe to use concurrently.

func NewDatabase

func NewDatabase(db driver.Database) *Database

NewDatabase creates a new database from an existing database instance. The default namespace is the root namespace; most users should follow this call with .WithNamespace().

func (*Database) Begin

func (db *Database) Begin(readOnly bool) (*Transaction, error)

Begin starts a transaction.

func (*Database) Close

func (db *Database) Close() error

Close closes the database if it implements io.Closer.

func (*Database) Delete

func (db *Database) Delete(prefix []byte) error

Delete deletes the given prefix from the database in a single transaction.

func (*Database) DeleteFields

func (db *Database) DeleteFields(fields string) error

Delete deletes the given dot-syntax prefix from the database in a single transaction.

func (*Database) Descend

func (db *Database) Descend(namespaces ...string) *Database

Descend returns a new database that has been moved into the children namespaces of the previous namespace. For example, if "app-name" is the string passed into WithNamespace, then calling Descend("users") will give "app-name.users" in dot-syntax.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"
	"strings"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	// cleanNamespace will output namespaces similar to regular file paths. It
	// is mostly used for pretty-printing.
	prettyNamespace := func(db *kvpack.Database) string {
		namespace := strings.ReplaceAll(db.Namespace(), kvpack.Separator, "/")
		return strings.TrimPrefix(namespace, kvpack.Namespace)
	}

	db := mustDB("app-name")
	defer db.Close()

	fmt.Println(prettyNamespace(db))

	userDB := db.Descend("users")
	fmt.Println(prettyNamespace(userDB))

}

Output:

/app-name
/app-name/users

func (*Database) Each

func (db *Database) Each(fields string, v interface{}, eachFn func(k []byte) error) error

Each iterates over the dot-syntax fields key from the database in a single transaction. Refer to Transaction's Each for more documentation.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"
	"strconv"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"github.com/pkg/errors"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	type User struct {
		ID    int
		Name  string
		Color string
	}

	db := mustDB("kvpack-demo", "each-example")
	defer db.Close()

	userDB := db.Descend("users")

	users := []User{
		{0, "diamondburned", "pink"},
		{1, "somebody else", "cyan"},
		{2, "anonymous", "red"},
	}

	for _, user := range users {
		if err := userDB.Put([]byte(strconv.Itoa(user.ID)), &user); err != nil {
			log.Fatalln("failed to put user", err)
		}
	}

	// A partial struct can be declared to partially get the data. Here, the
	// color field is dropped.
	var user struct {
		Name string
	}
	var fullUser User
	var found bool

	if err := userDB.Each("", &user, func(key []byte) error {
		if user.Name != "diamondburned" {
			return nil
		}

		// Only get the full user once we've scanned for what we want.
		if err := userDB.Get(key, &fullUser); err != nil {
			return errors.Wrap(err, "failed to get the full user")
		}

		found = true
		return kvpack.Break
	}); err != nil {
		log.Fatalln("failed to iterate:", err)
	}

	if !found {
		log.Fatalln("user not found")
	}

	fmt.Printf(
		"Found user %s (ID %d) who has the color %s",
		fullUser.Name, fullUser.ID, fullUser.Color,
	)

}

Output:

Found user diamondburned (ID 0) who has the color pink

func (*Database) Get

func (db *Database) Get(k []byte, v interface{}) error

Get gets the given key and unmarshals its value into the given pointer in a single read-only transaction.

func (*Database) GetFields

func (db *Database) GetFields(fields string, v interface{}) error

GetFields gets the given dot-syntax key and unmarshals its value into the given pointer in a single read-only transaction. For more information, see Transaction's GetFields.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	type User struct {
		Name  string
		Color string
	}

	db := mustDB("kvpack-demo", "getfields-example")
	defer db.Close()

	users := []User{
		{"diamondburned", "pink"},
		{"somebody else", "cyan"},
		{"anonymous", "red"},
	}

	if err := db.Put([]byte("users"), &users); err != nil {
		log.Fatalln("failed to put user", err)
	}

	// The target can be a string, but it can also be a structure (as in Each's
	// example).
	var color string
	// userDB can be used here with "users." omitted; it is only here as an
	// example.
	if err := db.GetFields("users.0.Color", &color); err != nil {
		log.Fatalln("failed to get user 0:", err)
	}

	fmt.Println("User 0's color is", color)
}

Output:

User 0's color is pink

func (*Database) Namespace

func (db *Database) Namespace() string

Namespace returns the database's namespace, which is the prefix that is always prepended into keys. The returned namespace string is raw, meaning it is not dot-syntax.

func (*Database) Put

func (db *Database) Put(k []byte, v interface{}) error

Put puts the given value into the database with the key in a single transaction.

func (*Database) PutFields

func (db *Database) PutFields(fields string, v interface{}) error

PutFields puts the given value into the database with the given dot-syntax key in a single transaction.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	type User struct {
		Name  string
		Color string
	}

	db := mustDB("kvpack-demo", "getfields-example")
	defer db.Close()

	users := []User{
		{"diamondburned", "pink"},
		{"somebody else", "cyan"},
		{"anonymous", "red"},
	}

	if err := db.Put([]byte("users"), &users); err != nil {
		log.Fatalln("failed to put user", err)
	}

	// Override a user's color.
	pink := "pink"
	if err := db.PutFields("users.2.Color", &pink); err != nil {
		log.Fatalln("failed to change color:", err)
	}

	var thirdUser User
	if err := db.GetFields("users.2", &thirdUser); err != nil {
		log.Fatalln("failed to get 3rd user:", err)
	}

	fmt.Println("User", thirdUser.Name, "now has color", thirdUser.Color)

}

Output:

User anonymous now has color pink

func (*Database) Update

func (db *Database) Update(f func(*Transaction) error) error

Update opens a read-write transaction and runs the given function with that opened transaction, then commits the transaction.

func (*Database) View

func (db *Database) View(f func(*Transaction) error) error

View opens a read-only transaction and runs the given function with that opened transaction, then cleans it up.

func (*Database) WithNamespace

func (db *Database) WithNamespace(namespaces ...string) *Database

WithNamespace creates a new database instance from the existing one with a different namespace. If multiple namespaces are given, then it is treated as nested field keys. Because of this, the meaning of how nested the fields are is entirely up to the user.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"
	"strings"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	// cleanNamespace will output namespaces similar to regular file paths. It
	// is mostly used for pretty-printing.
	prettyNamespace := func(db *kvpack.Database) string {
		namespace := strings.ReplaceAll(db.Namespace(), kvpack.Separator, "/")
		return strings.TrimPrefix(namespace, kvpack.Namespace)
	}

	db := mustDB("app-name")
	defer db.Close()

	fmt.Println(prettyNamespace(db))

	userDB := db.WithNamespace("app-name", "users")
	fmt.Println(prettyNamespace(userDB))

	// Reaccess an upper-level namespace.
	topLevelDB := userDB.WithNamespace("app-name")
	fmt.Println(prettyNamespace(topLevelDB))

}

Output:

/app-name
/app-name/users
/app-name

type Transaction

type Transaction struct {
	Tx driver.Transaction
	// contains filtered or unexported fields
}

Transaction describes a transaction of a database managed by kvpack. A transaction must not be shared across goroutines, as it is not concurrently safe. To work around this, create multiple transactions.

func NewTransaction

func NewTransaction(tx driver.Transaction, fullNamespace []byte, ro bool) *Transaction

NewTransaction creates a new transaction from an existing one. This is useful for working around Database's limited APIs. Users shouldn't call this directly, as this function is primarily used for drivers.

func (*Transaction) Commit

func (tx *Transaction) Commit() error

Commit commits the transaction.

func (*Transaction) Delete

func (tx *Transaction) Delete(k []byte) error

Delete deletes the value with the given key.

func (*Transaction) DeleteFields

func (tx *Transaction) DeleteFields(fields string) error

Delete deletes the value with the given dot-syntax key.

func (*Transaction) Each

func (tx *Transaction) Each(fields string, v interface{}, eachFn func(k []byte) error) error

Each iterates over each instance of the given dot-syntax fields key and calls the eachFn callback on each iteration. If fields is empty, then the current namespace is iterated over.

The callback must capture the pointer passed in, and it must not move or take any of the fields inside the given value until Each exits. The callback must also not take the given key away; it has to copy it into a new slice. If the callback returns kvpack.Break, then a nil error is returned. Otherwise, the error is passed over.

The order of iteration is undefined and unguaranteed by kvpack, however, that is entirely up to the driver and its order of iteration. Refer to the driver's documentation if possible.

Below is an example with error checking omitted for brevity:

tx.PutFields("app.users.1", User{Name: "don't need this user"})
tx.PutFields("app.users.2", User{Name: "don't need this user either"})
tx.PutFields("app.users.3", User{Name: "need this user"})
tx.PutFields("app.users.4", User{Name: "but not this user"})

var user User
return &user, tx.Each("app.users", &user, func(k []byte) bool {
    log.Println("found user with ID", string(k))
    return user.Name == "need this user"
})

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"
	"strconv"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"github.com/pkg/errors"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	type User struct {
		ID    int
		Name  string
		Color string
	}

	db := mustDB("kvpack-demo", "each-example")
	defer db.Close()

	userDB := db.Descend("users")

	if err := userDB.Update(func(tx *kvpack.Transaction) error {
		users := []User{
			{0, "diamondburned", "pink"},
			{1, "somebody else", "cyan"},
			{2, "anonymous", "red"},
		}

		for _, user := range users {
			if err := tx.Put([]byte(strconv.Itoa(user.ID)), &user); err != nil {
				return errors.Wrapf(err, "failed to put user %d", user.ID)
			}
		}
		return nil
	}); err != nil {
		log.Fatalln("failed to update:", err)
	}

	// A partial struct can be declared to partially get the data. Here, the
	// color field is dropped.
	var user struct {
		Name string
	}
	var fullUser User
	var found bool

	if err := userDB.View(func(tx *kvpack.Transaction) error {
		return userDB.Each("", &user, func(key []byte) error {
			if user.Name != "diamondburned" {
				return nil
			}

			// Only get the full user once we've scanned for what we want.
			if err := userDB.Get(key, &fullUser); err != nil {
				return errors.Wrap(err, "failed to get the full user")
			}

			found = true
			return kvpack.Break
		})
	}); err != nil {
		log.Fatalln("failed to iterate:", err)
	}

	if !found {
		log.Fatalln("user not found")
	}

	fmt.Printf(
		"Found user %s (ID %d) who has the color %s",
		fullUser.Name, fullUser.ID, fullUser.Color,
	)

}

Output:

Found user diamondburned (ID 0) who has the color pink

func (*Transaction) Get

func (tx *Transaction) Get(k []byte, v interface{}) error

func (*Transaction) GetFields

func (tx *Transaction) GetFields(fields string, v interface{}) error

GetFields is a convenient function around Get that accesses struct or struct fields using the period syntax. Each field inside the given fields string is delimited by a period, for example, "raining.Cats.Dogs", where "raining" is the key.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	type User struct {
		Name  string
		Color string
	}

	db := mustDB("kvpack-demo", "getfields-example")
	defer db.Close()

	users := []User{
		{"diamondburned", "pink"},
		{"somebody else", "cyan"},
		{"anonymous", "red"},
	}

	if err := db.Update(func(tx *kvpack.Transaction) error {
		return tx.Put([]byte("users"), &users)
	}); err != nil {
		log.Fatalln("failed to put user", err)
	}

	// The target can be a string, but it can also be a structure (as in Each's
	// example).
	var color string
	// userDB can be used here with "users." omitted; it is only here as an
	// example.
	if err := db.View(func(tx *kvpack.Transaction) error {
		return tx.GetFields("users.0.Color", &color)
	}); err != nil {
		log.Fatalln("failed to get user 0:", err)
	}

	fmt.Println("User 0's color is", color)
}

Output:

User 0's color is pink

func (*Transaction) Put

func (tx *Transaction) Put(k []byte, v interface{}) error

Put puts the given value into the database ID'd by the given key. If v's type is a value or a pointer to a byte slice or a string, then a fast path is used, and the values are put into the database as-is.

func (*Transaction) PutFields

func (tx *Transaction) PutFields(fields string, v interface{}) error

PutFields puts the given value into the databse ID'd by the given dot-syntax fields key. This function is similar to GetFields, except it's Put.

Example

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/diamondburned/kvpack"
	"github.com/diamondburned/kvpack/driver/bboltpack"
	"go.etcd.io/bbolt"
)

func mustDB(namespaces ...string) *kvpack.Database {
	path := filepath.Join(os.TempDir(), namespaces[0]+".db")
	opts := bbolt.DefaultOptions

	d, err := bboltpack.Open(path, os.ModePerm, opts)
	if err != nil {
		log.Fatalln("failed to open db:", err)
	}

	return d.WithNamespace(namespaces...)
}

func main() {
	type User struct {
		Name  string
		Color string
	}

	db := mustDB("kvpack-demo", "getfields-example")
	defer db.Close()

	users := []User{
		{"diamondburned", "pink"},
		{"somebody else", "cyan"},
		{"anonymous", "red"},
	}

	if err := db.Update(func(tx *kvpack.Transaction) error {
		return tx.Put([]byte("users"), &users)
	}); err != nil {
		log.Fatalln("failed to put user", err)
	}

	// Override a user's color.
	pink := "pink"

	if err := db.Update(func(tx *kvpack.Transaction) error {
		return tx.PutFields("users.2.Color", &pink)
	}); err != nil {
		log.Fatalln("failed to change color:", err)
	}

	var thirdUser User
	if err := db.GetFields("users.2", &thirdUser); err != nil {
		log.Fatalln("failed to get 3rd user:", err)
	}

	fmt.Println("User", thirdUser.Name, "now has color", thirdUser.Color)

}

Output:

User anonymous now has color pink

func (*Transaction) Rollback

func (tx *Transaction) Rollback() error

Rollback rolls back the transaction. Use of a transaction after rolling back will cause a panic.