buckets

package module
v0.0.0-...-95fcbf1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 26, 2016 License: MIT Imports: 4 Imported by: 6

README

buckets

GoDoc License Report Card

A simple key/value store based on Bolt.

buckets

In the parlance of key/value stores, a "bucket" is a collection of unique keys that are associated with values. A buckets database is a set of buckets. The underlying datastore is represented by a single file on disk.

Note that buckets is just an extension of Bolt, providing a Bucket type with some nifty convenience methods for operating on the items (key/value pairs) within instances of it. It streamlines simple transactions (a single put, get, or delete) and working with subsets of items within a bucket (via prefix and range scans).

For example, here's how you put an item in a bucket and get it back out. (Note we're omitting proper error handling here.)

// Open a buckets database.
bx, _ := buckets.Open("data.db")
defer bx.Close()

// Create a new `things` bucket.
things, _ := bx.New([]byte("things"))

// Put key/value into the `things` bucket.
key, value := []byte("A"), []byte("alpha")
things.Put(key, value)

// Read value back in a different read-only transaction.
got, _ := things.Get(key)

fmt.Printf("The value of %q in `things` is %q\n", key, got)

Output:

The value of "A" in `things` is "alpha"

Overview

As noted above, buckets is a wrapper for Bolt, streamlining basic transactions. If you're unfamiliar with Bolt, check out the README and intro articles.

A buckets/bolt database contains a set of buckets. What's a bucket? It's basically just an associative array, mapping keys to values. For simplicity, we say that a bucket contains key/values pairs and we refer to these k/v pairs as "items". You use buckets for storing and retrieving such items.

Since Bolt stores keys in byte-sorted order, we can take advantage of this sorted key namespace for fast prefix and range scanning of keys. In particular, it gives us a way to easily retrieve a subset of items. (See the PrefixItems and RangeItems methods, described below.)

Read/write transactions
Read-only transactions

Getting Started

Use go get github.com/joyrexus/buckets to install and see the docs for details.

To open a database, use buckets.Open():

package main

import (
    "log"

    "github.com/joyrexus/buckets"
)

func main() {
    bx, err := buckets.Open("my.db")
    if err != nil {
        log.Fatal(err)
    }
    defer bx.Close()

    ...
}

Note that buckets obtains a file lock on the data file so multiple processes cannot open the same database at the same time.

Examples

The docs contain numerous examples demonstrating basic usage.

See also the examples directory for standalone examples, demonstrating use of buckets for persistence in a web service context.

Documentation

Overview

Package buckets provides a simplified interface to a Bolt database.

A buckets DB is a Bolt database, but it allows you to easily create new bucket instances. The database is represented by a single file on disk. A bucket is a collection of unique keys that are associated with values.

The Bucket type has nifty convenience methods for operating on key/value pairs within it. It streamlines simple transactions (a single put, get, or delete) and working with subsets of items within a bucket (via prefix and range scans). It's not designed to handle more complex or batch transactions. For such cases, use the standard techniques offered by Bolt.

---

What is bolt?

"Bolt implements a low-level key/value store in pure Go. It supports fully serializable transactions, ACID semantics, and lock-free MVCC with multiple readers and a single writer. Bolt can be used for projects that want a simple data store without the need to add large dependencies such as Postgres or MySQL."

See https://github.com/boltdb/bolt for important details.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Bucket 

type Bucket struct {
	Name []byte
	// contains filtered or unexported fields
}

Bucket represents a collection of key/value pairs inside the database.

Example 
package main

import (
	"encoding/binary"
	"fmt"
	"io/ioutil"
	"log"
	"math/rand"
	"net/http"
	"net/http/httptest"
	"os"

	"github.com/joyrexus/buckets"
)

// Set this to see how the counts are actually updated.
const verbose = false

// Counter updates a the hits bucket for every URL path requested.
type counter struct {
	hits *buckets.Bucket
}

// Our handler communicates the new count from a successful database
// transaction.
func (c counter) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
	key := []byte(req.URL.String())

	// Decode handles key not found for us.
	value, _ := c.hits.Get(key)
	count := decode(value) + 1

	if err := c.hits.Put(key, encode(count)); err != nil {
		http.Error(rw, err.Error(), 500)
		return
	}

	if verbose {
		log.Printf("server: %s: %d", req.URL.String(), count)
	}

	// Reply with the new count .
	rw.Header().Set("Content-Type", "application/octet-stream")
	fmt.Fprintf(rw, "%d\n", count)
}

func client(id int, base string, paths []string) error {
	// Process paths in random order.
	rng := rand.New(rand.NewSource(int64(id)))
	permutation := rng.Perm(len(paths))

	for i := range paths {
		path := paths[permutation[i]]
		resp, err := http.Get(base + path)
		if err != nil {
			return err
		}
		defer resp.Body.Close()
		buf, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			return err
		}
		if verbose {
			log.Printf("client: %s: %s", path, buf)
		}
	}
	return nil
}

func main() {
	// Open the database.
	bx, _ := buckets.Open(tempfile())
	defer os.Remove(bx.Path())
	defer bx.Close()

	// Create a hits bucket
	hits, _ := bx.New([]byte("hits"))

	// Start our web server
	count := counter{hits}
	srv := httptest.NewServer(count)
	defer srv.Close()

	// Get every path multiple times.
	paths := []string{
		"/foo",
		"/bar",
		"/baz",
		"/quux",
		"/thud",
		"/xyzzy",
	}
	for id := 0; id < 10; id++ {
		if err := client(id, srv.URL, paths); err != nil {
			fmt.Printf("client error: %v", err)
		}
	}

	// Check the final result
	do := func(k, v []byte) error {
		fmt.Printf("hits to %s: %d\n", k, decode(v))
		return nil
	}
	hits.Map(do)
	// outputs ...
	// hits to /bar: 10
	// hits to /baz: 10
	// hits to /foo: 10
	// hits to /quux: 10
	// hits to /thud: 10
	// hits to /xyzzy: 10

}

// encode marshals a counter.
func encode(n uint64) []byte {
	buf := make([]byte, 8)
	binary.BigEndian.PutUint64(buf, n)
	return buf
}

// decode unmarshals a counter. Nil buffers are decoded as 0.
func decode(buf []byte) uint64 {
	if buf == nil {
		return 0
	}
	return binary.BigEndian.Uint64(buf)
}

Output:

hits to /bar: 10
hits to /baz: 10
hits to /foo: 10
hits to /quux: 10
hits to /thud: 10
hits to /xyzzy: 10

func (*Bucket) Delete 

func (bk *Bucket) Delete(k []byte) error

Delete removes key `k`.

func (*Bucket) Get 

func (bk *Bucket) Get(k []byte) (value []byte, err error)

Get retrieves the value for key `k`.

func (*Bucket) Insert 

func (bk *Bucket) Insert(items []struct{ Key, Value []byte }) error

Insert iterates over a slice of k/v pairs, putting each item in the bucket as part of a single transaction. For large insertions, be sure to pre-sort your items (by Key in byte-sorted order), which will result in much more efficient insertion times and storage costs.

Example

Show we can insert items into a bucket and get them back out. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

letters, _ := bx.New([]byte("letters"))

// Setup items to insert in `letters` bucket.
items := []struct {
	Key, Value []byte
}{
	{[]byte("A"), []byte("alpha")},
	{[]byte("B"), []byte("beta")},
	{[]byte("C"), []byte("gamma")},
}

// Insert items into `letters` bucket.
if err := letters.Insert(items); err != nil {
	fmt.Println("could not insert items!")
}

// Get items back out in separate read-only transaction.
results, _ := letters.Items()

for _, item := range results {
	fmt.Printf("%s -> %s\n", item.Key, item.Value)
}

Output:

A -> alpha
B -> beta
C -> gamma

func (*Bucket) InsertNX 

func (bk *Bucket) InsertNX(items []struct{ Key, Value []byte }) error

InsertNX (insert-if-not-exists) iterates over a slice of k/v pairs, putting each item in the bucket as part of a single transaction. Unlike Insert, however, InsertNX will not update the value for an existing key.

func (*Bucket) Items 

func (bk *Bucket) Items() (items []Item, err error)

Items returns a slice of key/value pairs. Each k/v pair in the slice is of type Item (`struct{ Key, Value []byte }`).

func (*Bucket) Map 

func (bk *Bucket) Map(do func(k, v []byte) error) error

Map applies `do` on each key/value pair.

func (*Bucket) MapPrefix 

func (bk *Bucket) MapPrefix(do func(k, v []byte) error, pre []byte) error

MapPrefix applies `do` on each k/v pair of keys with prefix.

Example

Show that we can apply a function to the k/v pairs of keys with a given prefix. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

// Create a new things bucket.
things, _ := bx.New([]byte("things"))

// Setup items to insert.
items := []struct {
	Key, Value []byte
}{
	{[]byte("A"), []byte("1")},   // `A` prefix match
	{[]byte("AA"), []byte("2")},  // match
	{[]byte("AAA"), []byte("3")}, // match
	{[]byte("AAB"), []byte("2")}, // match
	{[]byte("B"), []byte("O")},
	{[]byte("BA"), []byte("0")},
	{[]byte("BAA"), []byte("0")},
}

// Insert 'em.
if err := things.Insert(items); err != nil {
	fmt.Printf("could not insert items in `things` bucket: %v\n", err)
}

// Now collect each item whose key starts with "A".
prefix := []byte("A")

// Setup slice of items.
type item struct {
	Key, Value []byte
}
results := []item{}

// Anon func to map over matched keys.
do := func(k, v []byte) error {
	results = append(results, item{k, v})
	return nil
}

if err := things.MapPrefix(do, prefix); err != nil {
	fmt.Printf("could not map items with prefix %s: %v\n", prefix, err)
}

for _, item := range results {
	fmt.Printf("%s -> %s\n", item.Key, item.Value)
}

Output:

A -> 1
AA -> 2
AAA -> 3
AAB -> 2

func (*Bucket) MapRange 

func (bk *Bucket) MapRange(do func(k, v []byte) error, min, max []byte) error

MapRange applies `do` on each k/v pair of keys within range.

Example

Show that we can apply a function to the k/v pairs of keys within a given range. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

// Delete any existing bucket named "years".
bx.Delete([]byte("years"))

// Create a new bucket named "years".
years, _ := bx.New([]byte("years"))

// Setup items to insert in `years` bucket
items := []struct {
	Key, Value []byte
}{
	{[]byte("1970"), []byte("70")},
	{[]byte("1975"), []byte("75")},
	{[]byte("1980"), []byte("80")},
	{[]byte("1985"), []byte("85")},
	{[]byte("1990"), []byte("90")}, // min = 1990
	{[]byte("1995"), []byte("95")}, // min < 1995 < max
	{[]byte("2000"), []byte("00")}, // max = 2000
	{[]byte("2005"), []byte("05")},
	{[]byte("2010"), []byte("10")},
}

// Insert 'em.
if err := years.Insert(items); err != nil {
	fmt.Printf("could not insert items in `years` bucket: %v\n", err)
}

// Time range to map over: 1990 <= key <= 2000.
min := []byte("1990")
max := []byte("2000")

// Setup slice of items to collect results.
type item struct {
	Key, Value []byte
}
results := []item{}

// Anon func to map over matched keys.
do := func(k, v []byte) error {
	results = append(results, item{k, v})
	return nil
}

if err := years.MapRange(do, min, max); err != nil {
	fmt.Printf("could not map items within range: %v\n", err)
}

for _, item := range results {
	fmt.Printf("%s -> %s\n", item.Key, item.Value)
}

Output:

1990 -> 90
1995 -> 95
2000 -> 00

func (*Bucket) NewPrefixScanner 

func (bk *Bucket) NewPrefixScanner(pre []byte) *PrefixScanner

NewPrefixScanner initializes a new prefix scanner.

func (*Bucket) NewRangeScanner 

func (bk *Bucket) NewRangeScanner(min, max []byte) *RangeScanner

NewRangeScanner initializes a new range scanner. It takes a `min` and a `max` key for specifying the range paramaters.

func (*Bucket) PrefixItems 

func (bk *Bucket) PrefixItems(pre []byte) (items []Item, err error)

PrefixItems returns a slice of key/value pairs for all keys with a given prefix. Each k/v pair in the slice is of type Item (`struct{ Key, Value []byte }`).

Example

Show that we can get items for all keys with a given prefix. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

// Create a new things bucket.
things, _ := bx.New([]byte("things"))

// Setup items to insert.
items := []struct {
	Key, Value []byte
}{
	{[]byte("A"), []byte("1")},   // `A` prefix match
	{[]byte("AA"), []byte("2")},  // match
	{[]byte("AAA"), []byte("3")}, // match
	{[]byte("AAB"), []byte("2")}, // match
	{[]byte("B"), []byte("O")},
	{[]byte("BA"), []byte("0")},
	{[]byte("BAA"), []byte("0")},
}

// Insert 'em.
if err := things.Insert(items); err != nil {
	fmt.Printf("could not insert items in `things` bucket: %v\n", err)
}

// Now get items whose key starts with "A".
prefix := []byte("A")

results, err := things.PrefixItems(prefix)
if err != nil {
	fmt.Printf("could not get items with prefix %q: %v\n", prefix, err)
}

for _, item := range results {
	fmt.Printf("%s -> %s\n", item.Key, item.Value)
}

Output:

A -> 1
AA -> 2
AAA -> 3
AAB -> 2

func (*Bucket) Put 

func (bk *Bucket) Put(k, v []byte) error

Put inserts value `v` with key `k`.

Example

Show we can put an item in a bucket and get it back out. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

// Create a new `things` bucket.
bucket := []byte("things")
things, _ := bx.New(bucket)

// Put key/value into the `things` bucket.
key, value := []byte("A"), []byte("alpha")
if err := things.Put(key, value); err != nil {
	fmt.Printf("could not insert item: %v", err)
}

// Read value back in a different read-only transaction.
got, _ := things.Get(key)

fmt.Printf("The value of %q in `%s` is %q\n", key, bucket, got)

Output:

The value of "A" in `things` is "alpha"

func (*Bucket) PutNX 

func (bk *Bucket) PutNX(k, v []byte) error

PutNX (put-if-not-exists) inserts value `v` with key `k` if key doesn't exist.

Example

Show we don't overwrite existing values when using PutNX. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

// Create a new `things` bucket.
bucket := []byte("things")
things, _ := bx.New(bucket)

// Put key/value into the `things` bucket.
key, value := []byte("A"), []byte("alpha")
if err := things.Put(key, value); err != nil {
	fmt.Printf("could not insert item: %v", err)
}

// Read value back in a different read-only transaction.
got, _ := things.Get(key)

fmt.Printf("The value of %q in `%s` is %q\n", key, bucket, got)

// Try putting another value with same key.
things.PutNX(key, []byte("beta"))

// Read value back in a different read-only transaction.
got, _ = things.Get(key)

fmt.Printf("The value of %q in `%s` is still %q\n", key, bucket, got)

Output:

The value of "A" in `things` is "alpha"
The value of "A" in `things` is still "alpha"

func (*Bucket) RangeItems 

func (bk *Bucket) RangeItems(min []byte, max []byte) (items []Item, err error)

RangeItems returns a slice of key/value pairs for all keys within a given range. Each k/v pair in the slice is of type Item (`struct{ Key, Value []byte }`).

Example

Show that we get items for keys within a given range. 

bx, _ := buckets.Open(tempfile())
defer os.Remove(bx.Path())
defer bx.Close()

// Create a new bucket named "years".
years, _ := bx.New([]byte("years"))

// Setup items to insert in `years` bucket
items := []struct {
	Key, Value []byte
}{
	{[]byte("1970"), []byte("70")},
	{[]byte("1975"), []byte("75")},
	{[]byte("1980"), []byte("80")},
	{[]byte("1985"), []byte("85")},
	{[]byte("1990"), []byte("90")}, // min = 1990
	{[]byte("1995"), []byte("95")}, // min < 1995 < max
	{[]byte("2000"), []byte("00")}, // max = 2000
	{[]byte("2005"), []byte("05")},
	{[]byte("2010"), []byte("10")},
}

// Insert 'em.
if err := years.Insert(items); err != nil {
	fmt.Printf("could not insert items in `years` bucket: %v\n", err)
}

// Time range: 1990 <= key <= 2000.
min := []byte("1990")
max := []byte("2000")

results, err := years.RangeItems(min, max)
if err != nil {
	fmt.Printf("could not get items within range: %v\n", err)
}

for _, item := range results {
	fmt.Printf("%s -> %s\n", item.Key, item.Value)
}

Output:

1990 -> 90
1995 -> 95
2000 -> 00

type DB 

type DB struct {
	*bolt.DB
}

A DB is a bolt database with convenience methods for working with buckets.

A DB embeds the exposed bolt.DB methods.

func Open 

func Open(path string) (*DB, error)

Open creates/opens a buckets database at the specified path.

func (*DB) Delete 

func (db *DB) Delete(name []byte) error

Delete removes the named bucket.

func (*DB) New 

func (db *DB) New(name []byte) (*Bucket, error)

New creates/opens a named bucket.

type Item 

type Item struct {
	Key   []byte
	Value []byte
}

An Item holds a key/value pair.

type PrefixScanner 

type PrefixScanner struct {
	BucketName []byte
	Prefix     []byte
	// contains filtered or unexported fields
}

A PrefixScanner scans a bucket for keys with a given prefix.

func (*PrefixScanner) Count 

func (ps *PrefixScanner) Count() (count int, err error)

Count returns a count of the keys with prefix.

func (*PrefixScanner) ItemMapping 

func (ps *PrefixScanner) ItemMapping() (map[string][]byte, error)

ItemMapping returns a map of key/value pairs for keys with prefix. This only works with buckets whose keys are byte-sliced strings.

func (*PrefixScanner) Items 

func (ps *PrefixScanner) Items() (items []Item, err error)

Items returns a slice of key/value pairs for keys with prefix.

func (*PrefixScanner) Keys 

func (ps *PrefixScanner) Keys() (keys [][]byte, err error)

Keys returns a slice of keys with prefix.

func (*PrefixScanner) Map 

func (ps *PrefixScanner) Map(do func(k, v []byte) error) error

Map applies `do` on each key/value pair for keys with prefix.

func (*PrefixScanner) Values 

func (ps *PrefixScanner) Values() (values [][]byte, err error)

Values returns a slice of values for keys with prefix.

type RangeScanner 

type RangeScanner struct {
	BucketName []byte
	Min        []byte
	Max        []byte
	// contains filtered or unexported fields
}

A RangeScanner scans a bucket for keys within a given range.

func (*RangeScanner) Count 

func (rs *RangeScanner) Count() (count int, err error)

Count returns a count of the keys within the range.

func (*RangeScanner) ItemMapping 

func (rs *RangeScanner) ItemMapping() (map[string][]byte, error)

ItemMapping returns a map of key/value pairs for keys within the range. This only works with buckets whose keys are byte-sliced strings.

func (*RangeScanner) Items 

func (rs *RangeScanner) Items() (items []Item, err error)

Items returns a slice of key/value pairs for keys within the range. Note that the returned slice contains elements of type Item.

func (*RangeScanner) Keys 

func (rs *RangeScanner) Keys() (keys [][]byte, err error)

Keys returns a slice of keys within the range.

func (*RangeScanner) Map 

func (rs *RangeScanner) Map(do func(k, v []byte) error) error

Map applies `do` on each key/value pair for keys within range.

func (*RangeScanner) Values 

func (rs *RangeScanner) Values() (values [][]byte, err error)

Values returns a slice of values for keys within the range.

type Scanner 

type Scanner interface {
	// Map applies a func on each key/value pair scanned.
	Map(func(k, v []byte) error) error
	// Count returns a count of the scanned keys.
	Count() (int, error)
	// Keys returns a slice of the scanned keys.
	Keys() ([][]byte, error)
	// Values returns a slice of values from scanned keys.
	Values() ([][]byte, error)
	// Items returns a slice of k/v pairs from scanned keys.
	Items() ([]Item, error)
	// ItemMapping returns a mapping of k/v pairs from scanned keys.
	ItemMapping() (map[string][]byte, error)
}

A Scanner implements methods for scanning a subset of keys in a bucket and retrieving data from or about those keys.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.