fsdb

README

FSDB

FSDB is a collection of Go libraries providing a key-value store on your file system.

(Example code on godoc)

Why?

tl;dr: It's for larger (10k+ per entry), less latency sensitive data store.

Most key-value store libraries are optimized for small, in-memory size. Even for on-disk libraries, they are usually not optimized for larger (10k+) values. Also on-disk libraries usually uses write amplify for better performance, which means they will take more disk space than the actual data stored. FSDB store the data as-is or use optional gzip compression, making it a better solution for companies that need to store huge amount of data and is less sensitive to data latency.

FSDB could also be used as the last layer of your layered key-value stores: Use in-memory libraries for samllest, most time sensitive data; Use other on-disk libraries for larger, less time sensitive data; And use FSDB for largest, least latency sensitive data.

Further more, FSDB provided a hybrid implementation, which allows you to put some of your data on a remote bucket (AWS S3, Google Cloud Storage, etc.), providing an exra layer for larger and higher latency data.

It can also be used to implement mocks of remote libraries. For example, package bucket uses local FSDB to implement its mock for testing.

Highlights

FSDB has a minimal overhead, which means its performance is almost identical to the disk I/O performance. The local implementation has no locks. The hybrid implementation only has an optional row lock, please refer to the package documentation for more details.

The local and hybrid implementation has the same interface, which means you can use the local implementation first, and move the the hybrid implementation later as your data grows.

Packages Index

• Package fsdb defines the interface. It does not provide implementations.
• Package local provides the local implementation.
• Package hybrid provides the hybrid implementation.
• Package bucket defines the bucket interface. It does not provide implementations. Implementations for AWS S3 and Google Cloud Storage can be found in external libraries. There's also an implementation based on Go-Cloud Blob interface so you can use any Go-Cloud Blob implementation in hybrid FSDB.

Test

All packages provide its own tests can be run with go test. If you want to test every package within this repository, on Go 1.11+ you can use go test ./.... If you are on an older version of Go, you can use vgo by running vgo test all under the repository root directory.

There are some tests with sleep calls, you can skip them by running go test -short instead. Package local provides a benchmark test can be run with go test -bench ..

License

BSD License.

Documentation

Overview

Package fsdb defines the interfaces of a key-value store on top of file systems.

The interface FSDB defines basic Read, Write and Delete functions.

The interface Local defines extra functions for local implementations.

Example

package main

import (
	"context"
	"strings"

	"github.com/fishy/fsdb"
)

func main() {
	key := fsdb.Key("key")
	ctx := context.Background()
	var db fsdb.Local
	// TODO: open from an implementation

	if err := db.Write(ctx, key, strings.NewReader("content")); err != nil {
		// TODO: handle error
	}

	reader, err := db.Read(ctx, key)
	if err != nil {
		// TODO: handle error
	}
	defer reader.Close()
	// TODO: read from reader

	if err := db.ScanKeys(
		ctx,
		func(key fsdb.Key) bool {
			// TODO: emit the key
			return true // return true to continue the scan
		},
		fsdb.StopAll,
	); err != nil {
		// TODO: handle error
	}

	if err := db.Delete(ctx, key); err != nil {
		// TODO: handle error
	}
}

Index

• func IgnoreAll(path string, err error) bool
• func IsNoSuchKeyError(err error) bool
• func StopAll(path string, err error) bool
• type ErrFunc
• type FSDB
• type Key
  • func (key Key) Equals(other Key) bool
  • func (key Key) String() string
• type KeyFunc
• type Local
• type NoSuchKeyError
  • func (err *NoSuchKeyError) Error() string

Examples

• Package

Functions

func IgnoreAll

func IgnoreAll(path string, err error) bool

IgnoreAll is an ErrFunc that can be used in Local.ScanKeys().

It always returns true, means that the scan ignores all I/O errors it could ignore.

func IsNoSuchKeyError

func IsNoSuchKeyError(err error) bool

IsNoSuchKeyError checks whether a given error is NoSuchKeyError.

func StopAll

func StopAll(path string, err error) bool

StopAll is an ErrFunc that can be used in Local.ScanKeys().

It always returns false, means that the scan stops at the first I/O error it encounters.

Types

type ErrFunc

type ErrFunc func(path string, err error) bool

ErrFunc is used in ScanKeys function in Local interface.

It's the callback function called when the scan encounters an I/O error that is possible to be ignored.

It should return true to ignore the error, or false to abort the scan.

type FSDB

type FSDB interface {
	// Read opens an entry and returns a ReadCloser.
	//
	// If the key does not exist, it should return a NoSuchKeyError.
	//
	// It should never return both nil reader and nil err.
	//
	// It's the caller's responsibility to close the ReadCloser returned.
	Read(ctx context.Context, key Key) (reader io.ReadCloser, err error)

	// Write opens an entry.
	//
	// If the key already exists, it will be overwritten.
	//
	// If data is actually a ReadCloser,
	// it's the caller's responsibility to close it after Write function returns.
	Write(ctx context.Context, key Key, data io.Reader) error

	// Delete deletes an entry.
	//
	// If the key does not exist, it should return a NoSuchKeyError.
	Delete(ctx context.Context, key Key) error
}

FSDB defines the interface for an FSDB implementation.

type Key

type Key []byte

Key is the key type of an FSDB.

func (Key) Equals

func (key Key) Equals(other Key) bool

Equals compares the key to another key.

func (Key) String

func (key Key) String() string

String represents the key.

If the key is valid UTF-8, it will be treated as string. Otherwise it will be treated as []byte.

type KeyFunc

type KeyFunc func(key Key) bool

KeyFunc is used in ScanKeys function in Local interface.

It's the callback function called for every key scanned.

It should return true to continue the scan and false to abort the scan.

It's OK for KeyFunc to block.

type Local

type Local interface {
	FSDB

	// ScanKeys scans all the keys locally.
	//
	// This function would be heavy on IO and takes a long time. Use with caution.
	//
	// The behavior is undefined for keys changed after the scan started,
	// but it should never visit the same key twice in a single scan.
	ScanKeys(ctx context.Context, keyFunc KeyFunc, errFunc ErrFunc) error
}

Local defines extra interface for a local FSDB implementation.

type NoSuchKeyError

type NoSuchKeyError struct {
	Key Key
}

NoSuchKeyError is an error returned by Read and Delete functions when the key requested does not exists.

func (*NoSuchKeyError) Error

func (err *NoSuchKeyError) Error() string