zkv

README

zkv

Simple key-value store for single-user applications.

Pros

• Simple two file structure (data file and index file)
• Internal Zstandard compression by klauspost/compress/zstd
• Threadsafe operations through sync.RWMutex

Cons

• Index stored in memory (map[key hash (28 bytes)]file offset (int64))
• No transaction system
• Index file is fully rewrited on every store commit
• No way to recover disk space from deleted records
• Write/Delete operations block Read and each other operations

Usage

Create or open existing file:

db, err := zkv.Open("path to file")

Data operations:

// Write data
err = db.Set(key, value) // key and value can be any of type

// Read data
var value ValueType
err = db.Get(key, &value)

// Delete data
err = db.Delete(key)

Other methods:

// Flush data to disk
err = db.Flush()

// Backup data to another file
err = db.Backup("new/file/path")

Store options

type Options struct {
	// Maximum number of concurrent reads
	MaxParallelReads int

	// Compression level
	CompressionLevel zstd.EncoderLevel

	// Memory write buffer size in bytes
	MemoryBufferSize int

	// Disk write buffer size in bytes
	DiskBufferSize int
}

File structure

Record is encoding/gob structure:

Field	Description	Size
Type	Record type	uint8
KeyHash	Key hash	28 bytes
ValueBytes	Value gob-encoded bytes	variable

File is log stuctured list of commands:

Field	Description	Size
Length	Record body bytes length	int64
Body	Gob-encoded record	variable

Index file is simple gob-encoded map:

map[string]struct {
	BlockOffset  int64
	RecordOffset int64
}

where map key is data key hash and value - data offset in data file.

Resource consumption

Store requirements:

• around 300 Mb of RAM per 1 million of keys
• around 34 Mb of disk space for index file per 1 million of keys

TODO

• Add recovery previous state of store file on write error
• Add method for index rebuild

Documentation

Index

• Variables
• type Offsets
• type Options
• type Record
  • func (r *Record) Marshal() ([]byte, error)
• type RecordType
• type Store
  • func Open(filePath string) (*Store, error)
  • func OpenWithOptions(filePath string, options Options) (*Store, error)
  • func (s *Store) Backup(filePath string) error
  • func (s *Store) BackupWithOptions(filePath string, newFileOptions Options) error
  • func (s *Store) Close() error
  • func (s *Store) Delete(key interface{}) error
  • func (s *Store) Flush() error
  • func (s *Store) Get(key, value interface{}) error
  • func (s *Store) RebuildIndex() error
  • func (s *Store) Set(key, value interface{}) error

Variables

var ErrNotExists = errors.New("not exists")

Types

type Offsets

type Offsets struct {
	BlockOffset  int64
	RecordOffset int64
}

type Options

type Options struct {
	// Maximum number of concurrent reads
	MaxParallelReads int

	// Compression level
	CompressionLevel zstd.EncoderLevel

	// Memory write buffer size in bytes
	MemoryBufferSize int

	// Disk write buffer size in bytes
	DiskBufferSize int
	// contains filtered or unexported fields
}

type Record

type Record struct {
	Type       RecordType
	KeyHash    [28]byte
	ValueBytes []byte
}

func (*Record) Marshal

func (r *Record) Marshal() ([]byte, error)

type RecordType

type RecordType uint8

const (
	RecordTypeSet RecordType = iota + 1
	RecordTypeDelete
)

type Store

type Store struct {
	// contains filtered or unexported fields
}

func Open

func Open(filePath string) (*Store, error)

func OpenWithOptions

func OpenWithOptions(filePath string, options Options) (*Store, error)

func (*Store) Backup

func (s *Store) Backup(filePath string) error

func (*Store) BackupWithOptions

func (s *Store) BackupWithOptions(filePath string, newFileOptions Options) error

func (*Store) Close

func (s *Store) Close() error

func (*Store) Delete

func (s *Store) Delete(key interface{}) error

func (*Store) Flush

func (s *Store) Flush() error

func (*Store) Get

func (s *Store) Get(key, value interface{}) error

func (*Store) RebuildIndex

func (s *Store) RebuildIndex() error

RebuildIndex renews index from store file

func (*Store) Set

func (s *Store) Set(key, value interface{}) error