kvcache

README

kvcache

Simple, optimized, embedded, persistent (file-based) key-value cache.

Intended use case

kvcache is intended for caching data

• that can be reproduced at any time
• but reproduction is time or resource consuming
• not feasible to store all in memory
• desirable to persist (remains between application restarts)

Features

• Very simple interface. Basically just a Get and a Put operation. Does not support changing or removing elements, but it supports removing all elements with the Clear method.
• Optimized. Keys are kept in memory for fast lookups.
• Embedded. You init / create your cache from within your app. No external services need to run.
• Persistent. Data (key-value pairs) are written to files in a folder given at creation time.
• Simple data versioning (global to the cache). If version supplied at create / init time differs, cache will be auto-cleared.
• Supports concurrent access.

Notes

Since element removal and changing is not supported, the usability of this cache implementation is limited, but in exchange it provides fast access and very compact storage: basically the required storage size equals to the total size of values and total size of keys (plus a tiny overhead per key).

Implementation restrictions

• Length of the data version and (individual) keys must be less than 64 KB (1<<16 - 1), exposed as KeySizeLimit.
• Total data size (total size of values) must not exceed 4 GB (1<<32 - 1), exposed as DataSizeLimit.

Documentation

Overview

Package kvcache is a simple, optimized, embedded, persistent (file-based) key-value cache.

Intended use case

kvcache is intended for caching data

- that can be reproduced at any time

- but reproduction is time or resource consuming

- not feasible to store all in memory

- desirable to persist (remains between application restarts)

Features

- Very simple interface. Basically just a Get and a Put operation. Does not support changing or removing elements, but it supports removing all elements with the Clear method.

- Optimized. Keys are kept in memory for fast lookups.

- Embedded. You init / create your cache from within your app. No external services need to run.

- Persistent. Data (key-value pairs) are written to files in a folder given at creation time.

- Simple data versioning (global to the cache). If version supplied at create / init time differs, cache will be auto-cleared.

- Supports concurrent access.

Notes

Since element removal and changing is not supported, the usability of this cache implementation is limited, but in exchange it provides fast access and very compact storage: basically the required storage size equals to the total size of values and total size of keys (plus a tiny overhead per key).

Implementation restrictions

- Length of the data version and keys must be less than 64 KB (1<<16 - 1), exposed as KeySizeLimit.

- Total data size (total size of values) must not exceed 4 GB (1<<32 - 1), exposed as DataSizeLimit.

Index

• Constants
• Variables
• type Cache
  • func New(folder, version string) (cch Cache, err error)
• type Stat

Constants

const (
	// KeySizeLimit is the max allowed length for (individual) keys
	// and the version string
	KeySizeLimit = 1<<16 - 1 // 64 KB

	// DataSizeLimit is the max allowed total data size
	DataSizeLimit = 1<<32 - 1 // 4 GB
)

Variables

var (
	// ErrKeyExists is returned when attempting to put an existing key into the cache
	ErrKeyExists = errors.New("key already in cache")

	// ErrKeySize is returned when attempting to put a too long key into the cache
	ErrKeySize = errors.New("key too long")

	// ErrDataSize is returned when attempting to put a value into the cache
	// which would raise total data size over the limit (DataSizeLimit)
	ErrDataSize = errors.New("total data too big")
)

Types

type Cache

type Cache interface {
	// Get returns the value associated with the given key.
	// nil slice and nil error is returned if the key is not found.
	Get(key string) ([]byte, error)

	// Put puts a new key-value pair into the cache.
	//
	// ErrKeyExists is returned if key is already in the case.
	// ErrKeySize is returned if key is too long (> KeySizeLimit).
	// ErrDataSize is returned if putting the value would increase total
	// data size over the limit (DataSizeLimit).
	Put(key string, value []byte) error

	// Clear removes all key-value pairs from the cache.
	Clear() error

	// Folder returns the folder of the cache where data is persisted.
	Folder() string

	// Len returns the number of key-value pairs in the Cache.
	Len() int

	// Stat returns some basic statistics about the cache,
	// including the occupied storage size of the cache in bytes.
	Stat() (*Stat, error)

	// Close closes the cache, releases any associated resources.
	Close() error
}

Cache describes the operations of the key-value cache.

func New

func New(folder, version string) (cch Cache, err error)

New creates a new Cache, using the given folder for persisting the data. You also have to provide the version of the data. If a persisted cache already exists, and its version is different from this, it will be cleared before return.

ErrKeySize is returned if version is too long (>KeySizeLimit).

type Stat

type Stat struct {
	// Len is the number of key-value pairs in the Cache (same as Cache.Len()).
	Len int

	// StorageSize is the total storage size of the cache in bytes,
	// it is the sum of IndexSize and DataSize.
	StorageSize int64

	// IndexSize is the total size of the keys (plus the version plus
	// 8 bytes of metadata per key) in bytes.
	IndexSize int64

	// DataSize is the total size of the values in bytes.
	DataSize int64
}

Stat wraps basic statistics about the cache.