fastdb

README

FastDB

FastDB is an (persistent) in-memory key/value store in Go.

Motivation

I wanted to be able to access my data as fast as possible without it being persisted to disk.
In the past I've used several other key-value solutions but as a challenge to myself, tried if I could make
something that was faster than the fastest I've used.

It is also important to know that memory access will always be faster than disk access,
but it goes without saying that memory is more limited than disk storage.

Semi-technical information

This is in fact a key-value store, working with buckets.
(Buckets are a kind of a "box" in which you store key-values of the same kind.)

The "trick" behind this, is that the real key is a combination of bucket_key when
the data is persisted to disk.

When you open the database, you can set the timer (in milliseconds) which will be the
trigger to persist to disk. A value of 100 should be okay.
That means there is a tiny risk that data from within the last 100 milliseconds isn't
written to disk when there is a power failure.

If you want to minimize that risk, use a sync-time of 0.
(but this will be slower!)

How it works

Set

The way to store things:

	err = store.Set(bucket, key, value)

bucket - string
key - int
value - []byte

So it is ideal for storing JSON, for example:

	record := &someRecord{
		ID:   1,
		UUID: "UUIDtext",
		Text: "a text",
	}

	recordData, _ := json.Marshal(record)

	err = store.Set("texts", record.ID, recordData)

Get

The way to retrieve 1 record:

	value, ok := store.Get(bucket, key)

bucket - string
key - int
value - []byte

GetAll

The way to retrieve all the data from one bucket:

	records, ok := store.GetAll(bucket)

bucket - string
key - int
records - map[int][]byte

Info

To get information about the storage:

	info := store.Info()

Will show the number of buckets and the total of records.

Del

The way to delete 1 record:

	 ok, err := store.Del(bucket, key)

bucket - string
key - int
ok - bool (true: key was found and deleted)

Defrag

If overtime there are many deletions, the database could be compressed,
so the next time you'll open and read the file, it will be faster.

	err := store.Defrag()

if there's an error, the original file will exist as a.bak file.

Some simple figures

Done on my Macbook Pro M1.

created 100000 records in 62.726042ms
read 100000 records in 250ns
sort 100000 records by key in 23.126917ms
sort 100000 records by UUID in 41.55275ms

Benchmarks

goos: darwin
goarch: arm64
pkg: codeberg.org/marcelloh/fastdb
Benchmark_Get_Memory_1000
Benchmark_Get_Memory_1000-8         	51353016	        23.30 ns/op	       0 B/op	       0 allocs/op
Benchmark_Set_File_NoSyncTime
Benchmark_Set_File_NoSyncTime-8     	     157	      7094052 ns/op	     265 B/op	       3 allocs/op
Benchmark_Set_Memory
Benchmark_Set_Memory-8              	 3586731	        297.6 ns/op	      93 B/op	       1 allocs/op
Benchmark_Set_File_WithSyncTime
Benchmark_Set_File_WithSyncTime-8   	  468963	         2407 ns/op	     209 B/op	       3 allocs/op
Benchmark_Get_File_1000
Benchmark_Get_File_1000-8           	44613194	        26.18 ns/op	       0 B/op	       0 allocs/op

Example(s)

In the examples directory, you will find an example on how to sort the data.

If more examples are needed, I will write them.

Documentation

Overview

Package fastdb holds all the fastdb functionalities.

Index

• type DB
  • func Open(path string, syncTime int) (*DB, error)
  • func (fdb *DB) BucketInfo(bucket string) int
  • func (fdb *DB) Close() error
  • func (fdb *DB) Defrag() error
  • func (fdb *DB) Del(bucket string, key int) (bool, error)
  • func (fdb *DB) Get(bucket string, key int) ([]byte, bool)
  • func (fdb *DB) GetAll(bucket string) (map[int][]byte, error)
  • func (fdb *DB) GetAllSorted(bucket string) ([]SortRecord, error)
  • func (fdb *DB) GetNewIndex(bucket string) (int, error)
  • func (fdb *DB) Info() string
  • func (fdb *DB) Set(bucket string, key int, value []byte) error
• type SortRecord

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB represents a collection of key-value pairs that persist on disk or memory.

func Open

func Open(path string, syncTime int) (*DB, error)

Open opens a database at the provided path. If the file doesn't exist, it will be created automatically. If the path is ':memory:' then the database will be opened in memory only.

func (*DB) BucketInfo

func (fdb *DB) BucketInfo(bucket string) int

BucketInfo returns number of records in a bucket.

func (*DB) Close

func (fdb *DB) Close() error

Close closes the database.

func (*DB) Defrag

func (fdb *DB) Defrag() error

Defrag optimises the file to reflect the latest state.

func (*DB) Del

func (fdb *DB) Del(bucket string, key int) (bool, error)

Del deletes one map value in a bucket.

func (*DB) Get

func (fdb *DB) Get(bucket string, key int) ([]byte, bool)

Get returns one map value from a bucket.

func (*DB) GetAll

func (fdb *DB) GetAll(bucket string) (map[int][]byte, error)

GetAll returns all map values from a bucket in random order.

func (*DB) GetAllSorted

func (fdb *DB) GetAllSorted(bucket string) ([]SortRecord, error)

GetAllSorted returns all map values from a bucket in Key sorted order.

func (*DB) GetNewIndex

func (fdb *DB) GetNewIndex(bucket string) (int, error)

GetNewIndex returns the next available index for a bucket.

func (*DB) Info

func (fdb *DB) Info() string

Info returns info about the storage.

func (*DB) Set

func (fdb *DB) Set(bucket string, key int, value []byte) error

Set stores one map value in a bucket.

type SortRecord

type SortRecord struct {
	SortField any
	Data      []byte
}

SortRecord represents a record from a sorted collection of sliced records