go-scdb

README

go-scdb

A very simple and fast key-value pure-go store but persisting data to disk, with a "localStorage-like" API.

This is the pure-golang version of the original scdb

scdb may not be production-ready yet. It works, quite well but it requires more vigorous testing.

Purpose

Coming from front-end web development, localStorage was always a convenient way of quickly persisting data to be used later by a given application even after a restart. Its API was extremely simple i.e. localStorage.getItem(), localStorage.setItem(), localStorage.removeItem() , localStorage.clear().

Coming to the backend (or even desktop) development, such an embedded persistent data store with a simple API was hard to come by.

scdb is meant to be like the 'localStorage' of backend and desktop (and possibly mobile) systems. Of course to make it a little more appealing, it has some extra features like:

• Time-to-live (TTL) where a key-value pair expires after a given time
• Non-blocking reads from separate processes, and threads.
• Fast Sequential writes to the store, queueing any writes from multiple processes and threads.

Dependencies

• golang +v1.18

Quick Start

• Ensure you have golang +v1.18 installed. You can check the official instructions for how to do that.

• Initialize a new go modules project

mkdir example-go-scdb
cd example-go-scdb
go mod init github.com/<your-username>/example-go-scdb

• Install the package

go get github.com/sopherapps/go-scdb/scdb

• Create a main.go file

touch main.go

• Add the following code to the main.go file

package main

import (
	"fmt"
	"github.com/sopherapps/go-scdb/scdb"
	"log"
)

func main() {
	records := map[string][]byte{
		"hey":      []byte("English"),
		"hi":       []byte("English"),
		"salut":    []byte("French"),
		"bonjour":  []byte("French"),
		"hola":     []byte("Spanish"),
		"oi":       []byte("Portuguese"),
		"mulimuta": []byte("Runyoro"),
	}

	var maxKeys uint64 = 1_000_000
	var redundantBlocks uint16 = 1
	var poolCapacity uint64 = 10
	var compactionInterval uint32 = 1_800

	store, err := scdb.New(
		"db",
		&maxKeys,
		&redundantBlocks,
		&poolCapacity,
		&compactionInterval)
	if err != nil {
		log.Fatalf("error opening store: %s", err)
	}
	defer func() {
		_ = store.Close()
    }()

	// inserting without ttl
	for k, v := range records {
		err := store.Set([]byte(k), v, nil)
		if err != nil {
			log.Fatalf("error inserting without ttl: %s", err)
		}
	}

	// inserting with ttl of 5 seconds
	var ttl uint64 = 5
	for k, v := range records {
		err := store.Set([]byte(k), v, &ttl)
		if err != nil {
			log.Fatalf("error inserting with ttl: %s", err)
		}
	}

	// updating - just set them again
	updates := map[string][]byte{
		"hey":      []byte("Jane"),
		"hi":       []byte("John"),
		"hola":     []byte("Santos"),
		"oi":       []byte("Ronaldo"),
		"mulimuta": []byte("Aliguma"),
	}
	for k, v := range updates {
		err := store.Set([]byte(k), v, nil)
		if err != nil {
			log.Fatalf("error updating: %s", err)
		}
	}

	// getting
	for k := range records {
		value, err := store.Get([]byte(k))
		if err != nil {
			log.Fatalf("error getting: %s", err)
		}

		fmt.Printf("Key: %s, Value: %s", k, value)
	}

	// deleting
	for k := range records {
		err := store.Delete([]byte(k))
		if err != nil {
			log.Fatalf("error deleting: %s", err)
		}
	}

	// clearing
	err = store.Clear()
	if err != nil {
		log.Fatalf("error clearing: %s", err)
	}

	// compacting (Use sparingly, say if database file is too big)
	err = store.Compact()
	if err != nil {
		log.Fatalf("error compacting: %s", err)
	}
}

• Run the module

go run main.go 

Contributing

Contributions are welcome. The docs have to maintained, the code has to be made cleaner, more idiomatic and faster, and there might be need for someone else to take over this repo in case I move on to other things. It happens!

Please look at the CONTRIBUTIONS GUIDELINES

You can also look in the ./docs folder of the rust scdb to get up to speed with the internals of scdb e.g.

• database file format
• how it works

Bindings

scdb is meant to be used in multiple languages of choice. However, the bindings for most of them are yet to be developed.

For other programming languages, see the main README

How to Test

• Ensure you have golang +v1.18 installed. You can check the official instructions for how to do that.
• Clone this repo and enter its root folder

git clone https://github.com/sopherapps/go-scdb.git && cd go-scdb

• Install dependencies

go mod tidy

• Run the tests command

go test ./... -timeout 30s -race

• Run benchmarks

go test -bench=. ./scdb -run=^#

Benchmarks

On a average PC

cpu: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
BenchmarkStore_Clear/Clear-8               37342             31261 ns/op
BenchmarkStore_Clear/Clear_with_ttl:_3600-8                37086             31756 ns/op
BenchmarkStore_Compact/Compact-8                               1        2070248915 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_hey-8          284754              5521 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_hi-8           185820              6594 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_salut-8        177535              6735 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_bonjour-8              179936              6525 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_hola-8                 177618              6735 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_oi-8                   173463              6704 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_mulimuta-8             175621              6668 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_hey-8                     280477              4010 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_hi-8                      307460              6632 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_salut-8                   194413              6775 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_bonjour-8                 180406              6496 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_hola-8                    184796              6556 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_oi-8                      178884              6620 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_mulimuta-8                185463              6597 ns/op
BenchmarkStore_GetWithoutTtl/Get_hey-8                           1000000              1024 ns/op
BenchmarkStore_GetWithoutTtl/Get_hi-8                            1000000              1031 ns/op
BenchmarkStore_GetWithoutTtl/Get_salut-8                         1000000              1032 ns/op
BenchmarkStore_GetWithoutTtl/Get_bonjour-8                        996567              1030 ns/op
BenchmarkStore_GetWithoutTtl/Get_hola-8                          1131050              1031 ns/op
BenchmarkStore_GetWithoutTtl/Get_oi-8                            1000000              1030 ns/op
BenchmarkStore_GetWithoutTtl/Get_mulimuta-8                      1000000              1023 ns/op
BenchmarkStore_GetWithTtl/Get_hey-8                               989518              1171 ns/op
BenchmarkStore_GetWithTtl/Get_hi-8                                990582              1158 ns/op
BenchmarkStore_GetWithTtl/Get_salut-8                             961626              1167 ns/op
BenchmarkStore_GetWithTtl/Get_bonjour-8                           996882              1168 ns/op
BenchmarkStore_GetWithTtl/Get_hola-8                              988996              1162 ns/op
BenchmarkStore_GetWithTtl/Get_oi-8                                989503              1165 ns/op
BenchmarkStore_GetWithTtl/Get_mulimuta-8                         1000000              1155 ns/op
BenchmarkStore_SetWithoutTtl/Set_hey_English-8                    164676              8012 ns/op
BenchmarkStore_SetWithoutTtl/Set_hi_English-8                     112162              9356 ns/op
BenchmarkStore_SetWithoutTtl/Set_salut_French-8                   127788              9220 ns/op
BenchmarkStore_SetWithoutTtl/Set_bonjour_French-8                 129494              9266 ns/op
BenchmarkStore_SetWithoutTtl/Set_hola_Spanish-8                   120920              9238 ns/op
BenchmarkStore_SetWithoutTtl/Set_oi_Portuguese-8                  123501              9433 ns/op
BenchmarkStore_SetWithoutTtl/Set_mulimuta_Runyoro-8               126140              9325 ns/op
BenchmarkStore_SetWithTtl/Set_hey_English-8                       168462              7154 ns/op
BenchmarkStore_SetWithTtl/Set_hi_English-8                        126549              9474 ns/op
BenchmarkStore_SetWithTtl/Set_salut_French-8                      123495              9541 ns/op
BenchmarkStore_SetWithTtl/Set_bonjour_French-8                    118657              9908 ns/op
BenchmarkStore_SetWithTtl/Set_hola_Spanish-8                      120117              9700 ns/op
BenchmarkStore_SetWithTtl/Set_oi_Portuguese-8                     120649              9792 ns/op
BenchmarkStore_SetWithTtl/Set_mulimuta_Runyoro-8                  119839              9718 ns/op
PASS
ok      github.com/sopherapps/go-scdb/scdb      62.450s

TODO

• Optimize Get operation
• Optimize the Compact operation

Acknowledgements

• The GopherAcademy Article on avoiding GC overhead with large heaps was helpful in the validation of the memory representation of buffers as byte arrays.

License

Licensed under both the MIT License

Copyright (c) 2022 Martin Ahindura

Gratitude

"This is real love - not that we loved God, but that He loved us and sent His Son as a sacrifice to take away our sins."

-- 1 John 4: 10

All glory be to God.