slowpoke

package module
v2.0.1+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 24, 2018 License: MIT Imports: 5 Imported by: 8

README

Build Status Go Report Card Documentation

Description

Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), values stored on disk.

Description on russian: https://habr.com/post/354224/

slowpoke

Motivation

Replace Bolt with a simpler and more efficient engine.

Slowpoke (from version 2.0) based on pudge

How it works

Keys are stored in memory with persistence to disk. Values stored on disk only.

Slowpoke is parallel

Server

GRPC Server example: okdb

Complex examples

typegram

zen platform for authors and their subscribers with a minimalistic design and user-friendly interface.

golang-gin-realworld-example-app

This codebase was created to demonstrate a fully fledged fullstack application built with Golang/Gin/Slowpoke including CRUD operations, authentication, routing, pagination, and more.

Basic example

package main

import (
	"fmt"

	"github.com/recoilme/slowpoke"
)

func main() {
	// create database
	file := "test/example.db"
	// close all opened database
	defer slowpoke.CloseAll()
	// init key/val
	key := []byte("foo")
	val := []byte("bar")
	//store
	slowpoke.Set(file, key, val)
	// get
	res, _ := slowpoke.Get(file, key)
	//result
	fmt.Println(string(res))
}

Lazy example

func TestGob(t *testing.T) {
	file := "test/gob.db"
	DeleteFile(file)
	defer CloseAll()
	type Post struct {
		Id       int
		Content  string
		Category string
	}

	for i := 0; i < 20; i++ {
		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
		err := SetGob(file, post.Id, post)
		ch(err, t)
	}

	for i := 0; i < 20; i++ {
		var post = new(Post)
		err := GetGob(file, i, post)
		ch(err, t)
		fmt.Println("i:", i, "Post:", post)
	}
}

Advanced example

type Post struct {
	Id       int
	Content  string
	Category string
}

func main() {
	posts := "test/posts"
	tags := "test/tags"
	var pairs [][]byte
	for i := 0; i < 40; i++ {
		id := make([]byte, 4)
		binary.BigEndian.PutUint32(id, uint32(i))
		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)}
		b, _ := json.Marshal(post)
		pairs = append(pairs, id)
		pairs = append(pairs, b)
		tag := fmt.Sprintf("%s:%08d", strconv.Itoa(i/10), i)
		//store only tags keys
		slowpoke.Set(tags, []byte(tag), nil)
	}
	//store posts fast
	slowpoke.Sets(posts, pairs)

	//get last 2 post key with offset 2
	limit := uint32(2)
	offset := uint32(2)
	order := false //desc
	keys, _ := slowpoke.Keys(posts, nil, limit, offset, order)
	fmt.Println(keys) //[[0 0 0 37] [0 0 0 36]]

	//get key/ values
	res := slowpoke.Gets(posts, keys)
	for k, v := range res {
		if k%2 == 0 {
			fmt.Print(binary.BigEndian.Uint32(v))
		} else {
			var p Post
			json.Unmarshal(v, &p)
			fmt.Println(p)
		}
	}
	//37{37 Content:37 Category:3}
	//36{36 Content:36 Category:3}

	//free from memory
	slowpoke.Close(posts)
	slowpoke.Close(tags)

	//open Db and read tags by prefix 2:* in ascending order
	tagsKeys, _ := slowpoke.Keys(tags, []byte("2:*"), 0, 0, true)
	for _, v := range tagsKeys {
		fmt.Print(string(v) + ", ")
	}
	//2:00000020, 2:00000021, 2:00000022, 2:00000023, 2:00000024, 2:00000025, 2:00000026, 2:00000027, 2:00000028, 2:00000029,
}

Api

All methods are thread-safe.

  • Set/Sets/SetGob

Store val and key. If the file does not exist it will be created.

  • Get/Gets/GetGob

Return the value for the given key or nil and an error. Get will open the database if necessary.

  • Keys

Return keys in ascending/descending order.

With limit and offset.

If from is not nil, return keys lexicographically greater than the from value.

If from ends with asterix *, return keys with the prefix equal to from without the asterix.

Documentation

Status

Used in production (master branch)

Benchmark

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

pogreb goleveldb bolt badgerdb pudge slowpoke pudge(mem)
1M (Put+Get), seconds 187 38 126 34 23 23 2
1M Put, ops/sec 5336 34743 8054 33539 47298 46789 439581
1M Get, ops/sec 1782423 98406 499871 220597 499172 445783 1652069
FileSize,Mb 568 357 552 487 358 358 358

Documentation

Overview

Package slowpoke implements a low-level key/value store in pure Go. Keys stored in memory, Value stored on disk

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Close 

func Close(file string) (err error)

Close - close Db and free used memory It run finalizer and cancel goroutine

func CloseAll 

func CloseAll() (err error)

CloseAll - close all opened Db

func Count 

func Count(file string) (uint64, error)

Count return count of keys or error if any

func Counter 

func Counter(file string, key []byte) (counter uint64, err error)

Counter return unique uint64

func Delete 

func Delete(file string, key []byte) (bool, error)

Delete key (always return true) Delete not remove any data from files Return error if any

func DeleteFile 

func DeleteFile(file string) (err error)

DeleteFile close file key and file val and delete db from map and disk All data will be loss!

func Get 

func Get(file string, key []byte) (val []byte, err error)

Get return value by key or nil and error Get will open Db if it closed return error if any

func GetGob 

func GetGob(file string, key interface{}, val interface{}) (err error)

GetGob - experimental future for lazy usage, see tests

func Gets 

func Gets(file string, keys [][]byte) (result [][]byte)

Gets return key/value pairs in random order result contains key and value Gets not return error if key not found If no keys found return empty result

func Has 

func Has(file string, key []byte) (exist bool, err error)

Has return true if key exist or error if any

func Keys 

func Keys(file string, from []byte, limit, offset uint32, asc bool) ([][]byte, error)

Keys return keys in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset>0 - skip offset records If from not nil - return keys after from (from not included) If last byte of from == "*" - return keys with this prefix

func Open 

func Open(file string) (db *pudge.Db, err error)

Open open/create Db (with dirs) This operation is locked by mutex Return error if any Create .idx file for key storage

func Put 

func Put(file string, key []byte, val []byte) (err error)

Put store val and key with sync at end. It's wrapper for Set.

func Set 

func Set(file string, key []byte, val []byte) (err error)

Set store val and key with sync at end File - may be existing file or new If path to file contains dirs - dirs will be created If val is nil - will store only key

func SetGob 

func SetGob(file string, key interface{}, val interface{}) (err error)

SetGob - experimental future for lazy usage, see tests

func Sets 

func Sets(file string, pairs [][]byte) (err error)

Sets store vals and keys Sync will called only at end of insertion Use it for mass insertion every pair must contain key and value

Types

This section is empty.

Source Files

View all Source files

Directories

Path Synopsis
benchmark
boltbench
slowpokebench
Examples PUT: # params host/database/store/key and value in body curl -X PUT -H "Content-Type: application/octet-stream" --data-binary "@durov.jpg" localhost:5000/bolt/images/durov curl -X PUT -H "Content-Type: text/html" -d '{"username":"xyz","password":"xyz"}' localhost:5000/bolt/users/user1 curl -X PUT -H "Content-Type: text/html" -d 'some value' localhost:5000/bolt/users/user2 GET: # params host/database/backet/key curl localhost:5000/bolt/images/durov return: bytes curl localhost:5000/bolt/users/user1 return: {"username":"xyz","password":"xyz"} curl -v localhost:5000/bolt/images/durov2 return 404 Error POST: # params host/database/backet/key?cnt=1000&order=desc&vals=false key: first key, possible values "some_your_key" or "some_your_key*" for prefix scan, Last, First - default Last cnt: return count records, default 1000 order: sorting order (keys ordered as strings!), default desc vals: return values, default false curl -X POST localhost:5000/bolt/users return: {"user2","user1"} curl -X POST localhost:5000/bolt/users/ return: {"user1"} curl -X POST "http://localhost:5000/bolt/users/use*?order=asc&vals=true" return: {"user1":"{"username":"xyz","password":"xyz"}","user2":"some value"} curl -X POST "http://localhost:5000/bolt/users/user2?order=desc&vals=true" return: {"user2":"some value","user1":"{"username":"xyz","password":"xyz"}"} DELETE: curl -X DELETE http://localhost:5000/bolt/users/user2 return 200 Ok (or 404 Error if bucket! not found)
 Examples PUT: # params host/database/store/key and value in body curl -X PUT -H "Content-Type: application/octet-stream" --data-binary "@durov.jpg" localhost:5000/bolt/images/durov curl -X PUT -H "Content-Type: text/html" -d '{"username":"xyz","password":"xyz"}' localhost:5000/bolt/users/user1 curl -X PUT -H "Content-Type: text/html" -d 'some value' localhost:5000/bolt/users/user2 GET: # params host/database/backet/key curl localhost:5000/bolt/images/durov return: bytes curl localhost:5000/bolt/users/user1 return: {"username":"xyz","password":"xyz"} curl -v localhost:5000/bolt/images/durov2 return 404 Error POST: # params host/database/backet/key?cnt=1000&order=desc&vals=false key: first key, possible values "some_your_key" or "some_your_key*" for prefix scan, Last, First - default Last cnt: return count records, default 1000 order: sorting order (keys ordered as strings!), default desc vals: return values, default false curl -X POST localhost:5000/bolt/users return: {"user2","user1"} curl -X POST localhost:5000/bolt/users/ return: {"user1"} curl -X POST "http://localhost:5000/bolt/users/use*?order=asc&vals=true" return: {"user1":"{"username":"xyz","password":"xyz"}","user2":"some value"} curl -X POST "http://localhost:5000/bolt/users/user2?order=desc&vals=true" return: {"user2":"some value","user1":"{"username":"xyz","password":"xyz"}"} DELETE: curl -X DELETE http://localhost:5000/bolt/users/user2 return 200 Ok (or 404 Error if bucket! not found)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.