README

Documentation Go Report Card Build Status Mentioned in Awesome Go

Table of Contents

Description

Package pudge is a fast and simple key/value store written using Go's standard library.

It presents the following:

  • Supporting very efficient lookup, insertions and deletions
  • Performance is comparable to hash tables
  • Ability to get the data in sorted order, which enables additional operations like range scan
  • Select with limit/offset/from key, with ordering or by prefix
  • Safe for use in goroutines
  • Space efficient
  • Very short and simple codebase
  • Well tested, used in production

pudge

Usage

package main

import (
	"log"

	"github.com/recoilme/pudge"
)

func main() {
	// Close all database on exit
	defer pudge.CloseAll()

	// Set (directories will be created)
	pudge.Set("../test/test", "Hello", "World")

	// Get (lazy open db if needed)
	output := ""
	pudge.Get("../test/test", "Hello", &output)
	log.Println("Output:", output)

	ExampleSelect()
}


//ExampleSelect
func ExampleSelect() {
	cfg := &pudge.Config{
		SyncInterval: 1} // every second fsync
	db, err := pudge.Open("../test/db", cfg)
	if err != nil {
		log.Panic(err)
	}
	defer db.DeleteFile()
	type Point struct {
		X int
		Y int
	}
	for i := 100; i >= 0; i-- {
		p := &Point{X: i, Y: i}
		db.Set(i, p)
	}
	var point Point
	db.Get(8, &point)
	log.Println(point)
	// Output: {8 8}
	// Select 2 keys, from 7 in ascending order
	keys, _ := db.Keys(7, 2, 0, true)
	for _, key := range keys {
		var p Point
		db.Get(key, &p)
		log.Println(p)
	}
	// Output: {8 8}
	// Output: {9 9}
}

Cookbook

  • Store data of any type. Pudge uses Gob encoder/decoder internally. No limits on keys/values size.
pudge.Set("strings", "Hello", "World")
pudge.Set("numbers", 1, 42)

type User struct {
	Id int
	Name string
}
u := &User{Id: 1, Name: "name"}
pudge.Set("users", u.Id, u)

  • Pudge is stateless and safe for use in goroutines. You don't need to create/open files before use. Just write data to pudge, don't worry about state. web server example

  • Pudge is parallel. Readers don't block readers, but a writer - does, but by the stateless nature of pudge it's safe to use multiples files for storages.

Illustration from slowpoke (based on pudge)

  • Default store system: like memcache + file storage. Pudge uses in-memory hashmap for keys, and writes values to files (no value data stored in memory). But you may use inmemory mode for values, with custom config:
cfg = pudge.DefaultConfig()
cfg.StoreMode = 2
db, err := pudge.Open(dbPrefix+"/"+group, cfg)
...
db.Counter(key, val)

In that case, all data is stored in memory and will be stored on disk only on Close.

Example server for highload, with http api

  • You may use pudge as an engine for creating databases.

Example database

  • Don't forget to close all opened databases on shutdown/kill.
 	// Wait for interrupt signal to gracefully shutdown the server 
	quit := make(chan os.Signal)
	signal.Notify(quit, os.Interrupt, os.Kill)
	<-quit
	log.Println("Shutdown Server ...")
	if err := pudge.CloseAll(); err != nil {
		log.Println("Pudge Shutdown err:", err)
	}

example recovery function for gin framework

  • Pudge has a primitive select/query engine.
// Select 2 keys, from 7 in ascending order
   keys, _ := db.Keys(7, 2, 0, true)
// select keys from db where key>7 order by keys asc limit 2 offset 0
  • Pudge will work well on SSD or spined disks. Pudge doesn't eat memory or storage or your sandwich. No hidden compaction/rebalancing/resizing and so on tasks. No LSM Tree. No MMap. It's a very simple database with less than 500 LOC. It's good for simple social network or highload system

Disadvantages

  • No transaction system. All operations are isolated, but you don't may batching them with automatic rollback.
  • Keys function (select/query engine) may be slow. Speed of query may vary from 10ms to 1sec per million keys. Pudge don't use BTree/Skiplist or Adaptive radix tree for store keys in ordered way on every insert. Ordering operation is "lazy" and run only if needed.
  • No fsync on every insert. Most of database fsync data by the timer too
  • Deleted data don't remove from physically (but upsert will try to reuse space). You may shrink database only with backup right now
pudge.BackupAll("backup")
  • Keys automatically convert to binary and ordered with binary comparator. It's simple for use, but ordering will not work correctly for negative numbers for example
  • Author of project don't work at Google or Facebook and his name not Howard Chu or Brad Fitzpatrick. But I'm open for issue or contributions.

Motivation

Some databases very well for writing. Some of the databases very well for reading. But pudge is well balanced for both types of operations. It has small cute api, and don't have hidden graveyards. It's just hashmap where values written in files. And you may use one database for in-memory/persistent storage in a stateless stressfree way

Benchmarks

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

pogreb goleveldb bolt badgerdb pudge slowpoke pudge(mem)
1M (Put+Get), seconds 187 38 126 34 23 23 2
1M Put, ops/sec 5336 34743 8054 33539 47298 46789 439581
1M Get, ops/sec 1782423 98406 499871 220597 499172 445783 1652069
FileSize,Mb 568 357 552 487 358 358 358
Test 4

Number of keys: 10000000 Key size: 8 Value size: 16 Concurrency: 100

goleveldb badgerdb pudge
10M (Put+Get), seconds 165 120 243
10M Put, ops/sec 122933 135709 43843
10M Get, ops/sec 118722 214981 666067
FileSize,Mb 312 1370 381

Documentation

Overview

    Package pudge implements a low-level key/value store in pure Go. Keys stored in memory, Value stored on disk

    Usage

    package main
    
    import (
    	"log"
    
    	"github.com/recoilme/pudge"
    )
    
    func main() {
    	cfg := pudge.DefaultConfig()
    	cfg.SyncInterval = 0 //disable every second fsync
    	db, err := pudge.Open("../test/db", cfg)
    	if err != nil {
    		log.Panic(err)
    	}
    	defer db.DeleteFile()
    	type Point struct {
    		X int
    		Y int
    	}
    	for i := 100; i >= 0; i-- {
    		p := &Point{X: i, Y: i}
    		db.Set(i, p)
    	}
    	var point Point
    	db.Get(8, &point)
    	log.Println(point)
    	// Output: {8 8}
    }
    

    Index

    Examples

    Constants

    This section is empty.

    Variables

    View Source
    var DefaultConfig = &Config{
    	FileMode:     0666,
    	DirMode:      0777,
    	SyncInterval: 0,
    	StoreMode:    0}

      DefaultConfig is default config

      View Source
      var (
      
      	// ErrKeyNotFound - key not found
      	ErrKeyNotFound = errors.New("Error: key not found")
      )

      Functions

      func BackupAll

      func BackupAll(dir string) (err error)

        BackupAll - backup all opened Db if dir not set it will be backup delete old backup file before run ignore all errors

        func Close

        func Close(f string) error

          Close - sync & close files. Return error if any.

          func CloseAll

          func CloseAll() (err error)

            CloseAll - close all opened Db

            func Count

            func Count(f string) (int, error)

              Count returns the number of items in the Db.

              func Counter

              func Counter(f string, key interface{}, incr int) (int64, error)

                Counter return int64 incremented on incr with lazy open

                func Delete

                func Delete(f string, key interface{}) error

                  Delete remove key Returns error if key not found

                  func DeleteFile

                  func DeleteFile(file string) error

                    DeleteFile close db and delete file

                    func Get

                    func Get(f string, key, value interface{}) error

                      Get return value by key with opening if needed Return error if any.

                      Example
                      Output:
                      
                      World
                      

                      func Gets

                      func Gets(file string, keys []interface{}) (result [][]byte)

                        Gets return key/value pairs in random order result contains key and value Gets not return error if key not found If no keys found return empty result

                        func Has

                        func Has(f string, key interface{}) (bool, error)

                          Has return true if key exists. Return error if any.

                          func KeyToBinary

                          func KeyToBinary(v interface{}) ([]byte, error)

                            KeyToBinary return key in bytes

                            func Keys

                            func Keys(f string, from interface{}, limit, offset int, asc bool) ([][]byte, error)

                              Keys return keys in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset > 0 - skip offset records If from not nil - return keys after from (from not included)

                              func Set

                              func Set(f string, key, value interface{}) error

                                Set store any key value to db with opening if needed

                                Example
                                Output:
                                
                                

                                func Sets

                                func Sets(file string, pairs []interface{}) (err error)

                                  Sets store vals and keys Use it for mass insertion every pair must contain key and value

                                  func ValToBinary

                                  func ValToBinary(v interface{}) ([]byte, error)

                                    ValToBinary return value in bytes

                                    Types

                                    type Cmd

                                    type Cmd struct {
                                    	Seek    uint32
                                    	Size    uint32
                                    	KeySeek uint32
                                    	Val     []byte
                                    }

                                      Cmd represent keys and vals addresses

                                      type Config

                                      type Config struct {
                                      	FileMode     int // 0666
                                      	DirMode      int // 0777
                                      	SyncInterval int // in seconds
                                      	StoreMode    int // 0 - file first, 2 - memory first(with persist on close), 2 - with empty file - memory without persist
                                      }

                                        Config fo db Default FileMode = 0666 Default DirMode = 0777 Default SyncInterval = 0 sec, 0 - disable sync (os will sync, typically 30 sec or so) If StroreMode==2 && file == "" - pure inmemory mode

                                        type Db

                                        type Db struct {
                                        	sync.RWMutex
                                        	// contains filtered or unexported fields
                                        }

                                          Db represent database

                                          func Open

                                          func Open(f string, cfg *Config) (*Db, error)

                                            Open return db object if it opened. Create new db if not exist. Read db to obj if exist. Or error if any. Default Config (if nil): &Config{FileMode: 0666, DirMode: 0777, SyncInterval: 0}

                                            Example
                                            Output:
                                            
                                            {8 8}
                                            

                                            func (*Db) Close

                                            func (db *Db) Close() error

                                              Close - sync & close files. Return error if any.

                                              func (*Db) Count

                                              func (db *Db) Count() (int, error)

                                                Count returns the number of items in the Db.

                                                func (*Db) Counter

                                                func (db *Db) Counter(key interface{}, incr int) (int64, error)

                                                  Counter return int64 incremented on incr

                                                  func (*Db) Delete

                                                  func (db *Db) Delete(key interface{}) error

                                                    Delete remove key Returns error if key not found

                                                    func (*Db) DeleteFile

                                                    func (db *Db) DeleteFile() error

                                                      DeleteFile close and delete file

                                                      func (*Db) FileSize

                                                      func (db *Db) FileSize() (int64, error)

                                                        FileSize returns the total size of the disk storage used by the DB.

                                                        func (*Db) Get

                                                        func (db *Db) Get(key, value interface{}) error

                                                          Get return value by key Return error if any.

                                                          func (*Db) Has

                                                          func (db *Db) Has(key interface{}) (bool, error)

                                                            Has return true if key exists. Return error if any.

                                                            func (*Db) Keys

                                                            func (db *Db) Keys(from interface{}, limit, offset int, asc bool) ([][]byte, error)

                                                              Keys return keys in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset > 0 - skip offset records If from not nil - return keys after from (from not included)

                                                              func (*Db) KeysByPrefix

                                                              func (db *Db) KeysByPrefix(prefix []byte, limit, offset int, asc bool) ([][]byte, error)

                                                                KeysByPrefix return keys with prefix in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset > 0 - skip offset records If from not nil - return keys after from (from not included)

                                                                func (*Db) Set

                                                                func (db *Db) Set(key, value interface{}) error

                                                                  Set store any key value to db

                                                                  Directories

                                                                  Path Synopsis