Documentation ¶
Overview ¶
GravitonDB in short is "ZFS for key-value stores".
GravitonDB is a pure Go key/value store having features unmatched by other software (such as Boltdb, berkeleydb, mysql, postgresql etc). The goal of the project is to provide a simple, fast, reliable, versioned, authenticated database for projects which require such features.
Since GravitonDB is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only focus on getting values and setting values. That's it.
GravitonDB is a key value store having 1) cryptographically authenticated ( a single hash verifies complete data store) 2) append only 3) versioning 4) Writable snapshots 5) exabyte storage capable, (tested upto Terabytes, if you can help with testing petabyte capability please discuss with authors) 6) cryptographic proof capability to prove existence/non-existence of any arbitrary key 7) ability to diff 2 trees in linear time 8) designed and developed for DERO blockchain Features * Authenticated data store ( all keys, values are backed by blake 256 bit checksum) * Append only data store * Support of 2^64 trees ( Theoretically ) within a single data store. trees can be named and thus used as buckets * Versioning support ( all committed changes are versioned with ability to visit them at any point in time ) * Snapshots ( multi tree commits in a single version causing multi tree sync, each snapshot can be visited, appended and further modified, keys deleted, values modified etc, new keys, values stored ) * Ability to iterate over all key-value pairs in a tree * Ability to diff between 2 trees in linear time and report all changes (insertions,deletions,modifications) * Minimal, small, simplified API * Theoretically support Exabyte data store, multi Terabyte tested internally * Decoupled storage layer, allowing use of object stores such as Ceph, AWS etc * Ability to generate cryptographic proofs which can prove key existance or non-existance ( proofs are around 1 KB ) * Superfast proof generation time of around 1000 proofs per second per core * Support for disk based filesystem based persistant stores. * Support for memory based non-persistant stores * 100% code coverage * this is alpha software, we are still evaluating a number of ideas
Eg. Minimal code, to write and read back a value (error checking is skipped)
store, _ := NewDiskStore("/tmp/testdb") // create a new testdb in "/tmp/testdb" ss, _ := store.LoadSnapshot(0) // load most recent snapshot tree, _ := ss.GetTree("root") // use or create tree named "root" tree.Put([]byte("key"), []byte("value")) // insert a value Commit(tree) // commit the tree value, _ = tree.Get([]byte("key"))
Eg, Snapshots, see github.com/deroproject/gravitondb/examples/snapshot_example//snapshot_example.go
The design enables infinite trees with infinite snapshots. The design enables designs such as dedeuplicating backups, blockchains which enable proving their data ( both state and content) to users etc
Index ¶
- Constants
- Variables
- func Commit(trees ...*Tree) (committed_version uint64, err error)
- func Diff(base_tree, head_tree *Tree, deleted, modified, inserted DiffHandler) (err error)
- func Sum(key []byte) [HASHSIZE]byte
- type Cursor
- type DiffHandler
- type Proof
- func (p *Proof) Marshal() []byte
- func (p *Proof) MarshalTo(b *bytes.Buffer)
- func (p *Proof) Reset()
- func (p *Proof) Unmarshal(buf []byte) error
- func (p *Proof) Value() []byte
- func (p *Proof) VerifyMembership(root [HASHSIZE]byte, key []byte) bool
- func (p *Proof) VerifyNonMembership(root [HASHSIZE]byte, key []byte) bool
- type Snapshot
- func (s *Snapshot) GetTree(treename string) (*Tree, error)
- func (s *Snapshot) GetTreeHighestVersion(treename string) (uint64, error)
- func (s *Snapshot) GetTreeWithRootHash(roothash []byte) (*Tree, error)
- func (s *Snapshot) GetTreeWithTag(tag string) (*Tree, error)
- func (s *Snapshot) GetTreeWithVersion(treename string, version uint64) (*Tree, error)
- func (s *Snapshot) GetVersion() uint64
- type Store
- type Tree
- func (t *Tree) Commit(tags ...string) error
- func (t *Tree) Cursor() Cursor
- func (t *Tree) Delete(key []byte) error
- func (t *Tree) Discard() error
- func (t *Tree) GenerateProof(key []byte) (*Proof, error)
- func (t *Tree) Get(key []byte) ([]byte, error)
- func (t *Tree) GetKeyValueFromHash(keyhashc []byte) (int, []byte, []byte, error)
- func (t *Tree) GetKeyValueFromKey(key []byte) (int, []byte, []byte, error)
- func (t *Tree) GetName() string
- func (t *Tree) GetParentVersion() uint64
- func (t *Tree) GetVersion() uint64
- func (t *Tree) Hash() (h [HASHSIZE]byte, err error)
- func (t *Tree) IsDirty() bool
- func (t *Tree) KeyCountEstimate() (count int64)
- func (t *Tree) Put(key, value []byte) error
- func (t *Tree) Random() (k, v []byte, err error)
Constants ¶
const ( HASHSIZE_BYTES = 32 // we currently are using blake hash which is 256 bits or 32 bytes HASHSIZE = HASHSIZE_BYTES HASHSIZE_BITS = HASHSIZE_BYTES * 8 // hash size in bits MINBLOCK = 512 // max block size excluding value MAX_KEYSIZE = MINBLOCK - 64 // 64 bytes are reserved, keys are limited to 448 bytes MAX_FILE_SIZE = 2 * 1024 * 1024 * 1024 // 2GB since we use split files to store data chunks MAX_VALUE_SIZE = 100 * 1024 * 1024 // values are limited to this size TREE_NAME_LIMIT = 127 // TREE name cannot be larger than this in bytes ( not in utf8 chars) )
Variables ¶
Functions ¶
func Commit ¶
Commit the tree (or a number of trees) to persistance, write a new snapshot which can be accessed henceforth without any modifications Commiting multiple trees or multiple changes as batch is much more effecient than committing each change independently.
func Diff ¶
func Diff(base_tree, head_tree *Tree, deleted, modified, inserted DiffHandler) (err error)
This function can be used to diff 2 trees and thus find all the keys which have been deleted, modified, inserted. The algorithm is linear time in the number of changes. Eg. a tree with billion KVs can be diffed with parent almost instantaneously. Todo : API redesign, should we give only keys, since values can be obtained later on !!
Types ¶
type Cursor ¶
type Cursor struct {
// contains filtered or unexported fields
}
Cursor represents an iterator that can traverse over all key/value pairs in a tree in hash sorted order. Cursors can be obtained from a tree and are valid as long as the tree is valid. Keys and values returned from the cursor are only valid for the life of the transaction. Changing tree (before committing) while traversing with a cursor may cause it to be invalidated and return unexpected keys and/or values. You must reposition your cursor after mutating data.
func (*Cursor) First ¶
First moves the cursor to the first item in the tree and returns its key and value. If the tree is empty then an error is returned. The returned key and value are only valid for the life of the tree.
func (*Cursor) Last ¶
Last moves the cursor to the last item in the tree and returns its key and value. If the tree is empty then an error is returned. The returned key and value are only valid for the life of the tree.
func (*Cursor) Next ¶
Next moves the cursor to the next item in the tree and returns its key and value.If the tree is empty then an error is returned.If the cursor is at the end of the tree, then an error is returned. The returned key and value are only valid for the life of the tree.
type DiffHandler ¶
type DiffHandler func(k, v []byte)
All changes are reported of this type, deleted, modified, inserted
type Proof ¶
type Proof struct {
// contains filtered or unexported fields
}
This structure is used to prove existence/non-existence of a key
func (*Proof) MarshalTo ¶
Serialize the proof to a bytes Buffer
the following are the size requirements for proof 1 byte for version 1 byte for type varint trace length 32 byte(HASHSIZE) tracebits, bit 1 is set if hash is not zerohash 32 byte(HASHSIZE) * number of trace bits set if collision is there 32 byte(HASHSIZE) key , 32 byte(HASHSIZE) value if member, varint lenth var int length prefixed value dead end = 0
func (*Proof) Unmarshal ¶
Unmarshal follows reverse of marshal to deserialize the array of bytes to proof for verification.
func (*Proof) Value ¶
if the proof is for existence for a key, it's associated value can be read here
func (*Proof) VerifyMembership ¶
type Snapshot ¶
type Snapshot struct {
// contains filtered or unexported fields
}
Snapshot are used to access any arbitrary snapshot of entire database at any point in time snapshot refers to collective state of all trees + data (key-values) + history each commit ( tree.Commit() or Commit(tree1, tree2 .....)) creates a new snapshot each snapshot is represented by an incrementing uint64 number, 0 represents most recent snapshot.
TODO we may need to provide API to export DB at specific snapshot version
func (*Snapshot) GetTreeHighestVersion ¶
Gets highest stored version number of the specific tree
func (*Snapshot) GetTreeWithRootHash ¶
Gets the tree which has specific roothash
func (*Snapshot) GetTreeWithTag ¶
Gets the tree which has specific tag
NOTE: same tags might point to different trees in different snapshots of db
func (*Snapshot) GetTreeWithVersion ¶
Load a versioned tree from the store all trees have there own version number
func (*Snapshot) GetVersion ¶
Gets the snapshot version number
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the backend which is used to store the data in serialized form to disk. The data is stored in files in split format and total number of files can be 4 billion. each file is upto 2 GB in size, this limit has been placed to support FAT32 which restricts files to 4GB
func NewDiskStore ¶
open/create a disk based store, if the directory pre-exists, it is used as is. Since we are an append only keyvalue store, we do not delete any data.
func NewMemStore ¶
start a new memory backed store which may be useful for testing and other temporaray use cases.
func (*Store) LoadSnapshot ¶
Load a specific snapshot from the store, 0th version = load most recent version as a special case note: 0th tree is not stored in disk also note that commits are being done so versions might be change
type Tree ¶
type Tree struct { Tags []string // tags used while commit, will get cleaned after commit // contains filtered or unexported fields }
Tree structure which is the end result, TODO: tree does not cache anything currently, just caching the top level tree entries, will increase the speed by X multiplication factors.
func (*Tree) Cursor ¶
get Cursor which is used as an iterator that can traverse over all key/value pairs in a tree in hash sorted order.
func (*Tree) GenerateProof ¶
Generate proof of any key, which can be used to prove whether the key exists or not.Please note that the tree root hash (tree.Hash()) is not part of the structure and must be available to the verifier separately for eg. in an encrypted blockchain, the entire state is carried forward from block to block, this state can be queried from a number of sources and then it is verified
func (*Tree) Get ¶
Get a specifically value associated with a key TODO, we need to expose this in other forms so as memory allocations and better error detection could be done
func (*Tree) GetKeyValueFromHash ¶
we only have a keyhash and need to get both the key,value
func (*Tree) GetKeyValueFromKey ¶
we have a key and need to get both the key,value
func (*Tree) GetParentVersion ¶
Get parent version number of tree from which this tree was derived, they might not be sequential but they will be monotonically increasing this can be used to build out a DAG
func (*Tree) KeyCountEstimate ¶
estimate number of keys that exist in the tree very crude but only used for use display
func (*Tree) Put ¶
put a key value in the tree, if the value exists, it's overwritten. ToDO: it should ignore duplicate key value, if first using a get and then a put