versiondb

README

VersionDB

VersionDB is a solution for the size issue of IAVL database, aka. application.db, at current stage, it's only recommended for archive and non-validator nodes to try (validator nodes are recommended to do pruning anyway).

VersionDB stores multiple versions of on-chain state key-value pairs directly, without using a merklized tree structure like IAVL tree, both db size and query performance are much better than IAVL tree. The major lacking feature compared to IAVL tree is root hash and merkle proof generation, so we still need IAVL tree for those tasks.

Currently grpc query service don't need to support proof generation, so versiondb alone is enough to support grpc query service, there's already a --grpc-only flag for one to start a standalone grpc query service.

There could be different implementations for the idea of versiondb, the current implementation we delivered is based on rocksdb v7's experimental user-defined timestamp^1, it stores the data in a standalone rocksdb instance, it don't support other db backend yet, but the other databases in the node still support multiple backends as before.

After versiondb is enabled, there's no point to keep the full the archived IAVL tree anymore, it's recommended to prune the IAVL tree to keep only recent versions, for example versions within the unbonding period or even less.

Configuration

To enable versiondb, add versiondb to the list of store.streamers in app.toml like this:

[store]
streamers = ["versiondb"]

On startup, the node will create a StreamingService to subscribe to latest state changes in realtime and save them to versiondb, the db instance is placed at $NODE_HOME/data/versiondb directory, there's no way to customize the db path currently. It'll also switch grpc query service's backing store to versiondb from IAVL tree, you should migrate the legacy states in advance to make the transition smooth, otherwise, the grpc queries can't see the legacy versions.

If the versiondb is not empty and it's latest version doesn't match the IAVL db's last committed version, the startup will fail with error message "versiondb lastest version %d doesn't match iavl latest version %d", that's to avoid creating gaps in versiondb accidentally. When this error happens, you just need to update versiondb to the latest version in iavl tree manually, or restore IAVL db to the same version as versiondb (see ).

Migration

Since our chain is pretty big now, a lot of efforts have been put to make sure the transition process can finish in practical time. The migration process will try to parallelize the tasks as much as possible, and use significant ram, but there's flags for user to control the concurrency level and ram usage to make it runnable on different machine specs.

The legacy state migration process is done in two main steps:

• Extract state change sets from existing archived IAVL tree.
• Feed the change set files to versiondb.

Extract Change Sets

$ elysiumd changeset dump data --home /chain/.elysiumd

dump command will extract the change sets from the IAVL tree, and store each store in separate directories, it use the store list registered in current version of App by default, you can customize that with --stores parameter. The change set files are segmented into different block chunks and compressed with zlib level 6 by default, the chunk size defaults to 1m blocks, the result data directly looks like this:

data/acc/block-0.zz
data/acc/block-1000000.zz
data/acc/block-2000000.zz
...
data/authz/block-0.zz
data/authz/block-1000000.zz
data/authz/block-2000000.zz
...

The extraction is the slowest step, the test run on testnet archive node takes around 11 hours on a 8core ssd machine, but fortunately, the change set files can be verified pretty fast(a few minutes), so they can be share on CDN in a trustless manner, normal users should just download them from CDN and verify the correctness locally, should be much faster than extract by yourself.

For rocksdb backend, dump command opens the db in readonly mode, it can run on live node's db, but goleveldb backend don't support this feature yet.

Verify Change Sets

$ elysiumd changeset verify data
35b85a775ff51cbcc48537247eb786f98fc6a178531d48560126e00f545251be
{"version":"189","storeInfos":[{"name":"acc","commitId":{"version":"189" ...

verify command will replay all the change sets and rebuild the target IAVL tree and output the app hash and commit info of the target version (defaults to latest version in the change sets), then user can manually check the app hash against the block headers.

verify command takes several minutes and several gigabytes of ram to run, if ram usage is a problem, it can also run incrementally, you can export the snapshot for a middle version, then verify the remaining versions start from that snapshot:

$ elysiumd changeset verify data --save-snapshot snapshot --target-version 3000000
$ elysiumd changeset verify data --load-snapshot snapshot

The format of change set files are documented here.

Build VersionDB

To maximize the speed of initial data ingestion speed into rocksdb, we take advantage of the sst file writer feature to write out sst files first, then ingest them into final db, the sst files for each store can be written out in parallel. We also developed an external sorting algorithm to sort the data before writing the sst files, so the sst files don't have overlaps and can be ingested into the bottom-most level in final db.

$ elysiumd changeset build-versiondb-sst ./data ./sst
$ elysiumd changeset ingest-versiondb-sst /home/.elysiumd/data/versiondb sst/*.sst --move-files --maximum-version 189

User can control the peak ram usage by controlling the --concurrency and --sorter-chunk-size.

With default parameters it can finish in around 12minutes for testnet archive node on our test node (8cores, peak RSS 2G).

Restore IAVL Tree

When migrating an existing archive node to versiondb, it's recommended to rebuild the application.db from scratch to reclaim disk space faster, we provide a command to restore a single version of IAVL trees from memiavl snapshot.

$ # create memiavl snapshot
$ elysiumd changeset verify data --save-snapshot snapshot
$ # restore application.db
$ elysiumd changeset restore-app-db snapshot application.db

Then replace the whole application.db in the node with the newly generated one.

It only takes a few minutes to run on our testnet archive node, it only suppot generating rocksdb application.db right now, so please set app-db-backend="rocksdb" in app.toml.

Catch Up With IAVL Tree

If an non-empty versiondb lags behind from the current application.db, the node will refuse to startup, in this case user can either sync versiondb to catch up with application.db, or simply restore the application.db with the correct version of snapshot. To catch up, you can follow the similar procedure as migrating from genesis, just passing the block range in change set dump command.

Documentation

Index

• Constants
• func Run(t *testing.T, storeCreator func() VersionStore)
• func SetupTestDB(t *testing.T, store VersionStore)
• type MultiStore
  • func NewMultiStore(versionDB VersionStore, storeKeys []types.StoreKey) *MultiStore
  • func (s *MultiStore) CacheMultiStore() sdk.CacheMultiStore
  • func (s *MultiStore) CacheMultiStoreWithVersion(version int64) (sdk.CacheMultiStore, error)
  • func (s *MultiStore) CacheWrap() types.CacheWrap
  • func (s *MultiStore) CacheWrapWithTrace(_ io.Writer, _ types.TraceContext) types.CacheWrap
  • func (s *MultiStore) GetKVStore(storeKey types.StoreKey) sdk.KVStore
  • func (s *MultiStore) GetStore(storeKey types.StoreKey) sdk.Store
  • func (s *MultiStore) GetStoreType() types.StoreType
  • func (s *MultiStore) LatestVersion() int64
  • func (s *MultiStore) MountMemoryStores(keys map[string]*types.MemoryStoreKey)
  • func (s *MultiStore) MountTransientStores(keys map[string]*types.TransientStoreKey)
  • func (s *MultiStore) SetTracer(w io.Writer) types.MultiStore
  • func (s *MultiStore) SetTracingContext(tc types.TraceContext) types.MultiStore
  • func (s *MultiStore) TracingEnabled() bool
• type Store
  • func NewKVStore(store VersionStore, storeKey types.StoreKey, version *int64) *Store
  • func (st *Store) CacheWrap() types.CacheWrap
  • func (st *Store) CacheWrapWithListeners(storeKey types.StoreKey, listeners []types.WriteListener) types.CacheWrap
  • func (st *Store) CacheWrapWithTrace(w io.Writer, tc types.TraceContext) types.CacheWrap
  • func (st *Store) Delete(key []byte)
  • func (st *Store) Get(key []byte) []byte
  • func (st *Store) GetStoreType() types.StoreType
  • func (st *Store) Has(key []byte) (exists bool)
  • func (st *Store) Iterator(start, end []byte) types.Iterator
  • func (st *Store) ReverseIterator(start, end []byte) types.Iterator
  • func (st *Store) Set(key, value []byte)
• type StreamingService
  • func NewStreamingService(versionStore VersionStore, storeKeys []types.StoreKey) *StreamingService
  • func (fss *StreamingService) Close() error
  • func (fss *StreamingService) ListenBeginBlock(ctx context.Context, req abci.RequestBeginBlock, res abci.ResponseBeginBlock) error
  • func (fss *StreamingService) ListenCommit(ctx context.Context, res abci.ResponseCommit) error
  • func (fss *StreamingService) ListenDeliverTx(ctx context.Context, req abci.RequestDeliverTx, res abci.ResponseDeliverTx) error
  • func (fss *StreamingService) ListenEndBlock(ctx context.Context, req abci.RequestEndBlock, res abci.ResponseEndBlock) error
  • func (fss *StreamingService) Listeners() map[types.StoreKey][]types.WriteListener
  • func (fss *StreamingService) Stream(wg *sync.WaitGroup) error
• type VersionStore

Constants

const StoreTypeVersionDB = 100

Functions

func Run

func Run(t *testing.T, storeCreator func() VersionStore)

func SetupTestDB

func SetupTestDB(t *testing.T, store VersionStore)

Types

type MultiStore

type MultiStore struct {
	// contains filtered or unexported fields
}

MultiStore wraps `VersionStore` to implement `MultiStore` interface.

func NewMultiStore

func NewMultiStore(versionDB VersionStore, storeKeys []types.StoreKey) *MultiStore

NewMultiStore returns a new versiondb `MultiStore`.

func (*MultiStore) CacheMultiStore

func (s *MultiStore) CacheMultiStore() sdk.CacheMultiStore

CacheMultiStore implements `MultiStore` interface

func (*MultiStore) CacheMultiStoreWithVersion

func (s *MultiStore) CacheMultiStoreWithVersion(version int64) (sdk.CacheMultiStore, error)

CacheMultiStoreWithVersion implements `MultiStore` interface

func (*MultiStore) CacheWrap

func (s *MultiStore) CacheWrap() types.CacheWrap

CacheWrap implements CacheWrapper/MultiStore/CommitStore.

func (*MultiStore) CacheWrapWithTrace

func (s *MultiStore) CacheWrapWithTrace(_ io.Writer, _ types.TraceContext) types.CacheWrap

CacheWrapWithTrace implements the CacheWrapper interface.

func (*MultiStore) GetKVStore

func (s *MultiStore) GetKVStore(storeKey types.StoreKey) sdk.KVStore

GetKVStore implements `MultiStore` interface

func (*MultiStore) GetStore

func (s *MultiStore) GetStore(storeKey types.StoreKey) sdk.Store

GetStore implements `MultiStore` interface

func (*MultiStore) GetStoreType

func (s *MultiStore) GetStoreType() types.StoreType

GetStoreType implements `MultiStore` interface.

func (*MultiStore) LatestVersion

func (s *MultiStore) LatestVersion() int64

LatestVersion returns the latest version saved in versiondb

func (*MultiStore) MountMemoryStores

func (s *MultiStore) MountMemoryStores(keys map[string]*types.MemoryStoreKey)

MountMemoryStores simlates the same behavior as sdk to support grpc query service.

func (*MultiStore) MountTransientStores

func (s *MultiStore) MountTransientStores(keys map[string]*types.TransientStoreKey)

MountTransientStores simlates the same behavior as sdk to support grpc query service.

func (*MultiStore) SetTracer

func (s *MultiStore) SetTracer(w io.Writer) types.MultiStore

SetTracer sets the tracer for the MultiStore that the underlying stores will utilize to trace operations. A MultiStore is returned.

func (*MultiStore) SetTracingContext

func (s *MultiStore) SetTracingContext(tc types.TraceContext) types.MultiStore

SetTracingContext updates the tracing context for the MultiStore by merging the given context with the existing context by key. Any existing keys will be overwritten. It is implied that the caller should update the context when necessary between tracing operations. It returns a modified MultiStore.

func (*MultiStore) TracingEnabled

func (s *MultiStore) TracingEnabled() bool

TracingEnabled returns if tracing is enabled for the MultiStore.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store Implements types.KVStore

func NewKVStore

func NewKVStore(store VersionStore, storeKey types.StoreKey, version *int64) *Store

func (*Store) CacheWrap

func (st *Store) CacheWrap() types.CacheWrap

Implements Store.

func (*Store) CacheWrapWithListeners

func (st *Store) CacheWrapWithListeners(storeKey types.StoreKey, listeners []types.WriteListener) types.CacheWrap

CacheWrapWithListeners implements the CacheWrapper interface.

func (*Store) CacheWrapWithTrace

func (st *Store) CacheWrapWithTrace(w io.Writer, tc types.TraceContext) types.CacheWrap

CacheWrapWithTrace implements the Store interface.

func (*Store) Delete

func (st *Store) Delete(key []byte)

Implements types.KVStore.

func (*Store) Get

func (st *Store) Get(key []byte) []byte

Implements types.KVStore.

func (*Store) GetStoreType

func (st *Store) GetStoreType() types.StoreType

Implements Store.

func (*Store) Has

func (st *Store) Has(key []byte) (exists bool)

Implements types.KVStore.

func (*Store) Iterator

func (st *Store) Iterator(start, end []byte) types.Iterator

Implements types.KVStore.

func (*Store) ReverseIterator

func (st *Store) ReverseIterator(start, end []byte) types.Iterator

Implements types.KVStore.

func (*Store) Set

func (st *Store) Set(key, value []byte)

Implements types.KVStore.

type StreamingService

type StreamingService struct {
	// contains filtered or unexported fields
}

StreamingService is a concrete implementation of StreamingService that accumulate the state changes in current block, writes the ordered changeset out to version storage.

func NewStreamingService

func NewStreamingService(versionStore VersionStore, storeKeys []types.StoreKey) *StreamingService

NewStreamingService creates a new StreamingService for the provided writeDir, (optional) filePrefix, and storeKeys

func (*StreamingService) Close

func (fss *StreamingService) Close() error

Close satisfies the io.Closer interface, which satisfies the baseapp.StreamingService interface

func (*StreamingService) ListenBeginBlock

func (fss *StreamingService) ListenBeginBlock(ctx context.Context, req abci.RequestBeginBlock, res abci.ResponseBeginBlock) error

ListenBeginBlock satisfies the baseapp.ABCIListener interface It sets the currentBlockNumber.

func (*StreamingService) ListenCommit

func (fss *StreamingService) ListenCommit(ctx context.Context, res abci.ResponseCommit) error

func (*StreamingService) ListenDeliverTx

func (fss *StreamingService) ListenDeliverTx(ctx context.Context, req abci.RequestDeliverTx, res abci.ResponseDeliverTx) error

ListenDeliverTx satisfies the baseapp.ABCIListener interface

func (*StreamingService) ListenEndBlock

func (fss *StreamingService) ListenEndBlock(ctx context.Context, req abci.RequestEndBlock, res abci.ResponseEndBlock) error

ListenEndBlock satisfies the baseapp.ABCIListener interface It merge the state caches of all the listeners together, and write out to the versionStore.

func (*StreamingService) Listeners

func (fss *StreamingService) Listeners() map[types.StoreKey][]types.WriteListener

Listeners satisfies the baseapp.StreamingService interface

func (*StreamingService) Stream

func (fss *StreamingService) Stream(wg *sync.WaitGroup) error

Stream satisfies the baseapp.StreamingService interface

type VersionStore

type VersionStore interface {
	GetAtVersion(storeKey string, key []byte, version *int64) ([]byte, error)
	HasAtVersion(storeKey string, key []byte, version *int64) (bool, error)
	IteratorAtVersion(storeKey string, start, end []byte, version *int64) (types.Iterator, error)
	ReverseIteratorAtVersion(storeKey string, start, end []byte, version *int64) (types.Iterator, error)
	GetLatestVersion() (int64, error)

	// Persist the change set of a block,
	// the `changeSet` should be ordered by (storeKey, key),
	// the version should be latest version plus one.
	PutAtVersion(version int64, changeSet []types.StoreKVPair) error
}

VersionStore is a versioned storage of a flat key-value pairs. it don't need to support merkle proof, so could be implemented in a much more efficient way. `nil` version means the latest version.