database

README

README

General concept

Package database represents a database layer in Dusk Network. Current design should embody both backend-agnostic and storage-agnostic concepts.

Terminology

Storage: Underlying storage can be any kind of KV stores (BoltDB, LMDB, LevelDB, RocksDB, Redis), non-KV stores (SQLite) or even a custom ACID-complient flat-file store.

Backend/Driver: represents a DB implementation that uses one or multiple underlying storages to provide block chain database specifics with regard to particular purpose. It must implement Driver, DB and Tx intefaces.

Interfaces exposed to upper layers:

• database.Driver: DB connection initiator. Inspired by database/sql/driver
• database.DB: A thin layer on top of database.Tx providing a manageable Tx execution. Inspired by BoltDB
• database.Tx: A transaction layer tightly coupled with DUSK block chain specifics to fully benefit from underlying storage capabilities

Available Drivers

• /database/heavy driver is designed to provide efficient, robust and persistent DUSK block chain DB on top of syndtr/goleveldb/leveldb store (unofficial LevelDB porting). It must be Mainnet-complient.

Testing Drivers

• /database/testing implements a boilerplate method to verify if a registered driver does satisfy minimum database requirements. The package defines a set of unit tests that are executed only on registered drivers. It can serve also as a detailed and working database guideline.

Code example:

More code examples can be found in /database/heavy/database_test.go

// By changing Driver name, one can switch between different backends
// Each backend is bound to one or multiple underlying stores
readonly := false

// Retrieve
driver, _ := database.From(lite.DriverName)
db, err := driver.Open(path, protocol.DevNet, readonly)

if err != nil {
    ...
}

defer db.Close()

blocks := make([]*block.Block, 100)
// Populate blocks here ...

// a managed read-write DB Tx to store all blocks via atomic update
err = db.Update(func(tx database.Tx) error {
    for _, block := range blocks {
        err := tx.StoreBlock(block)
        if err != nil {
            return err
        }
    }
    return nil
})

if err != nil {
    fmt.Printf("Transaction failed. No blocks stored")
    return
}

 // a managed read-only DB Tx to check if all blocks have been stored
_ = db.View(func(tx database.Tx) error {
    for _, block := range blocks {
        exists, err := tx.FetchBlockExists(block.Header.Hash)
        if err != nil {
            fmt.Printf(err.Error())
            return nil
        }

        if !exists {
            fmt.Printf("Block with Height %d was not found", block.Header.Height)
            return nil
        }
    }
    return nil
})

Blockchain registry

Blockchain database supports (read/write) two additional KV records which store both hash of the blockchain latest block and hash of blockchain latest persisted block. A block has a meaning of a persisted block only if rusk.Persist grpc has been called for it successfully.

FetchRegistry method from Transaction interface should be used to fetch the abovementioned KV records.

Additional features

Additional features that can be provided by a Driver:

• Read-only transactions/connections
• Validated storage connection
• Profiling: Collect transactions and connections statistics
• Traces: Log transactions data
• Alarming: Trigger an event in case of critical backend or storage failure
• Iterators/Cursors
• Safe Concurrency model: One read-write transaction, many read-only transactions
• Redundancy: blockchain data stored in a secondary in-memory or on-disk structure

Documentation

Index

• Variables
• func Drivers() []string
• func Register(driver Driver) error
• type DB
• type Driver
  • func From(name string) (Driver, error)
• type Registry
• type Transaction

Variables

var (

	// ErrTxNotFound returned on a tx lookup by hash.
	ErrTxNotFound = errors.New("database: transaction not found")
	// ErrBlockNotFound returned on a block lookup by hash or height.
	ErrBlockNotFound = errors.New("database: block not found")
	// ErrStateNotFound returned on missing state db entry.
	ErrStateNotFound = errors.New("database: state not found")
	// ErrOutputNotFound returned on output lookup during tx verification.
	ErrOutputNotFound = errors.New("database: output not found")
	// ErrStateHashNotFound returned on state hash not linked to any block.
	ErrStateHashNotFound = errors.New("database: state hash was not found")

	// AnyTxType is used as a filter value on FetchBlockTxByHash.
	AnyTxType = transactions.TxType(math.MaxUint8)
)

Functions

func Drivers

func Drivers() []string

Drivers returns a sorted list of the names of the registered drivers.

func Register

func Register(driver Driver) error

Register makes a database driver available by the provided name. If Register is called twice with the same name or if driver is nil, it returns error.

Types

type DB

type DB interface {
	// View provides a managed execution of a read-only transaction.
	View(fn func(t Transaction) error) error

	// Update provides a managed execution of a read-write atomic transaction.
	//
	// An atomic transaction is an indivisible and irreducible series of
	// database operations such that either all occur, or nothing occurs.
	//
	// Transaction commit will happen only if no error is returned by `fn`
	// and no panic is raised on `fn` execution.
	Update(fn func(t Transaction) error) error

	Close() error
}

DB is a thin layer on top of Transaction providing a manageable execution.

type Driver

type Driver interface {
	// Open returns a new connection to a blockchain database. The path is a
	// string in a driver-specific format.
	Open(path string, readonly bool) (DB, error)

	// Close terminates all DB connections and closes underlying storage.
	Close() error

	// Name returns a unique identifier that can be used to register the driver.
	Name() string
}

A Driver represents an application programming interface for accessing blockchain database management systems.

It is conceptually similar to ODBC for DBMS.

func From

func From(name string) (Driver, error)

From returns a registered Driver by name.

type Registry

type Registry struct {
	TipHash       []byte
	PersistedHash []byte
}

Registry represents a set database records that provide chain metadata.

type Transaction

type Transaction interface {
	FetchBlockHeader(hash []byte) (*block.Header, error)
	// Fetch all of the Txs that belong to a block with this header.hash.
	FetchBlockTxs(hash []byte) ([]transactions.ContractCall, error)
	// Fetch tx by txID. If succeeds, it returns tx data, tx index and
	// hash of the block it belongs to.
	FetchBlockTxByHash(txID []byte) (tx transactions.ContractCall, txIndex uint32, blockHeaderHash []byte, err error)
	FetchBlockHashByHeight(height uint64) ([]byte, error)
	FetchBlockExists(hash []byte) (bool, error)
	FetchBlockByStateRoot(fromHeight uint64, stateRoot []byte) (*block.Block, error)

	// Fetch chain registry (chain tip hash, persisted block etc).
	FetchRegistry() (*Registry, error)

	// Read-write transactions
	// Store the next chain block in a append-only manner
	// Overwrites only if block with same hash already stored
	// Not to be called concurrently, as it updates chain tip.
	StoreBlock(block *block.Block, persisted bool) error

	// DeleteBlock deletes all records associated with a specified block.
	DeleteBlock(b *block.Block) error

	// FetchBlock will return a block, given a hash.
	FetchBlock(hash []byte) (*block.Block, error)

	// FetchCurrentHeight returns the height of the most recently stored
	// block in the database.
	FetchCurrentHeight() (uint64, error)

	// FetchBlockHeightSince try to find height of a block generated around
	// sinceUnixTime starting the search from height (tip - offset).
	FetchBlockHeightSince(sinceUnixTime int64, offset uint64) (uint64, error)

	// StoreCandidateMessage will...
	StoreCandidateMessage(cm block.Block) error

	FetchCandidateMessage(hash []byte) (block.Block, error)

	ClearCandidateMessages() error

	// ClearDatabase will remove all information from the database.
	ClearDatabase() error

	// Atomic storage.
	Commit() error
	Rollback() error
	Close()
}

Transaction represents transaction layer. Transaction should provide basic transactions to fetch and store blockchain data.

To simplify code reading with database pkg we should use 'Tx' to refer to blockchain transaction and 'Transaction' to refer to database transaction.