db

package module
v1.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 19, 2024 License: Apache-2.0 Imports: 18 Imported by: 1,144

README

Cosmos DB

version license API Reference codecov Lint Test Discord chat

Common database interface for various database backends. Primarily meant for applications built on Tendermint, such as the Cosmos SDK, but can be used independently of these as well.

Minimum Go Version

Go 1.19+

Supported Database Backends

  • MemDB [stable]: An in-memory database using Google's B-tree package. Has very high performance both for reads, writes, and range scans, but is not durable and will lose all data on process exit. Does not support transactions. Suitable for e.g. caches, working sets, and tests. Used for IAVL working sets when the pruning strategy allows it.

  • GoLevelDB: a pure Go implementation of LevelDB (see below). Currently the default on-disk database used in the Cosmos SDK.

  • LevelDB using levigo Go wrapper. Uses LSM-trees for on-disk storage, which have good performance for write-heavy workloads, particularly on spinning disks, but requires periodic compaction to maintain decent read performance and reclaim disk space. Does not support transactions.

  • RocksDB: A Go wrapper around RocksDB. Similarly to LevelDB (above) it uses LSM-trees for on-disk storage, but is optimized for fast storage media such as SSDs and memory. Supports atomic transactions, but not full ACID transactions.

  • Pebble: a RocksDB/LevelDB inspired key-value database in Go using RocksDB file format and LSM-trees for on-disk storage. Supports snapshots.

Meta-databases

  • PrefixDB [stable]: A database which wraps another database and uses a static prefix for all keys. This allows multiple logical databases to be stored in a common underlying databases by using different namespaces. Used by the Cosmos SDK to give different modules their own namespaced database in a single application database.

Tests

To test common databases, run make test. If all databases are available on the local machine, use make test-all to test them all.

Documentation

Index

Constants

View Source 
const DBFileSuffix = ".db"

Variables

View Source 
var (
	ForceSync = "0"
)

ForceSync

This is set at compile time. Could be 0 or 1, defaults is 0. It will force using Sync for NoSync functions (Set, Delete, Write)

Used as a workaround for chain-upgrade issue: At the upgrade-block, the sdk will panic without flushing data to disk or closing dbs properly.

Upgrade guide:

  1. After seeing `UPGRADE "xxxx" NEED at height....`, restart current version with `-X github.com/tendermint/tm-db.ForceSync=1`
  2. Restart new version as normal

Example: Upgrading sifchain from v0.14.0 to v0.15.0

# log: panic: UPGRADE "0.15.0" NEEDED at height: 8170210: {"binaries":{"linux/amd64":"https://github.com/Sifchain/sifnode/releases/download/v0.15.0/sifnoded-v0.15.0-linux-amd64.zip?checksum=0c03b5846c5a13dcc0d9d3127e4f0cee0aeddcf2165177b2f2e0d60dbcf1a5ea"}}

# step1 git reset --hard git checkout v0.14.0 go mod edit -replace github.com/tendermint/tm-db=github.com/baabeetaa/tm-db@pebble go mod tidy go install -ldflags "-w -s -X github.com/cosmos/cosmos-sdk/types.DBBackend=pebbledb -X github.com/tendermint/tm-db.ForceSync=1" ./cmd/sifnoded

$HOME/go/bin/sifnoded start --db_backend=pebbledb

# step 2 git reset --hard git checkout v0.15.0 go mod edit -replace github.com/tendermint/tm-db=github.com/baabeetaa/tm-db@pebble go mod tidy go install -ldflags "-w -s -X github.com/cosmos/cosmos-sdk/types.DBBackend=pebbledb" ./cmd/sifnoded

$HOME/go/bin/sifnoded start --db_backend=pebbledb

Functions

func FileExists 

func FileExists(filePath string) bool

func IsKeyInDomain 

func IsKeyInDomain(key, start, end []byte) bool

See DB interface documentation for more information.

Types

type BackendType 

type BackendType string
const (
	// GoLevelDBBackend represents goleveldb (github.com/syndtr/goleveldb - most
	// popular implementation)
	//   - pure go
	//   - stable
	GoLevelDBBackend BackendType = "goleveldb"
	// MemDBBackend represents in-memory key value store, which is mostly used
	// for testing.
	MemDBBackend BackendType = "memdb"
	// PebbleDBDBBackend represents pebble (uses github.com/cockroachdb/pebble)
	//   - pure go
	PebbleDBBackend BackendType = "pebbledb"
	// RocksDBBackend represents rocksdb (uses github.com/linxGnu/grocksdb)
	//   - requires gcc
	//   - use rocksdb build tag (go build -tags rocksdb)
	RocksDBBackend BackendType = "rocksdb"
)

These are valid backend types.

type Batch 

type Batch = interface {
	// Set sets a key/value pair.
	// CONTRACT: key, value readonly []byte
	Set(key, value []byte) error

	// Delete deletes a key/value pair.
	// CONTRACT: key readonly []byte
	Delete(key []byte) error

	// Write writes the batch, possibly without flushing to disk. Only Close() can be called after,
	// other methods will error.
	Write() error

	// WriteSync writes the batch and flushes it to disk. Only Close() can be called after, other
	// methods will error.
	WriteSync() error

	// Close closes the batch. It is idempotent, but calls to other methods afterwards will error.
	Close() error

	// GetByteSize that returns the current size of the batch in bytes. Depending on the implementation,
	// this may return the size of the underlying LSM batch, including the size of additional metadata
	// on top of the expected key and value total byte count.
	GetByteSize() (int, error)
}

Batch represents a group of writes. They may or may not be written atomically depending on the backend. Callers must call Close on the batch when done.

As with DB, given keys and values should be considered read-only, and must not be modified after passing them to the batch.

type DB 

type DB interface {
	// Get fetches the value of the given key, or nil if it does not exist.
	// CONTRACT: key, value readonly []byte
	Get([]byte) ([]byte, error)

	// Has checks if a key exists.
	// CONTRACT: key, value readonly []byte
	Has(key []byte) (bool, error)

	// Set sets the value for the given key, replacing it if it already exists.
	// CONTRACT: key, value readonly []byte
	Set([]byte, []byte) error

	// SetSync sets the value for the given key, and flushes it to storage before returning.
	SetSync([]byte, []byte) error

	// Delete deletes the key, or does nothing if the key does not exist.
	// CONTRACT: key readonly []byte
	Delete([]byte) error

	// DeleteSync deletes the key, and flushes the delete to storage before returning.
	DeleteSync([]byte) error

	// Iterator returns an iterator over a domain of keys, in ascending order. The caller must call
	// Close when done. End is exclusive, and start must be less than end. A nil start iterates
	// from the first key, and a nil end iterates to the last key (inclusive). Empty keys are not
	// valid.
	// CONTRACT: No writes may happen within a domain while an iterator exists over it.
	// CONTRACT: start, end readonly []byte
	Iterator(start, end []byte) (Iterator, error)

	// ReverseIterator returns an iterator over a domain of keys, in descending order. The caller
	// must call Close when done. End is exclusive, and start must be less than end. A nil end
	// iterates from the last key (inclusive), and a nil start iterates to the first key (inclusive).
	// Empty keys are not valid.
	// CONTRACT: No writes may happen within a domain while an iterator exists over it.
	// CONTRACT: start, end readonly []byte
	ReverseIterator(start, end []byte) (Iterator, error)

	// Close closes the database connection.
	Close() error

	// NewBatch creates a batch for atomic updates. The caller must call Batch.Close.
	NewBatch() Batch

	// NewBatchWithSize create a new batch for atomic updates, but with pre-allocated size.
	// This will does the same thing as NewBatch if the batch implementation doesn't support pre-allocation.
	NewBatchWithSize(int) Batch

	// Print is used for debugging.
	Print() error

	// Stats returns a map of property values for all keys and the size of the cache.
	Stats() map[string]string
}

DB is the main interface for all database backends. DBs are concurrency-safe. Callers must call Close on the database when done.

Keys cannot be nil or empty, while values cannot be nil. Keys and values should be considered read-only, both when returned and when given, and must be copied before they are modified.

func NewDB 

func NewDB(name string, backend BackendType, dir string) (DB, error)

NewDB creates a new database of type backend with the given name.

func NewDBwithOptions 

func NewDBwithOptions(name string, backend BackendType, dir string, opts Options) (DB, error)

func NewPebbleDB added in v1.1.0 

func NewPebbleDB(name string, dir string, opts Options) (DB, error)

type GoLevelDB 

type GoLevelDB struct {
	// contains filtered or unexported fields
}

func NewGoLevelDB 

func NewGoLevelDB(name string, dir string, opts Options) (*GoLevelDB, error)

func NewGoLevelDBWithOpts 

func NewGoLevelDBWithOpts(name string, dir string, o *opt.Options) (*GoLevelDB, error)

func (*GoLevelDB) Close 

func (db *GoLevelDB) Close() error

Close implements DB.

func (*GoLevelDB) DB 

func (db *GoLevelDB) DB() *leveldb.DB

func (*GoLevelDB) Delete 

func (db *GoLevelDB) Delete(key []byte) error

Delete implements DB.

func (*GoLevelDB) DeleteSync 

func (db *GoLevelDB) DeleteSync(key []byte) error

DeleteSync implements DB.

func (*GoLevelDB) ForceCompact 

func (db *GoLevelDB) ForceCompact(start, limit []byte) error

func (*GoLevelDB) Get 

func (db *GoLevelDB) Get(key []byte) ([]byte, error)

Get implements DB.

func (*GoLevelDB) Has 

func (db *GoLevelDB) Has(key []byte) (bool, error)

Has implements DB.

func (*GoLevelDB) Iterator 

func (db *GoLevelDB) Iterator(start, end []byte) (Iterator, error)

Iterator implements DB.

func (*GoLevelDB) NewBatch 

func (db *GoLevelDB) NewBatch() Batch

NewBatch implements DB.

func (*GoLevelDB) NewBatchWithSize 

func (db *GoLevelDB) NewBatchWithSize(size int) Batch

NewBatchWithSize implements DB.

func (*GoLevelDB) Print 

func (db *GoLevelDB) Print() error

Print implements DB.

func (*GoLevelDB) ReverseIterator 

func (db *GoLevelDB) ReverseIterator(start, end []byte) (Iterator, error)

ReverseIterator implements DB.

func (*GoLevelDB) Set 

func (db *GoLevelDB) Set(key []byte, value []byte) error

Set implements DB.

func (*GoLevelDB) SetSync 

func (db *GoLevelDB) SetSync(key []byte, value []byte) error

SetSync implements DB.

func (*GoLevelDB) Stats 

func (db *GoLevelDB) Stats() map[string]string

Stats implements DB.

type Iterator 

type Iterator = interface {
	// Domain returns the start (inclusive) and end (exclusive) limits of the iterator.
	// CONTRACT: start, end readonly []byte
	Domain() (start []byte, end []byte)

	// Valid returns whether the current iterator is valid. Once invalid, the Iterator remains
	// invalid forever.
	Valid() bool

	// Next moves the iterator to the next key in the database, as defined by order of iteration.
	// If Valid returns false, this method will panic.
	Next()

	// Key returns the key at the current position. Panics if the iterator is invalid.
	// CONTRACT: key readonly []byte
	Key() (key []byte)

	// Value returns the value at the current position. Panics if the iterator is invalid.
	// CONTRACT: value readonly []byte
	Value() (value []byte)

	// Error returns the last error encountered by the iterator, if any.
	Error() error

	// Close closes the iterator, relasing any allocated resources.
	Close() error
}

Iterator represents an iterator over a domain of keys. Callers must call Close when done. No writes can happen to a domain while there exists an iterator over it, some backends may take out database locks to ensure this will not happen.

Callers must make sure the iterator is valid before calling any methods on it, otherwise these methods will panic. This is in part caused by most backend databases using this convention.

As with DB, keys and values should be considered read-only, and must be copied before they are modified.

Typical usage:

var itr Iterator = ... defer itr.Close() 

for ; itr.Valid(); itr.Next() {
  k, v := itr.Key(); itr.Value()
  ...
}

if err := itr.Error(); err != nil {
  ...
}

func IteratePrefix 

func IteratePrefix(db DB, prefix []byte) (Iterator, error)

IteratePrefix is a convenience function for iterating over a key domain restricted by prefix.

type MemDB 

type MemDB struct {
	// contains filtered or unexported fields
}

MemDB is an in-memory database backend using a B-tree for storage.

For performance reasons, all given and returned keys and values are pointers to the in-memory database, so modifying them will cause the stored values to be modified as well. All DB methods already specify that keys and values should be considered read-only, but this is especially important with MemDB.

func NewMemDB 

func NewMemDB() *MemDB

NewMemDB creates a new in-memory database.

func (*MemDB) Close 

func (db *MemDB) Close() error

Close implements DB.

func (*MemDB) Delete 

func (db *MemDB) Delete(key []byte) error

Delete implements DB.

func (*MemDB) DeleteSync 

func (db *MemDB) DeleteSync(key []byte) error

DeleteSync implements DB.

func (*MemDB) Get 

func (db *MemDB) Get(key []byte) ([]byte, error)

Get implements DB.

func (*MemDB) Has 

func (db *MemDB) Has(key []byte) (bool, error)

Has implements DB.

func (*MemDB) Iterator 

func (db *MemDB) Iterator(start, end []byte) (Iterator, error)

Iterator implements DB. Takes out a read-lock on the database until the iterator is closed.

func (*MemDB) IteratorNoMtx 

func (db *MemDB) IteratorNoMtx(start, end []byte) (Iterator, error)

IteratorNoMtx makes an iterator with no mutex.

func (*MemDB) NewBatch 

func (db *MemDB) NewBatch() Batch

NewBatch implements DB.

func (*MemDB) NewBatchWithSize 

func (db *MemDB) NewBatchWithSize(_ int) Batch

NewBatchWithSize implements DB. It does the same thing as NewBatch because we can't pre-allocate memDBBatch

func (*MemDB) Print 

func (db *MemDB) Print() error

Print implements DB.

func (*MemDB) ReverseIterator 

func (db *MemDB) ReverseIterator(start, end []byte) (Iterator, error)

ReverseIterator implements DB. Takes out a read-lock on the database until the iterator is closed.

func (*MemDB) ReverseIteratorNoMtx 

func (db *MemDB) ReverseIteratorNoMtx(start, end []byte) (Iterator, error)

ReverseIteratorNoMtx makes an iterator with no mutex.

func (*MemDB) Set 

func (db *MemDB) Set(key []byte, value []byte) error

Set implements DB.

func (*MemDB) SetSync 

func (db *MemDB) SetSync(key []byte, value []byte) error

SetSync implements DB.

func (*MemDB) Stats 

func (db *MemDB) Stats() map[string]string

Stats implements DB.

type Options 

type Options interface {
	Get(string) interface{}
}

type OptionsMap 

type OptionsMap map[string]interface{}

OptionsMap is a stub implementing Options which can get data from a map

func (OptionsMap) Get 

func (m OptionsMap) Get(key string) interface{}

type PebbleDB added in v1.1.0 

type PebbleDB struct {
	// contains filtered or unexported fields
}

PebbleDB is a PebbleDB backend.

func (PebbleDB) Close added in v1.1.0 

func (db PebbleDB) Close() error

Close implements DB.

func (*PebbleDB) DB added in v1.1.0 

func (db *PebbleDB) DB() *pebble.DB

func (*PebbleDB) Delete added in v1.1.0 

func (db *PebbleDB) Delete(key []byte) error

Delete implements DB.

func (PebbleDB) DeleteSync added in v1.1.0 

func (db PebbleDB) DeleteSync(key []byte) error

DeleteSync implements DB.

func (*PebbleDB) Get added in v1.1.0 

func (db *PebbleDB) Get(key []byte) ([]byte, error)

Get implements DB.

func (*PebbleDB) Has added in v1.1.0 

func (db *PebbleDB) Has(key []byte) (bool, error)

Has implements DB.

func (*PebbleDB) Iterator added in v1.1.0 

func (db *PebbleDB) Iterator(start, end []byte) (Iterator, error)

Iterator implements DB.

func (*PebbleDB) NewBatch added in v1.1.0 

func (db *PebbleDB) NewBatch() Batch

NewBatch implements DB.

func (*PebbleDB) NewBatchWithSize added in v1.1.0 

func (db *PebbleDB) NewBatchWithSize(size int) Batch

NewBatchWithSize implements DB. It does the same thing as NewBatch because we can't pre-allocate pebbleDBBatch

func (*PebbleDB) Print added in v1.1.0 

func (db *PebbleDB) Print() error

Print implements DB.

func (*PebbleDB) ReverseIterator added in v1.1.0 

func (db *PebbleDB) ReverseIterator(start, end []byte) (Iterator, error)

ReverseIterator implements DB.

func (*PebbleDB) Set added in v1.1.0 

func (db *PebbleDB) Set(key []byte, value []byte) error

Set implements DB.

func (*PebbleDB) SetSync added in v1.1.0 

func (db *PebbleDB) SetSync(key []byte, value []byte) error

SetSync implements DB.

func (*PebbleDB) Stats added in v1.1.0 

func (db *PebbleDB) Stats() map[string]string

Stats implements DB.

type PrefixDB 

type PrefixDB struct {
	// contains filtered or unexported fields
}

PrefixDB wraps a namespace of another database as a logical database.

func NewPrefixDB 

func NewPrefixDB(db DB, prefix []byte) *PrefixDB

NewPrefixDB lets you namespace multiple DBs within a single DB.

func (*PrefixDB) Close 

func (pdb *PrefixDB) Close() error

Close implements DB.

func (*PrefixDB) Delete 

func (pdb *PrefixDB) Delete(key []byte) error

Delete implements DB.

func (*PrefixDB) DeleteSync 

func (pdb *PrefixDB) DeleteSync(key []byte) error

DeleteSync implements DB.

func (*PrefixDB) Get 

func (pdb *PrefixDB) Get(key []byte) ([]byte, error)

Get implements DB.

func (*PrefixDB) Has 

func (pdb *PrefixDB) Has(key []byte) (bool, error)

Has implements DB.

func (*PrefixDB) Iterator 

func (pdb *PrefixDB) Iterator(start, end []byte) (Iterator, error)

Iterator implements DB.

func (*PrefixDB) NewBatch 

func (pdb *PrefixDB) NewBatch() Batch

NewBatch implements DB.

func (*PrefixDB) NewBatchWithSize 

func (pdb *PrefixDB) NewBatchWithSize(size int) Batch

NewBatchWithSize implements DB.

func (*PrefixDB) Print 

func (pdb *PrefixDB) Print() error

Print implements DB.

func (*PrefixDB) ReverseIterator 

func (pdb *PrefixDB) ReverseIterator(start, end []byte) (Iterator, error)

ReverseIterator implements DB.

func (*PrefixDB) Set 

func (pdb *PrefixDB) Set(key []byte, value []byte) error

Set implements DB.

func (*PrefixDB) SetSync 

func (pdb *PrefixDB) SetSync(key []byte, value []byte) error

SetSync implements DB.

func (*PrefixDB) Stats 

func (pdb *PrefixDB) Stats() map[string]string

Stats implements DB.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.