ethdb

package
v0.0.0-...-215623d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 20, 2021 License: GPL-3.0 Imports: 35 Imported by: 0

README

Ethdb package hold's bouquet of objects to access DB

Words "KV" and "DB" have special meaning here:

  • KV - key-value-style API to access data: let developer manage transactions, stateful cursors.
  • DB - object-oriented-style API to access data: Get/Put/Delete/WalkOverTable/MultiPut, managing transactions internally.

So, DB abstraction fits 95% times and leads to more maintainable code - because it's looks stateless.

About "key-value-style": Modern key-value databases don't provide Get/Put/Delete methods, because it's very hard-drive-unfriendly - it pushes developers do random-disk-access which is order of magnitude slower than sequential read. To enforce sequential-reads - introduced stateful cursors/iterators - they intentionally look as file-api: open_cursor/seek/write_data_from_current_position/move_to_end/step_back/step_forward/delete_key_on_current_position/append.

Class diagram:

// This is not call graph, just show classes from low-level to high-level. 
// And show which classes satisfy which interfaces.

+-----------------------------------+   +-----------------------------------+   +-----------------------------------+ 
|  github.com/ledgerwatch/lmdb-go   |   |  github.com/torquem-ch/mdbx-go    |   | google.golang.org/grpc.ClientConn |                    
|  (app-agnostic LMDB go bindings)  |   |  (app-agnostic MDBX go bindings)  |   | (app-agnostic RPC and streaming)  |
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
                 |                                        |                                      |
                 |                                        |                                      |
                 v                                        v                                      v
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
|      ethdb/kv_lmdb.go             |   |       ethdb/kv_mdbx.go            |   |       ethdb/kv_remote.go          |                
| (tg-specific LMDB implementaion)  |   |  (tg-specific MDBX implementaion) |   |   (tg-specific remote DB access)  |              
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
                 |                                        |                                      |
                 |                                        |                                      |
                 v                                        v                                      v
            +----------------------------------------------------------------------------------------------+
            |                                       ethdb/kv_abstract.go                                   |  
            |         (Common KV interface. DB-friendly, disk-friendly, cpu-cache-friendly.                |
            |           Same app code can work with local or remote database.                              |
            |           Allows experiment with another database implementations.                           |
            |          Supports context.Context for cancelation. Any operation can return error)           |
            +----------------------------------------------------------------------------------------------+
                 |                                        |                                      |
                 |                                        |                                      |
                 v                                        v                                      v
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
|       ethdb/object_db.go          |   |          ethdb/tx_db.go           |   |    ethdb/remote/remotedbserver    |                
|     (thread-safe, stateless,      |   | (non-thread-safe, more performant |   | (grpc server, using kv_abstract,  |  
|   opens/close short transactions  |   |   than object_db, method Begin    |   |   kv_remote call this server, 1   |
|      internally when need)        |   |  DOESN'T create new TxDb object)  |   | transaction maps on 1 grpc stream |
+-----------------------------------+   +-----------------------------------+   +-----------------------------------+
                |                                          |                                     
                |                                          |                                     
                v                                          v                                     
            +-----------------------------------------------------------------------------------------------+
            |                                    ethdb/interface.go                                         |  
            |     (Common DB interfaces. ethdb.Database and ethdb.DbWithPendingMutations are widely used)   |
            +-----------------------------------------------------------------------------------------------+
                |                      
                |                      
                v                      
+--------------------------------------------------+ 
|             ethdb/mutation.go                    |                 
| (also known as "batch", recording all writes and |  
|   them flush to DB in sorted way only when call  | 
|     .Commit(), use it to avoid random-writes.    | 
|   It use and satisfy ethdb.Database in same time |
+--------------------------------------------------+

ethdb.AbstractKV design:

  • InMemory, ReadOnly: NewLMDB().Flags(lmdb.ReadOnly).InMem().Open()

  • MultipleDatabases, Customization: NewLMDB().Path(path).WithBucketsConfig(config).Open()

  • 1 Transaction object can be used only withing 1 goroutine.

  • Only 1 write transaction can be active at a time (other will wait).

  • Unlimited read transactions can be active concurrently (not blocked by write transaction).

  • Methods db.Update, db.View - can be used to open and close short transaction.

  • Methods Begin/Commit/Rollback - for long transaction.

  • it's safe to call .Rollback() after .Commit(), multiple rollbacks are also safe. Common transaction patter:

tx, err := db.Begin(true, ethdb.RW)
if err != nil {
    return err
}
defer tx.Rollback() // important to avoid transactions leak at panic or early return

// ... code which uses database in transaction
 
err := tx.Commit()
if err != nil {
    return err
}

  • No internal copies/allocations. It means: 1. app must copy keys/values before put to database. 2. Data after read from db - valid only during current transaction - copy it if plan use data after transaction Commit/Rollback.

  • Methods .Bucket() and .Cursor(), can’t return nil, can't return error.

  • Bucket and Cursor - are interfaces - means different classes can satisfy it: for example LmdbCursor and LmdbDupSortCursor classes satisfy it. If your are not familiar with "DupSort" concept, please read indices.md first.

  • If Cursor returns err!=nil then key SHOULD be != nil (can be []byte{} for example). Then traversal code look as:

for k, v, err := c.First(); k != nil; k, v, err = c.Next() {
    if err != nil {
        return err
    }
    // logic
}
  • Move cursor: cursor.Seek(key)

ethdb.Database design:

  • Allows pass multiple implementations
  • Allows traversal tables by db.Walk

ethdb.TxDb design:

  • holds inside 1 long-running transaction and 1 cursor per table
  • method Begin DOESN'T create new TxDb object, it means this object can be passed into other objects by pointer, and high-level app code can start/commit transactions when it needs without re-creating all objects which holds TxDb pointer.
  • This is reason why txDb.CommitAndBegin() method works: inside it creating new transaction object, pinter to TxDb stays valid.

How to dump/load table

Install all database tools: make db-tools - tools with prefix mdb_ is for lmdb, lmdbgo_ is for lmdb written in go, mdbx_ is for mdbx.

./build/bin/mdbx_dump -a <datadir>/tg/chaindata | lz4 > dump.lz4
lz4 -d < dump.lz4 | ./build/bin/mdbx_load -an <datadir>/tg/chaindata

How to get table checksum

./build/bin/mdbx_dump -s table_name <datadir>/tg/chaindata | tail -n +4 | sha256sum # tail here is for excluding header 

Header example:
VERSION=3
geometry=l268435456,c268435456,u25769803776,s268435456,g268435456
mapsize=756375552
maxreaders=120
format=bytevalue
database=TBL0001
type=btree
db_pagesize=4096
duplicates=1
dupsort=1
HEADER=END

Documentation

Overview

Package ethdb defines the interfaces for an Ethereum data store.

Index

Constants

View Source 
const (
	NonExistingDBI dbutils.DBI = 999_999_999
)
View Source 
const ReadersLimit = 2000 // MDBX_READERS_LIMIT on 64bit system

Variables

View Source 
var (
	ErrAttemptToDeleteNonDeprecatedBucket = errors.New("only buckets from dbutils.DeprecatedBuckets can be deleted")
	ErrUnknownBucket                      = errors.New("unknown bucket. add it to dbutils.Buckets")
)
View Source 
var (
	LMDBDefaultMapSize          = 2 * datasize.TB
	LMDBDefaultMaxFreelistReuse = uint(1000) // measured in pages
)
View Source 
var DefaultStorageMode = StorageMode{Initialised: true, Pruning: false, History: true, Receipts: true, TxIndex: true, CallTraces: false}
View Source 
var DeletedValue = []byte{0}
View Source 
var EndSuffix = []byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}
View Source 
var ErrKeyNotFound = errors.New("db: key not found")

ErrKeyNotFound is returned when key isn't found in the database.

View Source 
var ErrUnavailableSnapshot = errors.New("unavailable snapshot")

Functions

func Bytesmask 

func Bytesmask(fixedbits int) (fixedbytes int, mask byte)

func CustomCmpFunc 

func CustomCmpFunc(tx *lmdb.Txn, dbi lmdb.DBI) dbutils.CmpFunc

func CustomDupCmpFunc 

func CustomDupCmpFunc(tx *lmdb.Txn, dbi lmdb.DBI) dbutils.CmpFunc

func DefaultBucketConfigs 

func DefaultBucketConfigs(defaultBuckets dbutils.BucketsCfg) dbutils.BucketsCfg

func ForEach 

func ForEach(c Cursor, walker func(k, v []byte) (bool, error)) error

func Get 

func Get(tx Tx, bucket string, key []byte) ([]byte, error)

func InspectDatabase 

func InspectDatabase(db Database) error

func KeyCmpBackward 

func KeyCmpBackward(key1, key2 []byte) (int, bool)

func MultiPut 

func MultiPut(tx RwTx, tuples ...[]byte) error

func NewBatch 

func NewBatch(tx RwTx) *mutation

func NewRemote 

func NewRemote(versionMajor, versionMinor, versionPatch uint32) remoteOpts

NewRemote defines new remove KV connection (without actually opening it) version parameters represent the version the KV client is expecting, compatibility check will be performed when the KV connection opens

func NewRoTxDb 

func NewRoTxDb(tx Tx) *roTxDb

func NewSnapshotKV 

func NewSnapshotKV() snapshotOpts

func NewSplitCursor 

func NewSplitCursor(c Cursor, startkey []byte, matchBits int, part1end, part2start, part3start int) *splitCursor

func NewTestTx 

func NewTestTx(t testing.TB) (RwKV, RwTx)

func OverrideStorageMode 

func OverrideStorageMode(db Database, sm StorageMode) error

func SetStorageModeIfNotExist 

func SetStorageModeIfNotExist(db Database, sm StorageMode) error

func Walk 

func Walk(c Cursor, startkey []byte, fixedbits int, walker func(k, v []byte) (bool, error)) error

Types

type BucketConfigsFunc 

type BucketConfigsFunc func(defaultBuckets dbutils.BucketsCfg) dbutils.BucketsCfg

type BucketMigrator 

type BucketMigrator interface {
	DropBucket(string) error
	CreateBucket(string) error
	ExistsBucket(string) bool
	ClearBucket(string) error
	ExistingBuckets() ([]string, error)
}

BucketMigrator used for buckets migration, don't use it in usual app code

type BucketsMigrator 

type BucketsMigrator interface {
	BucketExists(bucket string) (bool, error) // makes them empty
	ClearBuckets(buckets ...string) error     // makes them empty
	DropBuckets(buckets ...string) error      // drops them, use of them after drop will panic
}

type Closer 

type Closer interface {
	Close()
}

type Cursor 

type Cursor interface {
	First() ([]byte, []byte, error)               // First - position at first key/data item
	Seek(seek []byte) ([]byte, []byte, error)     // Seek - position at first key greater than or equal to specified key
	SeekExact(key []byte) ([]byte, []byte, error) // SeekExact - position at first key greater than or equal to specified key
	Next() ([]byte, []byte, error)                // Next - position at next key/value (can iterate over DupSort key/values automatically)
	Prev() ([]byte, []byte, error)                // Prev - position at previous key
	Last() ([]byte, []byte, error)                // Last - position at last key and last possible value
	Current() ([]byte, []byte, error)             // Current - return key/data at current cursor position

	Count() (uint64, error) // Count - fast way to calculate amount of keys in bucket. It counts all keys even if Prefix was set.

	Close()
}

Cursor - class for navigating through a database CursorDupSort are inherit this class

If methods (like First/Next/Seek) return error, then returned key SHOULD not be nil (can be []byte{} for example). Then looping code will look as: c := kv.Cursor(bucketName) 

for k, v, err := c.First(); k != nil; k, v, err = c.Next() {
   if err != nil {
       return err
   }
   ... logic
}

type CursorDupSort 

type CursorDupSort interface {
	Cursor

	// SeekBothExact -
	// second parameter can be nil only if searched key has no duplicates, or return error
	SeekBothExact(key, value []byte) ([]byte, []byte, error)
	SeekBothRange(key, value []byte) ([]byte, error)
	FirstDup() ([]byte, error)          // FirstDup - position at first data item of current key
	NextDup() ([]byte, []byte, error)   // NextDup - position at next data item of current key
	NextNoDup() ([]byte, []byte, error) // NextNoDup - position at first data item of next key
	LastDup() ([]byte, error)           // LastDup - position at last data item of current key

	CountDuplicates() (uint64, error) // CountDuplicates - number of duplicates for the current key
}

type DBTX 

type DBTX interface {
	DBTX() RwTx
}

type DBVerbosityLvl 

type DBVerbosityLvl int8

type Database 

type Database interface {
	GetterBeginner
	Putter
	Deleter
	Closer

	// MultiPut inserts or updates multiple entries.
	// Entries are passed as an array:
	// bucket0, key0, val0, bucket1, key1, val1, ...
	MultiPut(tuples ...[]byte) (uint64, error)

	// NewBatch - starts in-mem batch
	//
	// Common pattern:
	//
	// batch := db.NewBatch()
	// defer batch.Rollback()
	// ... some calculations on `batch`
	// batch.Commit()
	//
	NewBatch() DbWithPendingMutations                                         //
	Begin(ctx context.Context, flags TxFlags) (DbWithPendingMutations, error) // starts db transaction
	Last(bucket string) ([]byte, []byte, error)

	Keys() ([][]byte, error)

	Append(bucket string, key, value []byte) error
	AppendDup(bucket string, key, value []byte) error
	IncrementSequence(bucket string, amount uint64) (uint64, error)
	ReadSequence(bucket string) (uint64, error)
	RwKV() RwKV
}

Database wraps all database operations. All methods are safe for concurrent use.

type DatabaseReader 

type DatabaseReader interface {
	KVGetter

	// Get returns the value for a given key if it's present.
	Get(bucket string, key []byte) ([]byte, error)
}

type DbCopier 

type DbCopier interface {
	NewDbWithTheSameParameters() *ObjectDatabase
}

type DbWithPendingMutations 

type DbWithPendingMutations interface {
	Database

	// Commit - commits transaction (or flush data into underlying db object in case of `mutation`)
	//
	// Common pattern:
	//
	// tx := db.Begin()
	// defer tx.Rollback()
	// ... some calculations on `tx`
	// tx.Commit()
	//
	Commit() error

	// CommitAndBegin - commits and starts new transaction inside same db object.
	// useful for periodical commits implementation.
	//
	// Common pattern:
	//
	// tx := db.Begin()
	// defer tx.Rollback()
	// for {
	// 		... some calculations on `tx`
	//       tx.CommitAndBegin()
	//       // defer here - is not useful, because 'tx' object is reused and first `defer` will work perfectly
	// }
	// tx.Commit()
	//
	CommitAndBegin(ctx context.Context) error
	RollbackAndBegin(ctx context.Context) error
	Rollback()
	BatchSize() int
}

DbWithPendingMutations is an extended version of the Database, where all changes are first made in memory. Later they can either be committed to the database or rolled back.

func NewTxDbWithoutTransaction 

func NewTxDbWithoutTransaction(db Database, flags TxFlags) DbWithPendingMutations

NewTxDbWithoutTransaction creates TxDb object without opening transaction, such TxDb not usable before .Begin() call on it It allows inject TxDb object into class hierarchy, but open write transaction later

type Deleter 

type Deleter interface {
	// Delete removes a single entry.
	Delete(bucket string, k, v []byte) error
}

Deleter wraps the database delete operations.

type Getter 

type Getter interface {
	DatabaseReader

	// Walk iterates over entries with keys greater or equal to startkey.
	// Only the keys whose first fixedbits match those of startkey are iterated over.
	// walker is called for each eligible entry.
	// If walker returns false or an error, the walk stops.
	Walk(bucket string, startkey []byte, fixedbits int, walker func(k, v []byte) (bool, error)) error
}

Getter wraps the database read operations.

type GetterBeginner 

type GetterBeginner interface {
	Getter

	BeginGetter(ctx context.Context) (GetterTx, error)
}

type GetterPutter 

type GetterPutter interface {
	Getter
	Putter
}

type GetterTx 

type GetterTx interface {
	Getter

	Rollback()
}

type Has 

type Has interface {
	// Has indicates whether a key exists in the database.
	Has(bucket string, key []byte) (bool, error)
}

type HasNetInterface 

type HasNetInterface interface {
	DB() Database
}

type HasRwKV 

type HasRwKV interface {
	RwKV() RwKV
	SetRwKV(kv RwKV)
}

type HasStats 

type HasStats interface {
	BucketSize(name string) (uint64, error)
	DiskSize(context.Context) (uint64, error) // db size
}

type HasTx 

type HasTx interface {
	Tx() Tx
}

type KVGetter 

type KVGetter interface {
	Has

	GetOne(bucket string, key []byte) (val []byte, err error)
}

type KvData 

type KvData struct {
	K []byte
	V []byte
}

type LmdbCursor 

type LmdbCursor struct {
	// contains filtered or unexported fields
}

func (*LmdbCursor) Append 

func (c *LmdbCursor) Append(k []byte, v []byte) error

Append - speedy feature of lmdb which is not part of KV interface. Cast your cursor to *LmdbCursor to use this method. Return error - if provided data will not sorted (or bucket have old records which mess with new in sorting manner).

func (*LmdbCursor) Close 

func (c *LmdbCursor) Close()

func (*LmdbCursor) Count 

func (c *LmdbCursor) Count() (uint64, error)

func (*LmdbCursor) Current 

func (c *LmdbCursor) Current() ([]byte, []byte, error)

Current - return key/data at current cursor position

func (*LmdbCursor) Delete 

func (c *LmdbCursor) Delete(k, v []byte) error

func (*LmdbCursor) DeleteCurrent 

func (c *LmdbCursor) DeleteCurrent() error

DeleteCurrent This function deletes the key/data pair to which the cursor refers. This does not invalidate the cursor, so operations such as MDB_NEXT can still be used on it. Both MDB_NEXT and MDB_GET_CURRENT will return the same record after this operation.

func (*LmdbCursor) First 

func (c *LmdbCursor) First() ([]byte, []byte, error)

func (*LmdbCursor) Last 

func (c *LmdbCursor) Last() ([]byte, []byte, error)

func (*LmdbCursor) Next 

func (c *LmdbCursor) Next() (k, v []byte, err error)

func (*LmdbCursor) Prefetch 

func (c *LmdbCursor) Prefetch(v uint) Cursor

func (*LmdbCursor) Prefix 

func (c *LmdbCursor) Prefix(v []byte) Cursor

func (*LmdbCursor) Prev 

func (c *LmdbCursor) Prev() (k, v []byte, err error)

func (*LmdbCursor) Put 

func (c *LmdbCursor) Put(key []byte, value []byte) error

func (*LmdbCursor) PutNoOverwrite 

func (c *LmdbCursor) PutNoOverwrite(key []byte, value []byte) error

func (*LmdbCursor) Seek 

func (c *LmdbCursor) Seek(seek []byte) (k, v []byte, err error)

func (*LmdbCursor) SeekExact 

func (c *LmdbCursor) SeekExact(key []byte) ([]byte, []byte, error)

type LmdbDupSortCursor 

type LmdbDupSortCursor struct {
	*LmdbCursor
}

func (*LmdbDupSortCursor) Append 

func (c *LmdbDupSortCursor) Append(k []byte, v []byte) error

func (*LmdbDupSortCursor) AppendDup 

func (c *LmdbDupSortCursor) AppendDup(k []byte, v []byte) error

func (*LmdbDupSortCursor) CountDuplicates 

func (c *LmdbDupSortCursor) CountDuplicates() (uint64, error)

Count returns the number of duplicates for the current key. See mdb_cursor_count

func (*LmdbDupSortCursor) DeleteCurrentDuplicates 

func (c *LmdbDupSortCursor) DeleteCurrentDuplicates() error

DeleteCurrentDuplicates - delete all of the data items for the current key.

func (*LmdbDupSortCursor) DeleteExact 

func (c *LmdbDupSortCursor) DeleteExact(k1, k2 []byte) error

DeleteExact - does delete

func (*LmdbDupSortCursor) FirstDup 

func (c *LmdbDupSortCursor) FirstDup() ([]byte, error)

func (*LmdbDupSortCursor) LastDup 

func (c *LmdbDupSortCursor) LastDup() ([]byte, error)

func (*LmdbDupSortCursor) NextDup 

func (c *LmdbDupSortCursor) NextDup() ([]byte, []byte, error)

NextDup - iterate only over duplicates of current key

func (*LmdbDupSortCursor) NextNoDup 

func (c *LmdbDupSortCursor) NextNoDup() ([]byte, []byte, error)

NextNoDup - iterate with skipping all duplicates

func (*LmdbDupSortCursor) PrevDup 

func (c *LmdbDupSortCursor) PrevDup() ([]byte, []byte, error)

func (*LmdbDupSortCursor) PrevNoDup 

func (c *LmdbDupSortCursor) PrevNoDup() ([]byte, []byte, error)

func (*LmdbDupSortCursor) PutNoDupData 

func (c *LmdbDupSortCursor) PutNoDupData(key, value []byte) error

func (*LmdbDupSortCursor) SeekBothExact 

func (c *LmdbDupSortCursor) SeekBothExact(key, value []byte) ([]byte, []byte, error)

func (*LmdbDupSortCursor) SeekBothRange 

func (c *LmdbDupSortCursor) SeekBothRange(key, value []byte) ([]byte, error)

type LmdbKV 

type LmdbKV struct {
	// contains filtered or unexported fields
}

func (*LmdbKV) AllBuckets 

func (db *LmdbKV) AllBuckets() dbutils.BucketsCfg

func (*LmdbKV) AllDBI 

func (db *LmdbKV) AllDBI() map[string]dbutils.DBI

func (*LmdbKV) BeginRo 

func (db *LmdbKV) BeginRo(_ context.Context) (txn Tx, err error)

func (*LmdbKV) BeginRw 

func (db *LmdbKV) BeginRw(_ context.Context) (txn RwTx, err error)

func (*LmdbKV) Close 

func (db *LmdbKV) Close()

Close closes db All transactions must be closed before closing the database.

func (*LmdbKV) CollectMetrics 

func (db *LmdbKV) CollectMetrics()

func (*LmdbKV) DiskSize 

func (db *LmdbKV) DiskSize(_ context.Context) (uint64, error)

func (*LmdbKV) Env 

func (db *LmdbKV) Env() *lmdb.Env

func (*LmdbKV) NewDbWithTheSameParameters 

func (db *LmdbKV) NewDbWithTheSameParameters() *ObjectDatabase

func (*LmdbKV) Update 

func (db *LmdbKV) Update(ctx context.Context, f func(tx RwTx) error) (err error)

func (*LmdbKV) View 

func (db *LmdbKV) View(ctx context.Context, f func(tx Tx) error) (err error)

type LmdbOpts 

type LmdbOpts struct {
	// contains filtered or unexported fields
}

func NewLMDB 

func NewLMDB() LmdbOpts

func NewMDBX 

func NewMDBX() LmdbOpts

func (LmdbOpts) DBVerbosity 

func (opts LmdbOpts) DBVerbosity(v DBVerbosityLvl) LmdbOpts

func (LmdbOpts) Exclusive 

func (opts LmdbOpts) Exclusive() LmdbOpts

func (LmdbOpts) Flags 

func (opts LmdbOpts) Flags(f func(uint) uint) LmdbOpts

func (LmdbOpts) InMem 

func (opts LmdbOpts) InMem() LmdbOpts

func (LmdbOpts) MapSize 

func (opts LmdbOpts) MapSize(sz datasize.ByteSize) LmdbOpts

func (LmdbOpts) MustOpen 

func (opts LmdbOpts) MustOpen() *LmdbKV

func (LmdbOpts) Open 

func (opts LmdbOpts) Open() (kv *LmdbKV, err error)

func (LmdbOpts) Path 

func (opts LmdbOpts) Path(path string) LmdbOpts

func (LmdbOpts) Readonly 

func (opts LmdbOpts) Readonly() LmdbOpts

func (LmdbOpts) Set 

func (opts LmdbOpts) Set(opt LmdbOpts) LmdbOpts

func (LmdbOpts) WithBucketsConfig 

func (opts LmdbOpts) WithBucketsConfig(f BucketConfigsFunc) LmdbOpts

type MinDatabase 

type MinDatabase interface {
	Get(bucket string, key []byte) ([]byte, error)
	Put(bucket string, key, value []byte) error
	Delete(bucket string, k, v []byte) error
}

MinDatabase is a minimalistic version of the Database interface.

type MultiPutTuples 

type MultiPutTuples [][]byte

Type which expecting sequence of triplets: dbi, key, value, .... It sorts entries by dbi name, then inside dbi clusters sort by keys

func (MultiPutTuples) Len 

func (t MultiPutTuples) Len() int

func (MultiPutTuples) Less 

func (t MultiPutTuples) Less(i, j int) bool

func (MultiPutTuples) Swap 

func (t MultiPutTuples) Swap(i, j int)

type MutationItem 

type MutationItem struct {
	// contains filtered or unexported fields
}

func (*MutationItem) Less 

func (mi *MutationItem) Less(than btree.Item) bool

type ObjectDatabase 

type ObjectDatabase struct {
	// contains filtered or unexported fields
}

ObjectDatabase - is an object-style interface of DB accessing

func MustOpen 

func MustOpen(path string) *ObjectDatabase

func NewDatabaseWithFreezer 

func NewDatabaseWithFreezer(db *ObjectDatabase, dir, suffix string) (*ObjectDatabase, error)

func NewMemDatabase 

func NewMemDatabase() *ObjectDatabase

func NewObjectDatabase 

func NewObjectDatabase(kv RwKV) *ObjectDatabase

NewObjectDatabase returns a AbstractDB wrapper.

func NewTestDB 

func NewTestDB(t testing.TB) *ObjectDatabase

func Open 

func Open(path string, readOnly bool) (*ObjectDatabase, error)

func (*ObjectDatabase) Append 

func (db *ObjectDatabase) Append(bucket string, key []byte, value []byte) error

Append appends a single entry to the end of the bucket.

func (*ObjectDatabase) AppendDup 

func (db *ObjectDatabase) AppendDup(bucket string, key []byte, value []byte) error

AppendDup appends a single entry to the end of the bucket.

func (*ObjectDatabase) Begin 

func (db *ObjectDatabase) Begin(ctx context.Context, flags TxFlags) (DbWithPendingMutations, error)

func (*ObjectDatabase) BeginGetter 

func (db *ObjectDatabase) BeginGetter(ctx context.Context) (GetterTx, error)

func (*ObjectDatabase) BucketExists 

func (db *ObjectDatabase) BucketExists(name string) (bool, error)

func (*ObjectDatabase) ClearBuckets 

func (db *ObjectDatabase) ClearBuckets(buckets ...string) error

func (*ObjectDatabase) Close 

func (db *ObjectDatabase) Close()

func (*ObjectDatabase) Delete 

func (db *ObjectDatabase) Delete(bucket string, k, v []byte) error

Delete deletes the key from the queue and database

func (*ObjectDatabase) DiskSize 

func (db *ObjectDatabase) DiskSize(ctx context.Context) (uint64, error)

func (*ObjectDatabase) DropBuckets 

func (db *ObjectDatabase) DropBuckets(buckets ...string) error

func (*ObjectDatabase) Get 

func (db *ObjectDatabase) Get(bucket string, key []byte) ([]byte, error)

func (*ObjectDatabase) GetOne 

func (db *ObjectDatabase) GetOne(bucket string, key []byte) ([]byte, error)

Get returns the value for a given key if it's present.

func (*ObjectDatabase) Has 

func (db *ObjectDatabase) Has(bucket string, key []byte) (bool, error)

func (*ObjectDatabase) IncrementSequence 

func (db *ObjectDatabase) IncrementSequence(bucket string, amount uint64) (res uint64, err error)

func (*ObjectDatabase) Keys 

func (db *ObjectDatabase) Keys() ([][]byte, error)

func (*ObjectDatabase) Last 

func (db *ObjectDatabase) Last(bucket string) ([]byte, []byte, error)

func (*ObjectDatabase) MemCopy 

func (db *ObjectDatabase) MemCopy() *ObjectDatabase

func (*ObjectDatabase) MultiPut 

func (db *ObjectDatabase) MultiPut(tuples ...[]byte) (uint64, error)

MultiPut - requirements: input must be sorted and without duplicates

func (*ObjectDatabase) NewBatch 

func (db *ObjectDatabase) NewBatch() DbWithPendingMutations

func (*ObjectDatabase) Put 

func (db *ObjectDatabase) Put(bucket string, key []byte, value []byte) error

Put inserts or updates a single entry.

func (*ObjectDatabase) ReadSequence 

func (db *ObjectDatabase) ReadSequence(bucket string) (res uint64, err error)

func (*ObjectDatabase) RwKV 

func (db *ObjectDatabase) RwKV() RwKV

func (*ObjectDatabase) SetRwKV 

func (db *ObjectDatabase) SetRwKV(kv RwKV)

func (*ObjectDatabase) Walk 

func (db *ObjectDatabase) Walk(bucket string, startkey []byte, fixedbits int, walker func(k, v []byte) (bool, error)) error

type Putter 

type Putter interface {
	// Put inserts or updates a single entry.
	Put(bucket string, key, value []byte) error
}

Putter wraps the database write operations.

type RemoteKV 

type RemoteKV struct {
	// contains filtered or unexported fields
}

func (*RemoteKV) AllBuckets 

func (db *RemoteKV) AllBuckets() dbutils.BucketsCfg

func (*RemoteKV) BeginRo 

func (db *RemoteKV) BeginRo(ctx context.Context) (Tx, error)

func (*RemoteKV) BeginRw 

func (db *RemoteKV) BeginRw(ctx context.Context) (RwTx, error)

func (*RemoteKV) Close 

func (db *RemoteKV) Close()

Close All transactions must be closed before closing the database.

func (*RemoteKV) CollectMetrics 

func (db *RemoteKV) CollectMetrics()

func (*RemoteKV) GrpcConn 

func (db *RemoteKV) GrpcConn() *grpc.ClientConn

func (*RemoteKV) Update 

func (db *RemoteKV) Update(ctx context.Context, f func(tx RwTx) error) (err error)

func (*RemoteKV) View 

func (db *RemoteKV) View(ctx context.Context, f func(tx Tx) error) (err error)

type RoKV 

type RoKV interface {
	Closer

	View(ctx context.Context, f func(tx Tx) error) error

	// BeginRo - creates transaction
	// 	tx may be discarded by .Rollback() method
	//
	// A transaction and its cursors must only be used by a single
	// 	thread (not goroutine), and a thread may only have a single transaction at a time.
	//  It happen automatically by - because this method calls runtime.LockOSThread() inside (Rollback/Commit releases it)
	//  By this reason application code can't call runtime.UnlockOSThread() - it leads to undefined behavior.
	//
	// If this `parent` is non-NULL, the new transaction
	//	will be a nested transaction, with the transaction indicated by parent
	//	as its parent. Transactions may be nested to any level. A parent
	//	transaction and its cursors may not issue any other operations than
	//	Commit and Rollback while it has active child transactions.
	BeginRo(ctx context.Context) (Tx, error)
	AllBuckets() dbutils.BucketsCfg

	CollectMetrics()
}

Read-only version of KV.

type RwCursor 

type RwCursor interface {
	Cursor

	Put(k, v []byte) error           // Put - based on order
	Append(k []byte, v []byte) error // Append - append the given key/data pair to the end of the database. This option allows fast bulk loading when keys are already known to be in the correct order.
	Delete(k, v []byte) error        // Delete - short version of SeekExact+DeleteCurrent or SeekBothExact+DeleteCurrent

	// DeleteCurrent This function deletes the key/data pair to which the cursor refers.
	// This does not invalidate the cursor, so operations such as MDB_NEXT
	// can still be used on it.
	// Both MDB_NEXT and MDB_GET_CURRENT will return the same record after
	// this operation.
	DeleteCurrent() error
}

type RwCursorDupSort 

type RwCursorDupSort interface {
	CursorDupSort
	RwCursor

	DeleteCurrentDuplicates() error    // DeleteCurrentDuplicates - deletes all of the data items for the current key
	AppendDup(key, value []byte) error // AppendDup - same as Append, but for sorted dup data
}

type RwKV 

type RwKV interface {
	RoKV

	Update(ctx context.Context, f func(tx RwTx) error) error

	BeginRw(ctx context.Context) (RwTx, error)
}

KV low-level database interface - main target is - to provide common abstraction over top of LMDB and RemoteKV.

Common pattern for short-living transactions: 

 if err := db.View(ctx, func(tx ethdb.Tx) error {
    ... code which uses database in transaction
 }); err != nil {
		return err
}

Common pattern for long-living transactions: 

tx, err := db.Begin()
if err != nil {
	return err
}
defer tx.Rollback()

... code which uses database in transaction

err := tx.Commit()
if err != nil {
	return err
}

func GenStateData 

func GenStateData(data []KvData) (RwKV, error)

func MustOpenKV 

func MustOpenKV(path string) RwKV

func NewMemKV 

func NewMemKV() RwKV

func NewTestKV 

func NewTestKV(t testing.TB) RwKV

func OpenKV 

func OpenKV(path string, readOnly bool) (RwKV, error)

Open - main method to open database. Choosing driver based on path suffix. If env TEST_DB provided - choose driver based on it. Some test using this method to open non-in-memory db

type RwTx 

type RwTx interface {
	Tx
	StatelessWriteTx
	BucketMigrator

	RwCursor(bucket string) (RwCursor, error)
	RwCursorDupSort(bucket string) (RwCursorDupSort, error)
}

type SnapshotKV 

type SnapshotKV struct {
	// contains filtered or unexported fields
}

func (*SnapshotKV) AllBuckets 

func (s *SnapshotKV) AllBuckets() dbutils.BucketsCfg

func (*SnapshotKV) BeginRo 

func (s *SnapshotKV) BeginRo(ctx context.Context) (Tx, error)

func (*SnapshotKV) BeginRw 

func (s *SnapshotKV) BeginRw(ctx context.Context) (RwTx, error)

func (*SnapshotKV) Close 

func (s *SnapshotKV) Close()

func (*SnapshotKV) CollectMetrics 

func (s *SnapshotKV) CollectMetrics()

func (*SnapshotKV) SnapshotKV 

func (s *SnapshotKV) SnapshotKV(bucket string) RoKV

func (*SnapshotKV) Update 

func (s *SnapshotKV) Update(ctx context.Context, f func(tx RwTx) error) error

func (*SnapshotKV) UpdateSnapshots 

func (s *SnapshotKV) UpdateSnapshots(buckets []string, snapshotKV RoKV, done chan struct{})

func (*SnapshotKV) View 

func (s *SnapshotKV) View(ctx context.Context, f func(tx Tx) error) error

func (*SnapshotKV) WriteDB 

func (s *SnapshotKV) WriteDB() RwKV

type SnapshotUpdater 

type SnapshotUpdater interface {
	UpdateSnapshots(buckets []string, snapshotKV RoKV, done chan struct{})
	SnapshotKV(bucket string) RoKV
}

type StatelessReadTx 

type StatelessReadTx interface {
	KVGetter

	Commit() error // Commit all the operations of a transaction into the database.
	Rollback()     // Rollback - abandon all the operations of the transaction instead of saving them.

	// ReadSequence - allows to create a linear sequence of unique positive integers for each table.
	// Can be called for a read transaction to retrieve the current sequence value, and the increment must be zero.
	// Sequence changes become visible outside the current write transaction after it is committed, and discarded on abort.
	// Starts from 0.
	ReadSequence(bucket string) (uint64, error)
}

type StatelessRwTx 

type StatelessRwTx interface {
	StatelessReadTx
	StatelessWriteTx
}

type StatelessWriteTx 

type StatelessWriteTx interface {
	Putter
	Deleter

	IncrementSequence(bucket string, amount uint64) (uint64, error)
}

type StorageMode 

type StorageMode struct {
	Initialised bool // Set when the values are initialised (not default)
	Pruning     bool
	History     bool
	Receipts    bool
	TxIndex     bool
	CallTraces  bool
}

func GetStorageModeFromDB 

func GetStorageModeFromDB(db KVGetter) (StorageMode, error)

func StorageModeFromString 

func StorageModeFromString(flags string) (StorageMode, error)

func (StorageMode) ToString 

func (m StorageMode) ToString() string

type Tx 

type Tx interface {
	StatelessReadTx

	// Cursor - creates cursor object on top of given bucket. Type of cursor - depends on bucket configuration.
	// If bucket was created with lmdb.DupSort flag, then cursor with interface CursorDupSort created
	// Otherwise - object of interface Cursor created
	//
	// Cursor, also provides a grain of magic - it can use a declarative configuration - and automatically break
	// long keys into DupSort key/values. See docs for `bucket.go:BucketConfigItem`
	Cursor(bucket string) (Cursor, error)
	CursorDupSort(bucket string) (CursorDupSort, error) // CursorDupSort - can be used if bucket has lmdb.DupSort flag

	ForEach(bucket string, fromPrefix []byte, walker func(k, v []byte) error) error
	ForPrefix(bucket string, prefix []byte, walker func(k, v []byte) error) error

	Comparator(bucket string) dbutils.CmpFunc

	CHandle() unsafe.Pointer // Pointer to the underlying C transaction handle (e.g. *C.MDB_txn)
	CollectMetrics()
}

type TxDb 

type TxDb struct {
	// contains filtered or unexported fields
}

TxDb - provides Database interface around ethdb.Tx It's not thread-safe! TxDb not usable after .Commit()/.Rollback() call, but usable after .CommitAndBegin() call you can put unlimited amount of data into this class Walk and MultiWalk methods - work outside of Tx object yet, will implement it later

func NewRwTxDb 

func NewRwTxDb(tx Tx) *TxDb

func WrapIntoTxDB 

func WrapIntoTxDB(tx RwTx) *TxDb

func (*TxDb) Append 

func (m *TxDb) Append(bucket string, key []byte, value []byte) error

func (*TxDb) AppendDup 

func (m *TxDb) AppendDup(bucket string, key []byte, value []byte) error

func (*TxDb) BatchSize 

func (m *TxDb) BatchSize() int

func (*TxDb) Begin 

func (m *TxDb) Begin(ctx context.Context, flags TxFlags) (DbWithPendingMutations, error)

func (*TxDb) BeginGetter 

func (m *TxDb) BeginGetter(ctx context.Context) (GetterTx, error)

func (*TxDb) BucketExists 

func (m *TxDb) BucketExists(name string) (bool, error)

func (*TxDb) ClearBuckets 

func (m *TxDb) ClearBuckets(buckets ...string) error

func (*TxDb) Close 

func (m *TxDb) Close()

func (*TxDb) Commit 

func (m *TxDb) Commit() error

func (*TxDb) CommitAndBegin 

func (m *TxDb) CommitAndBegin(ctx context.Context) error

func (*TxDb) Delete 

func (m *TxDb) Delete(bucket string, k, v []byte) error

func (*TxDb) DiskSize 

func (m *TxDb) DiskSize(ctx context.Context) (common.StorageSize, error)

func (*TxDb) DropBuckets 

func (m *TxDb) DropBuckets(buckets ...string) error

func (*TxDb) Get 

func (m *TxDb) Get(bucket string, key []byte) ([]byte, error)

func (*TxDb) GetOne 

func (m *TxDb) GetOne(bucket string, key []byte) ([]byte, error)

func (*TxDb) Has 

func (m *TxDb) Has(bucket string, key []byte) (bool, error)

func (*TxDb) IncrementSequence 

func (m *TxDb) IncrementSequence(bucket string, amount uint64) (res uint64, err error)

func (*TxDb) Keys 

func (m *TxDb) Keys() ([][]byte, error)

func (*TxDb) Last 

func (m *TxDb) Last(bucket string) ([]byte, []byte, error)

Last can only be called from the transaction thread

func (*TxDb) MultiPut 

func (m *TxDb) MultiPut(tuples ...[]byte) (uint64, error)

func (*TxDb) NewBatch 

func (m *TxDb) NewBatch() DbWithPendingMutations

func (*TxDb) Put 

func (m *TxDb) Put(bucket string, key []byte, value []byte) error

func (*TxDb) ReadSequence 

func (m *TxDb) ReadSequence(bucket string) (res uint64, err error)

func (*TxDb) Rollback 

func (m *TxDb) Rollback()

func (*TxDb) RollbackAndBegin 

func (m *TxDb) RollbackAndBegin(ctx context.Context) error

func (*TxDb) RwKV 

func (m *TxDb) RwKV() RwKV

func (*TxDb) Tx 

func (m *TxDb) Tx() Tx

func (*TxDb) Walk 

func (m *TxDb) Walk(bucket string, startkey []byte, fixedbits int, walker func([]byte, []byte) (bool, error)) error

type TxFlags 

type TxFlags uint
const (
	RW TxFlags = 0x00 // default
	RO TxFlags = 0x02
)

type WriteDB 

type WriteDB interface {
	WriteDB() RwKV
}

Source Files

View all Source files

Directories

Path Synopsis
mdbxarch
Package lmdbarch contains some architecture detection constants.
 Package lmdbarch contains some architecture detection constants.
remote
remotedbserver

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.