database

package module
v1.17.38 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 9, 2026 License: BSD-3-Clause Imports: 17 Imported by: 133

Documentation

Overview

Package database provides enhanced database utilities with VictoriaMetrics-style optimizations

Package database provides memory management utilities inspired by VictoriaMetrics

Index

Constants

View Source 
const (
	// MaxExcessCapacityFactor is the factor above which capacity is considered excessive
	MaxExcessCapacityFactor = 4
	// CapacityReductionFactor is the factor by which to reduce capacity when it's excessive
	CapacityReductionFactor = 2
)
View Source 
const (
	Uint64Size = 8 // bytes
	BoolSize   = 1 // bytes
	BoolFalse  = 0x00
	BoolTrue   = 0x01
)

Variables

View Source 
var (
	ErrClosed   = errors.New("closed")
	ErrNotFound = errors.New("not found")
)

common errors

Functions

func AtomicClear 

func AtomicClear(readerDB Iteratee, deleterDB KeyValueDeleter) error

func AtomicClearPrefix 

func AtomicClearPrefix(readerDB Iteratee, deleterDB KeyValueDeleter, prefix []byte) error

AtomicClearPrefix deletes from [deleterDB] all keys in [readerDB] that have the given [prefix].

func Clear 

func Clear(db Database, writeSize int) error

Remove all key-value pairs from [db]. Writes each batch when it reaches [writeSize].

func ClearPrefix 

func ClearPrefix(db Database, prefix []byte, writeSize int) error

Removes all keys with the given [prefix] from [db]. Writes each batch when it reaches [writeSize].

func Count 

func Count(db Iteratee) (int, error)

func GetBool 

func GetBool(db KeyValueReader, key []byte) (bool, error)

func GetID 

func GetID(db KeyValueReader, key []byte) (ids.ID, error)

func GetTimestamp 

func GetTimestamp(db KeyValueReader, key []byte) (time.Time, error)

func GetUInt32 

func GetUInt32(db KeyValueReader, key []byte) (uint32, error)

func GetUInt64 

func GetUInt64(db KeyValueReader, key []byte) (uint64, error)

func IsEmpty added in v1.0.3 

func IsEmpty(db Iteratee) (bool, error)

func NewErrBackendDisabled 

func NewErrBackendDisabled(backend string) error

NewErrBackendDisabled creates a new ErrBackendDisabled error

func PackUInt32 

func PackUInt32(val uint32) []byte

func PackUInt64 

func PackUInt64(val uint64) []byte

func ParseID added in v1.0.3 

func ParseID(b []byte) (ids.ID, error)

func ParseTimestamp 

func ParseTimestamp(b []byte) (time.Time, error)

func ParseUInt32 

func ParseUInt32(b []byte) (uint32, error)

func ParseUInt64 

func ParseUInt64(b []byte) (uint64, error)

func PutBool 

func PutBool(db KeyValueWriter, key []byte, b bool) error

func PutID 

func PutID(db KeyValueWriter, key []byte, val ids.ID) error

func PutTimestamp 

func PutTimestamp(db KeyValueWriter, key []byte, val time.Time) error

func PutUInt32 

func PutUInt32(db KeyValueWriter, key []byte, val uint32) error

func PutUInt64 

func PutUInt64(db KeyValueWriter, key []byte, val uint64) error

func Size 

func Size(db Iteratee) (int, error)

Types

type BadgerDBWithCache added in v1.17.37 

type BadgerDBWithCache struct {
	// contains filtered or unexported fields
}

BadgerDBWithCache wraps BadgerDB with caching and compression optimizations

func NewBadgerDBWithCache added in v1.17.37 

func NewBadgerDBWithCache(db Database, cacheSize int, compressionLevel zstd.EncoderLevel, maxConcurrency int, reg *metric.MetricsRegistry) (*BadgerDBWithCache, error)

NewBadgerDBWithCache creates a new BadgerDB wrapper with caching and compression

func (*BadgerDBWithCache) Delete added in v1.17.37 

func (bdb *BadgerDBWithCache) Delete(key []byte) error

Delete removes a key from the database and cache

func (*BadgerDBWithCache) Get added in v1.17.37 

func (bdb *BadgerDBWithCache) Get(key []byte) ([]byte, error)

Get retrieves a value from the database with caching

func (*BadgerDBWithCache) Has added in v1.17.37 

func (bdb *BadgerDBWithCache) Has(key []byte) (bool, error)

Has checks if a key exists in the database

func (*BadgerDBWithCache) Put added in v1.17.37 

func (bdb *BadgerDBWithCache) Put(key, value []byte) error

Put inserts a value into the database with caching and compression

type Batch 

type Batch interface {
	KeyValueWriterDeleter

	// Size retrieves the amount of data queued up for writing, this includes
	// the keys, values, and deleted keys.
	Size() int

	// Write flushes any accumulated data to disk.
	Write() error

	// Reset resets the batch for reuse.
	Reset()

	// Replay replays the batch contents in the same order they were written
	// to the batch.
	Replay(w KeyValueWriterDeleter) error

	// Inner returns a Batch writing to the inner database, if one exists. If
	// this batch is already writing to the base DB, then itself should be
	// returned.
	Inner() Batch
}

Batch is a write-only database that commits changes to its host database when Write is called. A batch cannot be used concurrently.

type BatchOp 

type BatchOp struct {
	Key    []byte
	Value  []byte
	Delete bool
}

type BatchOps 

type BatchOps struct {
	Ops []BatchOp
	// contains filtered or unexported fields
}

func (*BatchOps) Delete 

func (b *BatchOps) Delete(key []byte) error

func (*BatchOps) Put 

func (b *BatchOps) Put(key, value []byte) error

func (*BatchOps) Replay 

func (b *BatchOps) Replay(w KeyValueWriterDeleter) error

func (*BatchOps) Reset 

func (b *BatchOps) Reset()

func (*BatchOps) Size 

func (b *BatchOps) Size() int

type BatchedDatabase added in v1.17.37 

type BatchedDatabase struct {
	// contains filtered or unexported fields
}

BatchedDatabase provides batched operations with optimizations

func NewBatchedDatabase added in v1.17.37 

func NewBatchedDatabase(db Database, batchSize int, batchTimeout time.Duration, reg *metric.MetricsRegistry) *BatchedDatabase

NewBatchedDatabase creates a new batched database wrapper

func (*BatchedDatabase) Close added in v1.17.37 

func (b *BatchedDatabase) Close() error

Close closes the batched database and flushes any remaining operations

func (*BatchedDatabase) Delete added in v1.17.37 

func (b *BatchedDatabase) Delete(key []byte) error

Delete adds a delete operation to the batch

func (*BatchedDatabase) Flush added in v1.17.37 

func (b *BatchedDatabase) Flush() error

Flush forces a flush of all pending operations

func (*BatchedDatabase) Put added in v1.17.37 

func (b *BatchedDatabase) Put(key, value []byte) error

Put adds a put operation to the batch

type Batcher 

type Batcher interface {
	// NewBatch creates a write-only database that buffers changes to its host db
	// until a final write is called.
	NewBatch() Batch
}

Batcher wraps the NewBatch method of a backing data store.

type Compacter 

type Compacter interface {
	// Compact the underlying DB for the given key range.
	// Specifically, deleted and overwritten versions are discarded,
	// and the data is rearranged to reduce the cost of operations
	// needed to access the data. This operation should typically only
	// be invoked by users who understand the underlying implementation.
	//
	// A nil start is treated as a key before all keys in the DB.
	// And a nil limit is treated as a key after all keys in the DB.
	// Therefore if both are nil then it will compact entire DB.
	//
	// Note: [start] and [limit] are safe to modify and read after calling Compact.
	Compact(start []byte, limit []byte) error
}

Compacter wraps the Compact method of a backing data store.

type Database 

type Database interface {
	KeyValueReaderWriterDeleter
	Batcher
	Iteratee
	Compacter
	Syncer
	io.Closer
	HealthCheck(context.Context) (interface{}, error)

	// Backup performs a backup of the database to the provided writer.
	// This should be a consistent snapshot of the database state.
	// since: 0 for full backup, or the version of the last backup for incremental.
	// returns: the version (timestamp/sequence) of this backup, to be used for the next incremental.
	Backup(w io.Writer, since uint64) (uint64, error)

	// Load restores the database from the provided reader.
	// This will overwrite the current database state with the backup.
	// Note: The database should generally be empty or fresh when calling Load.
	Load(r io.Reader) error
}

Database contains all the methods required to allow handling different key-value data stores backing the database.

type ErrBackendDisabled 

type ErrBackendDisabled struct {
	Backend string
}

ErrBackendDisabled is returned when a database backend is disabled at compile time

func (*ErrBackendDisabled) Error 

func (e *ErrBackendDisabled) Error() string

type HeightIndex added in v1.2.8 

type HeightIndex interface {
	// Put inserts the value at the given height.
	//
	// Note: [value] is safe to modify and read after calling Put.
	//
	// If [value] is nil or an empty slice, then when it's retrieved
	// it may be nil or an empty slice.
	Put(height uint64, value []byte) error

	// Get retrieves the value at the given height.
	// Returns ErrNotFound if the height does not exist.
	//
	// The returned byte slice is safe to read, but cannot be modified.
	Get(height uint64) ([]byte, error)

	// Has retrieves if a value exists at the given height.
	Has(height uint64) (bool, error)

	// Delete removes the value at the given height.
	Delete(height uint64) error

	// Close releases any resources associated with the index.
	Close() error
}

HeightIndex provides height-based key-value storage for blockchain state. Heights are stored as big-endian uint64 keys for efficient range queries.

type Iteratee 

type Iteratee interface {
	// NewIterator creates an iterator over the entire keyspace contained within
	// the key-value database.
	NewIterator() Iterator

	// NewIteratorWithStart creates an iterator over a subset of database
	// content starting at a particular initial key.
	NewIteratorWithStart(start []byte) Iterator

	// NewIteratorWithPrefix creates an iterator over a subset of database
	// content with a particular key prefix.
	NewIteratorWithPrefix(prefix []byte) Iterator

	// NewIteratorWithStartAndPrefix creates an iterator over a subset of
	// database content with a particular key prefix starting at a specified
	// key.
	NewIteratorWithStartAndPrefix(start, prefix []byte) Iterator
}

Iteratee wraps the NewIterator methods of a backing data store.

type Iterator 

type Iterator interface {
	// Next moves the iterator to the next key/value pair. It returns whether
	// the iterator successfully moved to a new key/value pair.
	// The iterator may return false if the underlying database has been closed
	// before the iteration has completed, in which case future calls to Error()
	// must return [ErrClosed].
	Next() bool

	// Error returns any accumulated error. Exhausting all the key/value pairs
	// is not considered to be an error.
	// Error should be called after all key/value pairs have been exhausted ie.
	// after Next() has returned false.
	Error() error

	// Key returns the key of the current key/value pair, or nil if done.
	// If the database is closed, must still report the current contents.
	// Behavior is undefined after Release is called.
	Key() []byte

	// Value returns the value of the current key/value pair, or nil if done.
	// If the database is closed, must still report the current contents.
	// Behavior is undefined after Release is called.
	Value() []byte

	// Release releases associated resources. Release should always succeed and
	// can be called multiple times without causing error.
	Release()
}

Iterator iterates over a database's key/value pairs.

When it encounters an error any seek will return false and will yield no key/ value pairs. The error can be queried by calling the Error method. Calling Release is still necessary.

An iterator must be released after use, but it is not necessary to read an iterator until exhaustion. An iterator is not safe for concurrent use, but it is safe to use multiple iterators concurrently.

type IteratorError 

type IteratorError struct {
	Err error
}

IteratorError does nothing and returns the provided error

func (*IteratorError) Error 

func (i *IteratorError) Error() error

func (*IteratorError) Key 

func (*IteratorError) Key() []byte

func (*IteratorError) Next 

func (*IteratorError) Next() bool

func (*IteratorError) Release 

func (*IteratorError) Release()

func (*IteratorError) Value 

func (*IteratorError) Value() []byte

type KeyValueDeleter 

type KeyValueDeleter interface {
	// Delete removes the key from the key-value data store.
	//
	// Note: [key] is safe to modify and read after calling Delete.
	Delete(key []byte) error
}

KeyValueDeleter wraps the Delete method of a backing data store.

type KeyValueReader 

type KeyValueReader interface {
	// Has retrieves if a key is present in the key-value data store.
	//
	// Note: [key] is safe to modify and read after calling Has.
	Has(key []byte) (bool, error)

	// Get retrieves the given key if it's present in the key-value data store.
	// Returns ErrNotFound if the key is not present in the key-value data store.
	//
	// Note: [key] is safe to modify and read after calling Get.
	// The returned byte slice is safe to read, but cannot be modified.
	Get(key []byte) ([]byte, error)
}

KeyValueReader wraps the Has and Get method of a backing data store.

type KeyValueReaderWriter 

type KeyValueReaderWriter interface {
	KeyValueReader
	KeyValueWriter
}

KeyValueReaderWriter allows read/write access to a backing data store.

type KeyValueReaderWriterDeleter 

type KeyValueReaderWriterDeleter interface {
	KeyValueReader
	KeyValueWriter
	KeyValueDeleter
}

KeyValueReaderWriterDeleter allows read/write/delete access to a backing data store.

type KeyValueWriter 

type KeyValueWriter interface {
	// Put inserts the given value into the key-value data store.
	//
	// Note: [key] and [value] are safe to modify and read after calling Put.
	//
	// If [value] is nil or an empty slice, then when it's retrieved
	// it may be nil or an empty slice.
	//
	// Similarly, a nil [key] is treated the same as an empty slice.
	Put(key []byte, value []byte) error
}

KeyValueWriter wraps the Put method of a backing data store.

type KeyValueWriterDeleter 

type KeyValueWriterDeleter interface {
	KeyValueWriter
	KeyValueDeleter
}

KeyValueWriterDeleter allows write/delete access to a backing data store.

type MemoryManager added in v1.17.37 

type MemoryManager struct {
	// contains filtered or unexported fields
}

MemoryManager provides memory management similar to VictoriaMetrics

func NewMemoryManager added in v1.17.37 

func NewMemoryManager(allowedPercent float64, reg *metric.MetricsRegistry) (*MemoryManager, error)

NewMemoryManager creates a new memory manager with the specified percentage of system memory

func (*MemoryManager) Allowed added in v1.17.37 

func (mm *MemoryManager) Allowed() int64

Allowed returns the amount of memory allowed for use

func (*MemoryManager) Available added in v1.17.37 

func (mm *MemoryManager) Available() int64

Available returns the amount of memory available for allocation

func (*MemoryManager) GetMemoryStats added in v1.17.37 

func (mm *MemoryManager) GetMemoryStats() MemoryStats

GetMemoryStats returns current memory statistics

func (*MemoryManager) Release added in v1.17.37 

func (mm *MemoryManager) Release(size int64)

Release releases previously reserved memory

func (*MemoryManager) ReleaseCritical added in v1.17.37 

func (mm *MemoryManager) ReleaseCritical(size int64)

ReleaseCritical releases memory from the critical reserve pool

func (*MemoryManager) Reserve added in v1.17.37 

func (mm *MemoryManager) Reserve(size int64)

Reserve reserves memory, blocking until it's available

func (*MemoryManager) ReserveCritical added in v1.17.37 

func (mm *MemoryManager) ReserveCritical(size int64)

ReserveCritical reserves memory from the critical reserve pool This is for operations that must not fail due to memory limits

func (*MemoryManager) TryReserve added in v1.17.37 

func (mm *MemoryManager) TryReserve(size int64) bool

TryReserve attempts to reserve memory without blocking Returns true if successful, false otherwise

func (*MemoryManager) Used added in v1.17.37 

func (mm *MemoryManager) Used() int64

Used returns the amount of memory currently in use

func (*MemoryManager) WithReserve added in v1.17.37 

func (mm *MemoryManager) WithReserve(size int64, fn func() error) error

WithReserve executes a function with reserved memory

type MemoryStats added in v1.17.37 

type MemoryStats struct {
	Allowed   int64   // Allowed memory in bytes
	Used      int64   // Used memory in bytes
	Available int64   // Available memory in bytes
	System    uint64  // Total system memory in bytes
	Percent   float64 // Percentage of system memory allowed
}

MemoryStats holds memory statistics

type ObjectPool added in v1.17.37 

type ObjectPool[T any] struct {
	// contains filtered or unexported fields
}

ObjectPool provides an optimized object pool similar to VictoriaMetrics

func NewObjectPool added in v1.17.37 

func NewObjectPool[T any](newFn func() T, size int64, mm *MemoryManager, reg *metric.MetricsRegistry) *ObjectPool[T]

NewObjectPool creates a new object pool with memory management

func (*ObjectPool[T]) Clear added in v1.17.37 

func (op *ObjectPool[T]) Clear()

Clear clears the object pool

func (*ObjectPool[T]) Get added in v1.17.37 

func (op *ObjectPool[T]) Get() T

Get acquires an object from the pool

func (*ObjectPool[T]) GetPoolSize added in v1.17.37 

func (op *ObjectPool[T]) GetPoolSize() int64

GetPoolSize returns the approximate number of objects in the pool

func (*ObjectPool[T]) Put added in v1.17.37 

func (op *ObjectPool[T]) Put(obj T)

Put returns an object to the pool

type OpType added in v1.17.37 

type OpType int

OpType represents the type of operation 

const (
	PutOp OpType = iota
	DeleteOp
)

type Operation added in v1.17.37 

type Operation struct {
	Type  OpType
	Key   []byte
	Value []byte
}

Operation represents a database operation

type RangeIterator added in v1.17.37 

type RangeIterator struct {
	// contains filtered or unexported fields
}

RangeIterator provides optimized range iteration

func NewRangeIterator added in v1.17.37 

func NewRangeIterator(db Database, start, end []byte, limit int, reverse bool, reg *metric.MetricsRegistry) *RangeIterator

NewRangeIterator creates a new range iterator

func (*RangeIterator) Count added in v1.17.37 

func (r *RangeIterator) Count() int

Count returns the number of items iterated

func (*RangeIterator) Iterate added in v1.17.37 

func (r *RangeIterator) Iterate(callback func(key, value []byte) error) error

Iterate iterates over the range and calls the callback for each key-value pair

type Syncer added in v1.2.17 

type Syncer interface {
	// Sync flushes all buffered writes to persistent storage.
	// This ensures that data written before Sync is called will
	// survive a crash or power failure. This is important for
	// databases configured with async writes for performance.
	//
	// Implementations should ensure all in-memory buffers (memtables,
	// write-ahead logs, etc.) are flushed to disk.
	Sync() error
}

Syncer wraps the Sync method of a backing data store.

Source Files

View all Source files

Directories

Path Synopsis
linear
Package databasemock is a generated GoMock package.
 Package databasemock is a generated GoMock package.
heightindexdb
dbtest
memdb
Package hookdb wraps a database to fire async callbacks on write operations.
 Package hookdb wraps a database to fire async callbacks on write operations.
state
syncapi
Package syncapi exposes the *data* formats needed by trie-/code-sync.
 Package syncapi exposes the *data* formats needed by trie-/code-sync.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.