database

Documentation

Index

• Constants
• func BadgerTrain(ctx context.Context, namespace string, db string, output string, ...) (float64, float64, error)
• func DefaultBadgerOptions(dir string) badger.Options
• func PerformanceBadgerOptions(dir string) badger.Options
• type BadgerDatabase
  • func (b *BadgerDatabase) Close(ctx context.Context) error
  • func (b *BadgerDatabase) Encoder() *encoder.Encoder
  • func (b *BadgerDatabase) ReadTransaction(ctx context.Context) Transaction
  • func (b *BadgerDatabase) Transaction(ctx context.Context) Transaction
  • func (b *BadgerDatabase) WriteTransaction(ctx context.Context, identifier string, priority bool) Transaction
• type BadgerOption
  • func WithCompressorEntries(entries []*encoder.CompressorEntry) BadgerOption
  • func WithCustomSettings(settings badger.Options) BadgerOption
  • func WithIndexCacheSize(size int64) BadgerOption
  • func WithWriterShards(shards int) BadgerOption
  • func WithoutCompression() BadgerOption
• type BadgerTransaction
  • func (b *BadgerTransaction) Commit(context.Context) error
  • func (b *BadgerTransaction) Delete(ctx context.Context, key []byte) error
  • func (b *BadgerTransaction) Discard(context.Context)
  • func (b *BadgerTransaction) Get(ctx context.Context, key []byte) (bool, []byte, error)
  • func (b *BadgerTransaction) Scan(ctx context.Context, prefix []byte, seekStart []byte, ...) (int, error)
  • func (b *BadgerTransaction) Set(ctx context.Context, key []byte, value []byte, reclaimValue bool) error
• type CommitWorker
• type Database
  • func NewBadgerDatabase(ctx context.Context, dir string, storageOptions ...BadgerOption) (Database, error)
• type Transaction

Constants

const (
	// DefaultBlockCacheSize is 0 MB.
	DefaultBlockCacheSize = 0

	// DefaultIndexCacheSize is 2 GB.
	DefaultIndexCacheSize = 2000 << 20

	// TinyIndexCacheSize is 10 MB.
	TinyIndexCacheSize = 10 << 20

	// DefaultMaxTableSize is 256 MB. The larger
	// this value is, the larger database transactions
	// storage can handle (~15% of the max table size
	// == max commit size).
	DefaultMaxTableSize = 256 << 20

	// DefaultLogValueSize is 64 MB.
	DefaultLogValueSize = 64 << 20

	// PerformanceMaxTableSize is 3072 MB. The larger
	// this value is, the larger database transactions
	// storage can handle (~15% of the max table size
	// == max commit size).
	PerformanceMaxTableSize = 3072 << 20

	// PerformanceLogValueSize is 256 MB.
	PerformanceLogValueSize = 256 << 20

	// DefaultCompressionMode is the default block
	// compression setting.
	DefaultCompressionMode = options.None
)

Functions

func BadgerTrain

func BadgerTrain(
	ctx context.Context,
	namespace string,
	db string,
	output string,
	maxEntries int,
	compressorEntries []*encoder.CompressorEntry,
) (float64, float64, error)

BadgerTrain creates a zstd dictionary for a given BadgerDatabase DB namespace. Optionally, you can specify the maximum number of entries to load into storage (if -1 is provided, then all possible are loaded).

func DefaultBadgerOptions

func DefaultBadgerOptions(dir string) badger.Options

DefaultBadgerOptions are the default options used to initialized a new BadgerDB. These settings override many of the default BadgerDB settings to restrict memory usage to ~6 GB. If constraining memory usage is not desired for your use case, you can provide your own BadgerDB settings with the configuration option WithCustomSettings.

There are many threads about optimizing memory usage in Badger (which can grow to many GBs if left untuned). Our own research indicates that each MB increase in MaxTableSize and/or ValueLogFileSize corresponds to a 10 MB increase in RAM usage (all other settings equal). Our primary concern is large database transaction size, so we configure MaxTableSize to be 4 times the size of ValueLogFileSize (if we skewed any further to MaxTableSize, we would quickly hit the default open file limit on many OSes).

func PerformanceBadgerOptions

func PerformanceBadgerOptions(dir string) badger.Options

PerformanceBadgerOptions are performance geared BadgerDB options that use much more RAM than the default settings.

Types

type BadgerDatabase

type BadgerDatabase struct {
	// contains filtered or unexported fields
}

BadgerDatabase is a wrapper around Badger DB that implements the Database interface.

func (*BadgerDatabase) Close

func (b *BadgerDatabase) Close(ctx context.Context) error

Close closes the database to prevent corruption. The caller should defer this in main.

func (*BadgerDatabase) Encoder

func (b *BadgerDatabase) Encoder() *encoder.Encoder

Encoder returns the BadgerDatabase encoder.

func (*BadgerDatabase) ReadTransaction

func (b *BadgerDatabase) ReadTransaction(
	ctx context.Context,
) Transaction

ReadTransaction creates a new read BadgerTransaction.

func (*BadgerDatabase) Transaction

func (b *BadgerDatabase) Transaction(
	ctx context.Context,
) Transaction

Transaction creates a new exclusive write BadgerTransaction.

func (*BadgerDatabase) WriteTransaction

func (b *BadgerDatabase) WriteTransaction(
	ctx context.Context,
	identifier string,
	priority bool,
) Transaction

WriteTransaction creates a new write BadgerTransaction for a particular identifier.

type BadgerOption

type BadgerOption func(b *BadgerDatabase)

BadgerOption is used to overwrite default values in BadgerDatabase construction. Any Option not provided falls back to the default value.

func WithCompressorEntries

func WithCompressorEntries(entries []*encoder.CompressorEntry) BadgerOption

WithCompressorEntries provides zstd dictionaries for given namespaces.

func WithCustomSettings

func WithCustomSettings(settings badger.Options) BadgerOption

WithCustomSettings allows for overriding all default BadgerDB options with custom settings.

func WithIndexCacheSize

func WithIndexCacheSize(size int64) BadgerOption

WithIndexCacheSize override the DefaultIndexCacheSize setting for the BadgerDB. The size here is in bytes. If you provide custom BadgerDB settings, do not use this config as it will be overridden by your custom settings.

func WithWriterShards

func WithWriterShards(shards int) BadgerOption

WithWriterShards overrides the default shards used in the writer utils.MutexMap. It is recommended to set this value to your write concurrency to prevent lock contention.

func WithoutCompression

func WithoutCompression() BadgerOption

WithoutCompression disables zstd compression.

type BadgerTransaction

type BadgerTransaction struct {
	// contains filtered or unexported fields
}

BadgerTransaction is a wrapper around a Badger DB transaction that implements the DatabaseTransaction interface.

func (*BadgerTransaction) Commit

func (b *BadgerTransaction) Commit(context.Context) error

Commit attempts to commit and discard the transaction.

func (*BadgerTransaction) Delete

func (b *BadgerTransaction) Delete(ctx context.Context, key []byte) error

Delete removes the key and its value within the transaction.

func (*BadgerTransaction) Discard

func (b *BadgerTransaction) Discard(context.Context)

Discard discards an open transaction. All transactions must be either discarded or committed.

func (*BadgerTransaction) Get

func (b *BadgerTransaction) Get(
	ctx context.Context,
	key []byte,
) (bool, []byte, error)

Get accesses the value of the key within a transaction. It is up to the caller to reclaim any memory returned.

func (*BadgerTransaction) Scan

func (b *BadgerTransaction) Scan(
	ctx context.Context,
	prefix []byte,
	seekStart []byte,
	worker func([]byte, []byte) error,
	logEntries bool,
	reverse bool,
) (int, error)

Scan calls a worker for each item in a scan instead of reading all items into memory.

func (*BadgerTransaction) Set

func (b *BadgerTransaction) Set(
	ctx context.Context,
	key []byte,
	value []byte,
	reclaimValue bool,
) error

Set changes the value of the key to the value within a transaction.

type CommitWorker

type CommitWorker func(context.Context) error

CommitWorker is returned by a module to be called after changes have been committed. It is common to put logging activities in here (that shouldn't be printed until the block is committed).

type Database

type Database interface {
	// Transaction acquires an exclusive write lock on the database.
	// This ensures all other calls to Transaction and WriteTransaction
	// will block until the returned DatabaseTransaction is committed or
	// discarded. This is useful for making changes across
	// multiple prefixes but incurs a large performance overhead.
	Transaction(context.Context) Transaction

	// ReadTransaction allows for consistent, read-only access
	// to the database. This does not acquire any lock
	// on the database.
	ReadTransaction(context.Context) Transaction

	// WriteTransaction acquires a granular write lock for a particular
	// identifier. All subsequent calls to WriteTransaction with the same
	// identifier will block until the DatabaseTransaction returned is either
	// committed or discarded.
	WriteTransaction(ctx context.Context, identifier string, priority bool) Transaction

	// Close shuts down the database.
	Close(context.Context) error

	// Encoder returns the *Encoder used to store/read data
	// in the database. This *Encoder often performs some
	// form of compression on data.
	Encoder() *encoder.Encoder
}

Database is an interface that provides transactional access to a KV store.

func NewBadgerDatabase

func NewBadgerDatabase(
	ctx context.Context,
	dir string,
	storageOptions ...BadgerOption,
) (Database, error)

NewBadgerDatabase creates a new BadgerDatabase.

type Transaction

type Transaction interface {
	Set(context.Context, []byte, []byte, bool) error
	Get(context.Context, []byte) (bool, []byte, error)
	Delete(context.Context, []byte) error

	Scan(
		context.Context,
		[]byte,
		[]byte,
		func([]byte, []byte) error,
		bool,
		bool,
	) (int, error)

	Commit(context.Context) error
	Discard(context.Context)
}

Transaction is an interface that provides access to a KV store within some transaction context provided by a Database.

When a Transaction is committed or discarded, all memory utilized is reclaimed. If you want to persist any data retrieved, make sure to make a copy!