mdbx

package
v0.32.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 19, 2023 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package lmdb provides bindings to the lmdb C API. The package bindings are fairly low level and are designed to provide a minimal interface that prevents misuse to a reasonable extent. When in doubt refer to the C documentation as a reference. 

http://www.lmdb.tech/doc/
http://www.lmdb.tech/doc/starting.html
http://www.lmdb.tech/doc/modules.html

Environment

An LMDB environment holds named databases (key-value stores). An environment is represented as one file on the filesystem (though often a corresponding lock file exists).

LMDB recommends setting an environment's size as large as possible at the time of creation. On filesystems that support sparse files this should not adversely affect disk usage. Resizing an environment is possible but must be handled with care when concurrent access is involved.

Note that the package lmdb forces all Env objects to be opened with the NoTLS (MDB_NOTLS) flag. Without this flag LMDB would not be practically usable in Go (in the author's opinion). However, even for environments opened with this flag there are caveats regarding how transactions are used (see Caveats below).

Databases

A database in an LMDB environment is an ordered key-value store that holds arbitrary binary data. Typically the keys are unique but duplicate keys may be allowed (DupSort), in which case the values for each duplicate key are ordered.

A single LMDB environment can have multiple named databases. But there is also a 'root' (unnamed) database that can be used to store data. Use caution storing data in the root database when named databases are in use. The root database serves as an index for named databases.

A database is referenced by an opaque handle known as its DBI which must be opened inside a transaction with the OpenDBI or OpenRoot methods. DBIs may be closed but it is not required. Typically, applications acquire handles for all their databases immediately after opening an environment and retain them for the lifetime of the process.

Transactions

View (readonly) transactions in LMDB operate on a snapshot of the database at the time the transaction began. The number of simultaneously active view transactions is bounded and configured when the environment is initialized.

Update (read-write) transactions are serialized in LMDB. Attempts to create update transactions block until a lock may be obtained. Update transactions can create subtransactions which may be rolled back independently from their parent.

The lmdb package supplies managed and unmanaged transactions. Managed transactions do not require explicit calling of Abort/Commit and are provided through the Env methods Update, View, and RunTxn. The BeginTxn method on Env creates an unmanaged transaction but its use is not advised in most applications.

To provide ACID guarantees, a readonly transaction must acquire a "lock" in the LMDB environment to ensure that data it reads is consistent over the course of the transaction's lifetime, and that updates happening concurrently will not be seen. If a reader does not release its lock then stale data, which has been overwritten by later transactions, cannot be reclaimed by LMDB -- resulting in a rapid increase in file size.

Long-running read transactions may cause increase an applications storage requirements, depending on the application write workload. But, typically the complete failure of an application to terminate a read transactions will result in continual increase file size to the point where the storage volume becomes full or a quota has been reached.

There are steps an application may take to greatly reduce the possibility of unterminated read transactions. The first safety measure is to avoid the use of Env.BeginTxn, which creates unmanaged transactions, and always use Env.View or Env.Update to create managed transactions that are (mostly) guaranteed to terminate. If Env.BeginTxn must be used try to defer a call to the Txn's Abort method (this is useful even for update transactions). 

txn, err := env.BeginTxn(nil, 0)
if err != nil {
	// ...
}
defer txn.Abort() // Safe even if txn.Commit() is called later.

Because application crashes and signals from the operation system may cause unexpected termination of a readonly transaction before Txn.Abort may be called it is also important that applications clear any readers held for dead OS processes when they start. 

numStale, err := env.ReaderCheck()
if err != nil {
	// ...
}
if numStale > 0 {
	log.Printf("Released locks for %d dead readers", numStale)
}

If an application gets accessed by multiple programs concurrently it is also a good idea to periodically call Env.ReaderCheck during application execution. However, note that Env.ReaderCheck cannot find readers opened by the application itself which have since leaked. Because of this, the lmdb package uses a finalizer to abort unreachable Txn objects. But of course, applications must still be careful not to leak unterminated Txn objects in a way such that they fail get garbage collected.

Caveats

Write transactions (those created without the Readonly flag) must be created in a goroutine that has been locked to its thread by calling the function runtime.LockOSThread. Futhermore, all methods on such transactions must be called from the goroutine which created them. This is a fundamental limitation of LMDB even when using the NoTLS flag (which the package always uses). The Env.Update method assists the programmer by calling runtime.LockOSThread automatically but it cannot sufficiently abstract write transactions to make them completely safe in Go.

A goroutine must never create a write transaction if the application programmer cannot determine whether the goroutine is locked to an OS thread. This is a consequence of goroutine restrictions on write transactions and limitations in the runtime's thread locking implementation. In such situations updates desired by the goroutine in question must be proxied by a goroutine with a known state (i.e. "locked" or "unlocked"). See the included examples for more details about dealing with such situations.

Index

Constants

View Source 
const (
	First        = C.MDBX_FIRST          // The first item.
	FirstDup     = C.MDBX_FIRST_DUP      // The first value of current key (DupSort).
	GetBoth      = C.MDBX_GET_BOTH       // Get the key as well as the value (DupSort).
	GetBothRange = C.MDBX_GET_BOTH_RANGE // Get the key and the nearsest value (DupSort).
	GetCurrent   = C.MDBX_GET_CURRENT    // Get the key and value at the current position.
	GetMultiple  = C.MDBX_GET_MULTIPLE   // Get up to a page dup values for key at current position (DupFixed).
	Last         = C.MDBX_LAST           // Last item.
	LastDup      = C.MDBX_LAST_DUP       // Position at last value of current key (DupSort).
	Next         = C.MDBX_NEXT           // Next value.
	NextDup      = C.MDBX_NEXT_DUP       // Next value of the current key (DupSort).
	NextMultiple = C.MDBX_NEXT_MULTIPLE  // Get key and up to a page of values from the next cursor position (DupFixed).
	NextNoDup    = C.MDBX_NEXT_NODUP     // The first value of the next key (DupSort).
	Prev         = C.MDBX_PREV           // The previous item.
	PrevDup      = C.MDBX_PREV_DUP       // The previous item of the current key (DupSort).
	PrevNoDup    = C.MDBX_PREV_NODUP     // The last data item of the previous key (DupSort).
	PrevMultiple = C.MDBX_PREV_MULTIPLE  //
	Set          = C.MDBX_SET            // The specified key.
	SetKey       = C.MDBX_SET_KEY        // Get key and data at the specified key.
	SetRange     = C.MDBX_SET_RANGE      // The first key no less than the specified key.
)
View Source 
const (
	// Flags for Txn.Put and Cursor.Put.
	//
	// See mdb_put and mdb_cursor_put.
	Upsert      = C.MDBX_UPSERT      // Replace the item at the current key position (Cursor only)
	Current     = C.MDBX_CURRENT     // Replace the item at the current key position (Cursor only)
	NoDupData   = C.MDBX_NODUPDATA   // Store the key-value pair only if key is not present (DupSort).
	NoOverwrite = C.MDBX_NOOVERWRITE // Store a new key-value pair only if key is not present.
	Append      = C.MDBX_APPEND      // Append an item to the database.
	AppendDup   = C.MDBX_APPENDDUP   // Append an item to the database (DupSort).
	AllDups     = C.MDBX_ALLDUPS
)

The MDB_MULTIPLE and MDB_RESERVE flags are special and do not fit the calling pattern of other calls to Put. They are not exported because they require special methods, PutMultiple and PutReserve in which the flag is implied and does not need to be passed.

View Source 
const (
	EnvDefaults = C.MDBX_ENV_DEFAULTS
	LifoReclaim = C.MDBX_LIFORECLAIM
	//FixedMap    = C.MDBX_FIXEDMAP   // Danger zone. Map memory at a fixed address.
	NoSubdir      = C.MDBX_NOSUBDIR // Argument to Open is a file, not a directory.
	Accede        = C.MDBX_ACCEDE
	Coalesce      = C.MDBX_COALESCE
	Readonly      = C.MDBX_RDONLY     // Used in several functions to denote an object as readonly.
	WriteMap      = C.MDBX_WRITEMAP   // Use a writable memory map.
	NoMetaSync    = C.MDBX_NOMETASYNC // Don't fsync metapage after commit.
	UtterlyNoSync = C.MDBX_UTTERLY_NOSYNC
	SafeNoSync    = C.MDBX_SAFE_NOSYNC
	Durable       = C.MDBX_SYNC_DURABLE
	NoTLS         = C.MDBX_NOTLS // Danger zone. When unset reader locktable slots are tied to their thread.
	//NoLock      = C.MDBX_NOLOCK     // Danger zone. MDBX does not use any locks.
	NoReadahead = C.MDBX_NORDAHEAD // Disable readahead. Requires OS support.
	NoMemInit   = C.MDBX_NOMEMINIT // Disable MDBX memory initialization.
	Exclusive   = C.MDBX_EXCLUSIVE // Disable MDBX memory initialization.
)
View Source 
const (
	MinPageSize = C.MDBX_MIN_PAGESIZE
	MaxPageSize = C.MDBX_MAX_PAGESIZE
	MaxDbi      = C.MDBX_MAX_DBI
)
View Source 
const (
	LogLvlFatal       = C.MDBX_LOG_FATAL
	LogLvlError       = C.MDBX_LOG_ERROR
	LogLvlWarn        = C.MDBX_LOG_WARN
	LogLvlNotice      = C.MDBX_LOG_NOTICE
	LogLvlVerbose     = C.MDBX_LOG_VERBOSE
	LogLvlDebug       = C.MDBX_LOG_DEBUG
	LogLvlTrace       = C.MDBX_LOG_TRACE
	LogLvlExtra       = C.MDBX_LOG_EXTRA
	LogLvlDoNotChange = C.MDBX_LOG_DONTCHANGE
)
View Source 
const (
	DbgAssert          = C.MDBX_DBG_ASSERT
	DbgAudit           = C.MDBX_DBG_AUDIT
	DbgJitter          = C.MDBX_DBG_JITTER
	DbgDump            = C.MDBX_DBG_DUMP
	DbgLegacyMultiOpen = C.MDBX_DBG_LEGACY_MULTIOPEN
	DbgLegacyTxOverlap = C.MDBX_DBG_LEGACY_OVERLAP
	DbgDoNotChange     = C.MDBX_DBG_DONTCHANGE
)
View Source 
const (
	OptMaxDB                        = C.MDBX_opt_max_db
	OptMaxReaders                   = C.MDBX_opt_max_readers
	OptSyncBytes                    = C.MDBX_opt_sync_bytes
	OptSyncPeriod                   = C.MDBX_opt_sync_period
	OptRpAugmentLimit               = C.MDBX_opt_rp_augment_limit
	OptLooseLimit                   = C.MDBX_opt_loose_limit
	OptDpReverseLimit               = C.MDBX_opt_dp_reserve_limit
	OptTxnDpLimit                   = C.MDBX_opt_txn_dp_limit
	OptTxnDpInitial                 = C.MDBX_opt_txn_dp_initial
	OptSpillMaxDenominator          = C.MDBX_opt_spill_max_denominator
	OptSpillMinDenominator          = C.MDBX_opt_spill_min_denominator
	OptSpillParent4ChildDenominator = C.MDBX_opt_spill_parent4child_denominator
	OptMergeThreshold16dot16Percent = C.MDBX_opt_merge_threshold_16dot16_percent
)
View Source 
const (
	ReverseKey = C.MDBX_REVERSEKEY // Use reverse string keys.
	DupSort    = C.MDBX_DUPSORT    // Use sorted duplicates.
	DupFixed   = C.MDBX_DUPFIXED   // Duplicate items have a fixed size (DupSort).
	ReverseDup = C.MDBX_REVERSEDUP // Reverse duplicate values (DupSort).
	Create     = C.MDBX_CREATE     // Createt DB if not already existing.
	DBAccede   = C.MDBX_DB_ACCEDE  // Use sorted duplicates.
)

This flags are used exclusively for Txn.OpenDBI and Txn.OpenRoot. The Create flag must always be supplied when opening a non-root DBI for the first time.

BUG(bmatsuo): MDBX_INTEGERKEY and MDBX_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.

View Source 
const (
	TxRW        = C.MDBX_TXN_READWRITE
	TxRO        = C.MDBX_TXN_RDONLY
	TxPrepareRO = C.MDBX_TXN_RDONLY_PREPARE

	TxTry        = C.MDBX_TXN_TRY
	TxNoMetaSync = C.MDBX_TXN_NOMETASYNC
	TxNoSync     = C.MDBX_TXN_NOSYNC
)
View Source 
const (
	WarmupDefault    = C.MDBX_warmup_default
	WarmupForce      = C.MDBX_warmup_force
	WarmupOomSafe    = C.MDBX_warmup_oomsafe
	WarmupLock       = C.MDBX_warmup_lock
	WarmupTouchLimit = C.MDBX_warmup_touchlimit
	WarmupRelease    = C.MDBX_warmup_release
)
View Source 
const (
	AllowTxOverlap = C.MDBX_DBG_LEGACY_OVERLAP
)
View Source 
const (
	CopyCompact = C.MDBX_CP_COMPACT // Perform compaction while copying
)

These flags are exclusively used in the Env.CopyFlags and Env.CopyFDFlags methods.

View Source 
const Major = C.MDBX_VERSION_MAJOR
View Source 
const Minor = C.MDBX_VERSION_MINOR

Variables

View Source 
var CorruptErrorBacktraceRecommendations = "Otherwise - please create issue in Application repo." // with backtrace or coredump. To create coredump set compile option 'MDBX_FORCE_ASSERTIONS=1' and env variable 'GOTRACEBACK=crash'."
View Source 
var CorruptErrorHardwareRecommendations = "" /* 403-byte string literal not displayed */

App can re-define this messages from init() func

View Source 
var CorruptErrorMessage = CorruptErrorHardwareRecommendations + " " + CorruptErrorBacktraceRecommendations + " " + CorruptErrorRecoveryRecommendations
View Source 
var CorruptErrorRecoveryRecommendations = "" /* 191-byte string literal not displayed */
View Source 
var (
	LoggerDoNotChange = C.MDBX_LOGGER_DONTCHANGE
)

Functions

func CursorToPool added in v0.27.8 

func CursorToPool(c *Cursor)

func IsErrno 

func IsErrno(err error, errno Errno) bool

IsErrno returns true if err's errno is the given errno.

func IsErrnoFn 

func IsErrnoFn(err error, fn func(error) bool) bool

IsErrnoFn calls fn on the error underlying err and returns the result. If err is an *OpError then err.Errno is passed to fn. Otherwise err is passed directly to fn.

func IsErrnoSys 

func IsErrnoSys(err error, errno syscall.Errno) bool

IsErrnoSys returns true if err's errno is the given errno.

func IsKeyExists 

func IsKeyExists(err error) bool

func IsMapFull 

func IsMapFull(err error) bool

IsMapFull returns true if the environment map size has been reached.

func IsNotExist 

func IsNotExist(err error) bool

IsNotExist returns true the path passed to the Env.Open method does not exist.

func IsNotFound 

func IsNotFound(err error) bool

IsNotFound returns true if the key requested in Txn.Get or Cursor.Get does not exist or if the Cursor reached the end of the database without locating a value (EOF).

Types

type Cmp 

type Cmp func(k1, k2 []byte) int

type CmpFunc 

type CmpFunc *C.MDBX_cmp_func

type CommitLatency 

type CommitLatency struct {
	Preparation time.Duration
	GCWallClock time.Duration
	GCCpuTime   time.Duration
	Audit       time.Duration
	Write       time.Duration
	Sync        time.Duration
	Ending      time.Duration
	Whole       time.Duration
	GCDetails   CommitLatencyGC
}

type CommitLatencyGC added in v0.27.9 

type CommitLatencyGC struct {
	/** \brief Время затраченное на чтение и поиск внтури GC
	 *  ради данных пользователя. */
	WorkRtime time.Duration
	/** \brief Количество циклов поиска внутри GC при выделении страниц
	 *  ради данных пользователя. */
	WorkRsteps uint32
	/** \brief Количество запросов на выделение последовательностей страниц
	 *  ради данных пользователя. */
	WorkRxpages uint32
	WorkMajflt  uint32
	SelfMajflt  uint32
	WorkCounter uint32
	SelfCounter uint32

	/** \brief Время затраченное на чтение и поиск внтури GC
	 *  для целей поддержки и обновления самой GC. */
	SelfRtime time.Duration
	SelfXtime time.Duration
	WorkXtime time.Duration
	/** \brief Количество циклов поиска внутри GC при выделении страниц
	 *  для целей поддержки и обновления самой GC. */
	SelfRsteps uint32
	/** \brief Количество запросов на выделение последовательностей страниц
	 *  для самой GC. */
	SelfXpages uint32
	/** \brief Количество итераций обновления GC,
	 *  больше 1 если были повторы/перезапуски. */
	Wloops uint32
	/** \brief Количество итераций слияния записей GC. */
	Coalescences uint32
	Wipes        uint32
	Flushes      uint32
	Kicks        uint32
}

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor operates on data inside a transaction and holds a position in the database.

See MDB_cursor.

func CreateCursor added in v0.27.8 

func CreateCursor() *Cursor

func CursorFromPool added in v0.27.8 

func CursorFromPool() *Cursor

func (*Cursor) Bind added in v0.27.8 

func (c *Cursor) Bind(txn *Txn, db DBI) error

Bind Using of the `mdbx_cursor_bind()` is equivalent to calling mdbx_cursor_renew() but with specifying an arbitrary dbi handle.

func (*Cursor) Close 

func (c *Cursor) Close()

Close the cursor handle and clear the finalizer on c. Cursors belonging to write transactions are closed automatically when the transaction is terminated.

See mdb_cursor_close.

func (*Cursor) Count 

func (c *Cursor) Count() (uint64, error)

Count returns the number of duplicates for the current key.

See mdb_cursor_count.

func (*Cursor) DBI 

func (c *Cursor) DBI() DBI

DBI returns the cursor's database handle. If c has been closed than an invalid DBI is returned.

func (*Cursor) Del 

func (c *Cursor) Del(flags uint) error

Del deletes the item referred to by the cursor from the database.

See mdb_cursor_del.

func (*Cursor) Get 

func (c *Cursor) Get(setkey, setval []byte, op uint) (key, val []byte, err error)

Get retrieves items from the database. If c.Txn().RawRead is true the slices returned by Get reference readonly sections of memory that must not be accessed after the transaction has terminated.

In a Txn with RawRead set to true the Set op causes the returned key to share its memory with setkey (making it writable memory). In a Txn with RawRead set to false the Set op returns key values with memory distinct from setkey, as is always the case when using RawRead.

Get ignores setval if setkey is empty.

See mdb_cursor_get.

func (*Cursor) Put 

func (c *Cursor) Put(key, val []byte, flags uint) error

Put stores an item in the database.

See mdb_cursor_put.

func (*Cursor) PutMulti 

func (c *Cursor) PutMulti(key []byte, page []byte, stride int, flags uint) error

PutMulti stores a set of contiguous items with stride size under key. PutMulti panics if len(page) is not a multiple of stride. The cursor's database must be DupFixed and DupSort.

See mdb_cursor_put.

func (*Cursor) PutReserve 

func (c *Cursor) PutReserve(key []byte, n int, flags uint) ([]byte, error)

PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.

func (*Cursor) Renew 

func (c *Cursor) Renew(txn *Txn) error

Renew associates cursor with txn.

See mdb_cursor_renew.

func (*Cursor) Txn 

func (c *Cursor) Txn() *Txn

Txn returns the cursor's transaction.

type DBI 

type DBI C.MDBX_dbi

DBI is a handle for a database in an Env.

See MDBX_dbi

type Duration16dot16 added in v0.22.16 

type Duration16dot16 uint64

func NewDuration16dot16 added in v0.22.16 

func NewDuration16dot16(duration time.Duration) Duration16dot16

func (Duration16dot16) ToDuration added in v0.22.16 

func (d Duration16dot16) ToDuration() time.Duration

type EnfInfoPageOps added in v0.10.0 

type EnfInfoPageOps struct {
	Newly    uint64 /**< Quantity of a new pages added */
	Cow      uint64 /**< Quantity of pages copied for update */
	Clone    uint64 /**< Quantity of parent's dirty pages clones for nested transactions */
	Split    uint64 /**< Page splits */
	Merge    uint64 /**< Page merges */
	Spill    uint64 /**< Quantity of spilled dirty pages */
	Unspill  uint64 /**< Quantity of unspilled/reloaded pages */
	Wops     uint64 /**< Number of explicit write operations (not a pages) to a disk */
	Minicore uint64 /**< Number of mincore() calls */
	Prefault uint64 /**< Number of prefault write operations (not a pages) */
	Msync    uint64 /**< Number of explicit write operations (not a pages) to a disk */
	Fsync    uint64 /**< Number of explicit write operations (not a pages) to a disk */
}

type Env 

type Env struct {
	// contains filtered or unexported fields
}

Env is opaque structure for a database environment. A DB environment supports multiple databases, all residing in the same shared-memory map.

See MDBX_env.

func NewEnv 

func NewEnv() (*Env, error)

NewEnv allocates and initializes a new Env.

See mdbx_env_create.

func (*Env) BeginTxn 

func (env *Env) BeginTxn(parent *Txn, flags uint) (*Txn, error)

BeginTxn is an unsafe, low-level method to initialize a new transaction on env. The Txn returned by BeginTxn is unmanaged and must be terminated by calling either its Abort or Commit methods to ensure that its resources are released.

BeginTxn does not call runtime.LockOSThread. Unless the Readonly flag is passed goroutines must call runtime.LockOSThread before calling BeginTxn and the returned Txn must not have its methods called from another goroutine. Failure to meet these restrictions can have undefined results that may include deadlocking your application.

Instead of calling BeginTxn users should prefer calling the View and Update methods, which assist in management of Txn objects and provide OS thread locking required for write transactions.

Unterminated transactions can adversly effect database performance and cause the database to grow until the map is full.

See mdbx_txn_begin.

func (*Env) Close 

func (env *Env) Close()

Close shuts down the environment, releases the memory map, and clears the finalizer on env.

See mdbx_env_close.

func (*Env) CloseDBI 

func (env *Env) CloseDBI(db DBI)

CloseDBI closes the database handle, db. Normally calling CloseDBI explicitly is not necessary.

It is the caller's responsibility to serialize calls to CloseDBI.

See mdbx_dbi_close.

func (*Env) Flags 

func (env *Env) Flags() (uint, error)

Flags returns the flags set in the environment.

See mdbx_env_get_flags.

func (*Env) GetOption added in v0.12.0 

func (env *Env) GetOption(option uint) (uint64, error)

func (*Env) GetSyncPeriod added in v0.22.16 

func (env *Env) GetSyncPeriod() (time.Duration, error)

func (*Env) Info 

func (env *Env) Info(txn *Txn) (*EnvInfo, error)

Info returns information about the environment.

See mdbx_env_info. txn - can be nil

func (*Env) MaxKeySize 

func (env *Env) MaxKeySize() int

MaxKeySize returns the maximum allowed length for a key.

See mdbx_env_get_maxkeysize.

func (*Env) Open 

func (env *Env) Open(path string, flags uint, mode os.FileMode) error

Open an environment handle. If this function fails Close() must be called to discard the Env handle. Open passes flags|NoTLS to mdbx_env_open.

See mdbx_env_open.

func (*Env) ReaderCheck 

func (env *Env) ReaderCheck() (int, error)

ReaderCheck clears stale entries from the reader lock table and returns the number of entries cleared.

See mdbx_reader_check()

func (*Env) RunTxn 

func (env *Env) RunTxn(flags uint, fn TxnOp) error

RunTxn creates a new Txn and calls fn with it as an argument. Run commits the transaction if fn returns nil otherwise the transaction is aborted. Because RunTxn terminates the transaction goroutines should not retain references to it or its data after fn returns.

RunTxn does not call runtime.LockOSThread. Unless the Readonly flag is passed the calling goroutine should ensure it is locked to its thread and any goroutines started by fn must not call methods on the Txn object it is passed.

See mdbx_txn_begin.

func (*Env) SetDebug 

func (env *Env) SetDebug(logLvl LogLvl, dbg int, logger *C.MDBX_debug_func) error

func (*Env) SetFlags 

func (env *Env) SetFlags(flags uint) error

SetFlags sets flags in the environment.

See mdbx_env_set_flags.

func (*Env) SetGeometry 

func (env *Env) SetGeometry(sizeLower int, sizeNow int, sizeUpper int, growthStep int, shrinkThreshold int, pageSize int) error

func (*Env) SetOption 

func (env *Env) SetOption(option uint, value uint64) error

func (*Env) SetSyncPeriod added in v0.22.16 

func (env *Env) SetSyncPeriod(value time.Duration) error

func (*Env) Stat 

func (env *Env) Stat() (*Stat, error)

Stat returns statistics about the environment.

See mdbx_env_stat.

func (*Env) Sync 

func (env *Env) Sync(force bool, nonblock bool) error

Sync flushes buffers to disk. If force is true a synchronous flush occurs and ignores any UtterlyNoSync or MapAsync flag on the environment.

See mdbx_env_sync.

func (*Env) UnsetFlags 

func (env *Env) UnsetFlags(flags uint) error

UnsetFlags clears flags in the environment.

See mdbx_env_set_flags.

func (*Env) Update 

func (env *Env) Update(fn TxnOp) error

Update calls fn with a writable transaction. Update commits the transaction if fn returns a nil error otherwise Update aborts the transaction and returns the error.

Update calls runtime.LockOSThread to lock the calling goroutine to its thread and until fn returns and the transaction has been terminated, at which point runtime.UnlockOSThread is called. If the calling goroutine is already known to be locked to a thread, use UpdateLocked instead to avoid premature unlocking of the goroutine.

Neither Update nor UpdateLocked cannot be called safely from a goroutine where it isn't known if runtime.LockOSThread has been called. In such situations writes must either be done in a newly created goroutine which can be safely locked, or through a worker goroutine that accepts updates to apply and delivers transaction results using channels. See the package documentation and examples for more details.

Goroutines created by the operation fn must not use methods on the Txn object that fn is passed. Doing so would have undefined and unpredictable results for your program (likely including data loss, deadlock, etc).

Any call to Commit, Abort, Reset or Renew on a Txn created by Update will panic.

func (*Env) UpdateLocked 

func (env *Env) UpdateLocked(fn TxnOp) error

UpdateLocked behaves like Update but does not lock the calling goroutine to its thread. UpdateLocked should be used if the calling goroutine is already locked to its thread for another purpose.

Neither Update nor UpdateLocked cannot be called safely from a goroutine where it isn't known if runtime.LockOSThread has been called. In such situations writes must either be done in a newly created goroutine which can be safely locked, or through a worker goroutine that accepts updates to apply and delivers transaction results using channels. See the package documentation and examples for more details.

Goroutines created by the operation fn must not use methods on the Txn object that fn is passed. Doing so would have undefined and unpredictable results for your program (likely including data loss, deadlock, etc).

Any call to Commit, Abort, Reset or Renew on a Txn created by UpdateLocked will panic.

func (*Env) View 

func (env *Env) View(fn TxnOp) error

View creates a readonly transaction with a consistent view of the environment and passes it to fn. View terminates its transaction after fn returns. Any error encountered by View is returned.

Unlike with Update transactions, goroutines created by fn are free to call methods on the Txn passed to fn provided they are synchronized in their accesses (e.g. using a mutex or channel).

Any call to Commit, Abort, Reset or Renew on a Txn created by View will panic.

type EnvInfo 

type EnvInfo struct {
	MapSize int64 // Size of the data memory map
	LastPNO int64 // ID of the last used page
	Geo     EnvInfoGeo
	/** Statistics of page operations.
	 * \details Overall statistics of page operations of all (running, completed
	 * and aborted) transactions in the current multi-process session (since the
	 * first process opened the database). */
	PageOps           EnfInfoPageOps
	LastTxnID         int64         // ID of the last committed transaction
	MaxReaders        uint          // maximum number of threads for the environment
	NumReaders        uint          // maximum number of threads used in the environment
	PageSize          uint          //
	SystemPageSize    uint          //
	MiLastPgNo        uint64        //
	AutoSyncThreshold uint          //
	SinceSync         time.Duration //
	AutosyncPeriod    time.Duration //
	SinceReaderCheck  time.Duration //
	Flags             uint          //
}

EnvInfo contains information an environment.

See MDBX_envinfo.

type EnvInfoGeo added in v0.10.0 

type EnvInfoGeo struct {
	Lower   uint64
	Upper   uint64
	Current uint64
	Shrink  uint64
	Grow    uint64
}

type Errno 

type Errno C.int

Errno is an error type that represents the (unique) errno values defined by LMDB. Other errno values (such as EINVAL) are represented with type syscall.Errno. On Windows, LMDB return codes are translated into portable syscall.Errno constants (e.g. syscall.EINVAL, syscall.EACCES, etc.).

Most often helper functions such as IsNotFound may be used instead of dealing with Errno values directly. 

lmdb.IsNotFound(err)
lmdb.IsErrno(err, lmdb.TxnFull)
lmdb.IsErrnoSys(err, syscall.EINVAL)
lmdb.IsErrnoFn(err, os.IsPermission)

const (
	KeyExist        Errno = C.MDBX_KEYEXIST
	NotFound        Errno = C.MDBX_NOTFOUND
	PageNotFound    Errno = C.MDBX_PAGE_NOTFOUND
	Corrupted       Errno = C.MDBX_CORRUPTED
	Panic           Errno = C.MDBX_PANIC
	VersionMismatch Errno = C.MDBX_VERSION_MISMATCH
	Invalid         Errno = C.MDBX_INVALID
	MapFull         Errno = C.MDBX_MAP_FULL
	DBsFull         Errno = C.MDBX_DBS_FULL
	ReadersFull     Errno = C.MDBX_READERS_FULL
	TxnFull         Errno = C.MDBX_TXN_FULL
	CursorFull      Errno = C.MDBX_CURSOR_FULL
	PageFull        Errno = C.MDBX_PAGE_FULL
	Incompatible    Errno = C.MDBX_INCOMPATIBLE
	BadRSlot        Errno = C.MDBX_BAD_RSLOT
	BadTxn          Errno = C.MDBX_BAD_TXN
	BadValSize      Errno = C.MDBX_BAD_VALSIZE
	BadDBI          Errno = C.MDBX_BAD_DBI
	Perm            Errno = C.MDBX_EPERM
)

The most common error codes do not need to be handled explicity. Errors can be checked through helper functions IsNotFound, IsMapFull, etc, Otherwise they should be checked using the IsErrno function instead of direct comparison because they will typically be wrapped with an OpError.

func (Errno) Error 

func (e Errno) Error() string

type LogLvl added in v0.12.0 

type LogLvl = C.MDBX_log_level_t

type Multi 

type Multi struct {
	// contains filtered or unexported fields
}

Multi is a wrapper for a contiguous page of sorted, fixed-length values passed to Cursor.PutMulti or retrieved using Cursor.Get with the GetMultiple/NextMultiple flag.

Multi values are only useful in databases opened with DupSort|DupFixed.

func WrapMulti 

func WrapMulti(page []byte, stride int) *Multi

WrapMulti converts a page of contiguous values with stride size into a Multi. WrapMulti panics if len(page) is not a multiple of stride. 

_, val, _ := cursor.Get(nil, nil, lmdb.FirstDup)
_, page, _ := cursor.Get(nil, nil, lmdb.GetMultiple)
multi := lmdb.WrapMulti(page, len(val))

See mdb_cursor_get and MDB_GET_MULTIPLE.

func (*Multi) Len 

func (m *Multi) Len() int

Len returns the number of values in the Multi.

func (*Multi) Page 

func (m *Multi) Page() []byte

Page returns the Multi page data as a raw slice of bytes with length m.Size().

func (*Multi) Size 

func (m *Multi) Size() int

Size returns the total size of the Multi data and is equal to 

m.Len()*m.Stride()

func (*Multi) Stride 

func (m *Multi) Stride() int

Stride returns the length of an individual value in the m.

func (*Multi) Val 

func (m *Multi) Val(i int) []byte

Val returns the value at index i. Val panics if i is out of range.

func (*Multi) Vals 

func (m *Multi) Vals() [][]byte

Vals returns a slice containing the values in m. The returned slice has length m.Len() and each item has length m.Stride().

type OpError 

type OpError struct {
	Errno error
	Op    string
}

OpError is an error returned by the C API. Not all errors returned by lmdb-go have type OpError but typically they do. The Errno field will either have type Errno or syscall.Errno.

func (*OpError) Error 

func (err *OpError) Error() string

Error implements the error interface.

type Stat 

type Stat struct {
	PSize         uint   // Size of a database page. This is currently the same for all databases.
	Depth         uint   // Depth (height) of the B-tree
	BranchPages   uint64 // Number of internal (non-leaf) pages
	LeafPages     uint64 // Number of leaf pages
	OverflowPages uint64 // Number of overflow pages
	Entries       uint64 // Number of data items
	LastTxId      uint64 // Transaction ID of commited last modification
}

Stat contains database status information.

See MDBX_stat.

type TxInfo 

type TxInfo struct {
	Id uint64 // The ID of the transaction. For a READ-ONLY transaction, this corresponds to the snapshot being read
	/** For READ-ONLY transaction: the lag from a recent MVCC-snapshot, i.e. the
	  number of committed transaction since read transaction started. For WRITE
	  transaction (provided if `scan_rlt=true`): the lag of the oldest reader
	  from current transaction (i.e. at least 1 if any reader running). */
	ReadLag uint64
	/** Used space by this transaction, i.e. corresponding to the last used
	 * database page. */
	SpaceUsed uint64
	/** Current size of database file. */
	SpaceLimitSoft uint64
	/** Upper bound for size the database file, i.e. the value `size_upper`
	  argument of the appropriate call of \ref mdbx_env_set_geometry(). */
	SpaceLimitHard uint64
	/** For READ-ONLY transaction: The total size of the database pages that were
	  retired by committed write transactions after the reader's MVCC-snapshot,
	  i.e. the space which would be freed after the Reader releases the
	  MVCC-snapshot for reuse by completion read transaction.
	  For WRITE transaction: The summarized size of the database pages that were
	  retired for now due Copy-On-Write during this transaction. */
	SpaceRetired uint64
	/** For READ-ONLY transaction: the space available for writer(s) and that
	  must be exhausted for reason to call the Handle-Slow-Readers callback for
	  this read transaction. For WRITE transaction: the space inside transaction
	  that left to `MDBX_TXN_FULL` error. */
	SpaceLeftover uint64
	/** For READ-ONLY transaction (provided if `scan_rlt=true`): The space that
	  actually become available for reuse when only this transaction will be
	  finished.
	  For WRITE transaction: The summarized size of the dirty database
	  pages that generated during this transaction. */
	SpaceDirty uint64

	Spill   uint64
	Unspill uint64
}

type Txn 

type Txn struct {

	// Pooled may be set to true while a Txn is stored in a sync.Pool, after
	// Txn.Reset reset has been called and before Txn.Renew.  This will keep
	// the Txn finalizer from unnecessarily warning the application about
	// finalizations.
	Pooled bool
	// contains filtered or unexported fields
}

Txn is a database transaction in an environment.

WARNING: A writable Txn is not threadsafe and may only be used in the goroutine that created it.

See MDBX_txn.

func (*Txn) Abort 

func (txn *Txn) Abort()

Abort discards pending writes in the transaction and clears the finalizer on txn. A Txn cannot be used again after Abort is called.

See mdbx_txn_abort.

func (*Txn) Cmp 

func (txn *Txn) Cmp(dbi DBI, a []byte, b []byte) int

Cmp - this func follow bytes.Compare return style: The result will be 0 if a==b, -1 if a < b, and +1 if a > b.

func (*Txn) Commit 

func (txn *Txn) Commit() (CommitLatency, error)

Commit persists all transaction operations to the database and clears the finalizer on txn. A Txn cannot be used again after Commit is called.

See mdbx_txn_commit.

func (*Txn) CreateDBI 

func (txn *Txn) CreateDBI(name string) (DBI, error)

CreateDBI is a shorthand for OpenDBI that passed the flag lmdb.Create.

func (*Txn) DCmp 

func (txn *Txn) DCmp(dbi DBI, a []byte, b []byte) int

DCmp - this func follow bytes.Compare return style: The result will be 0 if a==b, -1 if a < b, and +1 if a > b.

func (*Txn) Del 

func (txn *Txn) Del(dbi DBI, key, val []byte) error

Del deletes an item from database dbi. Del ignores val unless dbi has the DupSort flag.

See mdbx_del.

func (*Txn) Drop 

func (txn *Txn) Drop(dbi DBI, del bool) error

Drop empties the database if del is false. Drop deletes and closes the database if del is true.

See mdbx_drop.

func (*Txn) EnvWarmup added in v0.32.0 

func (txn *Txn) EnvWarmup(flags uint, timeout time.Duration) error

func (*Txn) Flags 

func (txn *Txn) Flags(dbi DBI) (uint, error)

Flags returns the database flags for handle dbi.

func (*Txn) Get 

func (txn *Txn) Get(dbi DBI, key []byte) ([]byte, error)

Get retrieves items from database dbi. If txn.RawRead is true the slice returned by Get references a readonly section of memory that must not be accessed after txn has terminated.

See mdbx_get.

func (*Txn) ID 

func (txn *Txn) ID() uint64

ID returns the identifier for txn. A view transaction identifier corresponds to the Env snapshot being viewed and may be shared with other view transactions.

See mdbx_txn_id.

func (*Txn) Info 

func (txn *Txn) Info(scanRlt bool) (*TxInfo, error)

scan_rlt The boolean flag controls the scan of the read lock 

table to provide complete information. Such scan
is relatively expensive and you can avoid it
if corresponding fields are not needed.
See description of \ref MDBX_txn_info.

func (*Txn) ListDBI added in v0.15.0 

func (txn *Txn) ListDBI() (res []string, err error)

ListDBI - return all dbi names. they stored as keys of un-named (main) dbi

func (*Txn) OpenCursor 

func (txn *Txn) OpenCursor(dbi DBI) (*Cursor, error)

OpenCursor allocates and initializes a Cursor to database dbi.

See mdbx_cursor_open.

func (*Txn) OpenDBI 

func (txn *Txn) OpenDBI(name string, flags uint, cmp, dcmp CmpFunc) (DBI, error)

OpenDBI opens a named database in the environment. An error is returned if name is empty. The DBI returned by OpenDBI can be used in other transactions but not before Txn has terminated.

OpenDBI can only be called after env.SetMaxDBs() has been called to set the maximum number of named databases.

The C API uses null terminated strings for database names. A consequence is that names cannot contain null bytes themselves. OpenDBI does not check for null bytes in the name argument.

See mdbx_dbi_open.

func (*Txn) OpenDBISimple 

func (txn *Txn) OpenDBISimple(name string, flags uint) (DBI, error)

func (*Txn) OpenRoot 

func (txn *Txn) OpenRoot(flags uint) (DBI, error)

OpenRoot opens the root database. OpenRoot behaves similarly to OpenDBI but does not require env.SetMaxDBs() to be called beforehand. And, OpenRoot can be called without flags in a View transaction.

func (*Txn) Put 

func (txn *Txn) Put(dbi DBI, key, val []byte, flags uint) error

Put stores an item in database dbi.

See mdbx_put.

func (*Txn) PutReserve 

func (txn *Txn) PutReserve(dbi DBI, key []byte, n int, flags uint) ([]byte, error)

PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.

func (*Txn) Renew 

func (txn *Txn) Renew() error

Renew reuses a transaction that was previously reset by calling txn.Reset(). Renew panics if txn is managed by Update, View, etc.

See mdbx_txn_renew.

func (*Txn) Reset 

func (txn *Txn) Reset()

Reset aborts the transaction clears internal state so the transaction may be reused by calling Renew. If txn is not going to be reused txn.Abort() must be called to release its slot in the lock table and free its memory. Reset panics if txn is managed by Update, View, etc.

See mdbx_txn_reset.

func (*Txn) RunOp 

func (txn *Txn) RunOp(fn TxnOp, terminate bool) error

RunOp executes fn with txn as an argument. During the execution of fn no goroutine may call the Commit, Abort, Reset, and Renew methods on txn. RunOp returns the result of fn without any further action. RunOp will not abort txn if fn returns an error, unless terminate is true. If terminate is true then RunOp will attempt to commit txn if fn is successful, otherwise RunOp will abort txn before returning any failure encountered.

RunOp primarily exists to allow applications and other packages to provide variants of the managed transactions provided by lmdb (i.e. View, Update, etc). For example, the lmdbpool package uses RunOp to provide an Txn-friendly sync.Pool and a function analogous to Env.View that uses transactions from that pool.

func (*Txn) Sequence 

func (txn *Txn) Sequence(dbi DBI, increment uint64) (uint64, error)

func (*Txn) StatDBI 

func (txn *Txn) StatDBI(dbi DBI) (*Stat, error)

func (*Txn) Sub 

func (txn *Txn) Sub(fn TxnOp) error

Sub executes fn in a subtransaction. Sub commits the subtransaction iff a nil error is returned by fn and otherwise aborts it. Sub returns any error it encounters.

Sub may only be called on an Update Txn (one created without the Readonly flag). Calling Sub on a View transaction will return an error. Sub assumes the calling goroutine is locked to an OS thread and will not call runtime.LockOSThread.

Any call to Abort, Commit, Renew, or Reset on a Txn created by Sub will panic.

type TxnOp 

type TxnOp func(txn *Txn) error

TxnOp is an operation applied to a managed transaction. The Txn passed to a TxnOp is managed and the operation must not call Commit, Abort, Renew, or Reset on it.

IMPORTANT: TxnOps that write to the database (those passed to Env.Update or Txn.Sub) must not use the Txn in another goroutine (passing it directly or otherwise through closure). Doing so has undefined results.

Notes

Bugs

  • MDBX_INTEGERKEY and MDBX_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.

Source Files

View all Source files

Directories

Path Synopsis
Package mdbxarch contains some architecture detection constants.
 Package mdbxarch contains some architecture detection constants.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.