lmdbpool

Documentation

Overview

Package lmdbpool provides a TxnPool type that allows lmdb.Readonly transactions to safely be reused by other goroutines when the goroutine that created the transaction no longer has a use for it. The TxnPool type has benefits that would be absent in a naive use of sync.Pool with lmdb.Txn types.

Naively reusing lmdb.Readonly transactions can cause updates to continually allocate more pages for the database instead of reusing stale pages. The TxnPool type tracks transaction ids to make sure that lmdb.Readonly transactions are not reused when they are known to be holding stale pages which could be reclaimed by LMDB.

A general downside of pooling lmdb.Readonly transactions using a sync.Pool in applications with a very high rate of transacions is that the number of readers in an environment can be significantly higher than the number of goroutines actively trying to read from that environment. Because of this it is possible that applications may need to increase the maximum number of readers allowed in the environment at initialization time.

err := env.SetMaxReaders(maxReaders)

In a naive pooling implementation an application compiled with the -race flag may require an extremely large number of open readers. The TxnPool type attempts to keep the value required for Env.SetMaxReaders as low as possible in the presence of -race but there is a limited amount that can be done for a concurrent workload with a rapid enough rate of transactions.

Index

• type TxnPool
  • func NewTxnPool(env *lmdb.Env) *TxnPool
  • func (p *TxnPool) Abort(txn *lmdb.Txn)
  • func (p *TxnPool) BeginTxn(flags uint) (*lmdb.Txn, error)
  • func (p *TxnPool) Close()
  • func (p *TxnPool) CommitID(id uintptr)
  • func (p *TxnPool) Update(fn lmdb.TxnOp) error
  • func (p *TxnPool) View(fn lmdb.TxnOp) error
• type UpdateHandling

Types

type TxnPool

type TxnPool struct {
	// UpdateHandling determines how a TxnPool behaves after updates have been
	// committed.  It is not safe to modify UpdateHandling if TxnPool is being
	// used concurrently.
	UpdateHandling UpdateHandling
	// contains filtered or unexported fields
}

TxnPool is a pool for reusing transactions through their Reset and Renew methods. However, even though TxnPool can only reuse lmdb.Readonly transactions it this way it should be used to create and terminate all Txns if it is used at all. Update transactions can cause LMDB to use excessive numbers of pages when there are long-lived lmdb.Readonly transactions in a TxnPool. Executing all transactions using the TxnPool allows it to track updates and prevent long-lived updates from causing excessive disk utilization.

func NewTxnPool

func NewTxnPool(env *lmdb.Env) *TxnPool

NewTxnPool initializes returns a new TxnPool.

func (*TxnPool) Abort

func (p *TxnPool) Abort(txn *lmdb.Txn)

Abort aborts the txn and allows it to be reused if possible. Abort must only be passed lmdb.Txn objects which it returned from a call to BeginTxn. Aborting a transaction created through other means will have undefined results.

func (*TxnPool) BeginTxn

func (p *TxnPool) BeginTxn(flags uint) (*lmdb.Txn, error)

BeginTxn is analogous to the BeginTxn method on lmdb.Env but may only be used to create lmdb.Readonly transactions. Any call to BeginTxn that does not include lmdb.Readonly will return an error

func (*TxnPool) Close

func (p *TxnPool) Close()

Close flushes the pool of transactions and aborts them to free resources so that the pool Env may be closed.

func (*TxnPool) CommitID

func (p *TxnPool) CommitID(id uintptr)

CommitID sets the TxnPool's last-known transaction id to invalidate previously created lmdb.Readonly transactions and prevent their reuse.

CommitID should only be called if p is not used to create/commit update transactions.

func (*TxnPool) Update

func (p *TxnPool) Update(fn lmdb.TxnOp) error

Update is analogous to the Update method on lmdb.Env.

func (*TxnPool) View

func (p *TxnPool) View(fn lmdb.TxnOp) error

View is analogous to the View method on lmdb.Env.

type UpdateHandling

type UpdateHandling uint

UpdateHandling describes how a TxnPool handles existing lmdb.Readonly transactions when an environment update occurs. Applications with a high rate of large updates may need to choose non-default settings to reduce their storage requirements at the cost of read throughput.

The zero-value of UpdateHandling causes a TxnPool to ignore all updates and defers to the application and the lmdb.Txn finalizers clear stale readers (pulling an lmdb.Readonly transaction out of the pool is enough to release its stale pages).

const (
	// HandleOutstanding causes a TxnPool to abort any lmdb.Readonly
	// transactions that are being returned to the pool after an update.
	HandleOutstanding UpdateHandling = 1 << iota

	// HandleIdle causes a TxnPool to actively attempt aborting idle
	// transactions in the sync.Pool after an update has been committed.  There
	// is no guarantee when using AbortIdle that all idle readers will be
	// aborted.
	HandleIdle

	// HandleRenew modifies how other UpdateHandling flags are interpretted and
	// causes a TxnPool to renew transactions and put them back in the pool
	// instead of aborting them.
	HandleRenew

	// HandleAll is a convenient alias for the combination of HandleOutstanding
	// and HandleIdle.
	HandleAll = HandleOutstanding | HandleIdle
)