Documentation
¶
Overview ¶
Package lmdbsync provides advanced synchronization for LMDB environments at the cost of performance. The package provides a drop-in replacement for *lmdb.Env that can be used in situations where the database may be resized or where the flag lmdb.NoLock is used.
Bypassing an Env's methods to access the underlying lmdb.Env is not safe. The severity of such usage depends such behavior should be strictly avoided as it may produce undefined behavior from the LMDB C library.
Resizing the environment ¶
The Env type synchronizes all calls to Env.SetMapSize so that it may, with some caveats, be safely called in the presence of concurrent transactions after an environment has been opened. All running transactions must complete before the method will be called on the underlying lmdb.Env.
If an open transaction depends on a change in map size then the Env will deadlock and block all future transactions. When using a Handler to automatically resize the map this implies the restriction that transactions must terminate independently of the creation and termination of other transactions to avoid deadlock
In the simplest example, a view transaction that attempts an update on the underlying Env will deadlock the environment if the map is full and a Handler attempts to resize the map so the update may be retried.
env.View(func(txn *lmdb.Txn) (err error) {
v, err := txn.Get(db, key)
if err != nil {
return err
}
err = env.Update(func(txn *lmdb.Txn) (err error) { // deadlock on lmdb.MapFull!
txn.Put(dbi, key, append(v, b...))
})
return err
}
The update should instead be prepared inside the view and then executed following its termination. This removes the implicit dependence of the view on calls to Env.SetMapSize().
var v []byte
env.View(func(txn *lmdb.Txn) (err error) {
// RawRead isn't used because the value will be used outside the
// transaction.
v, err = txn.Get(db, key)
if err != nil {
return err
}
return nil
}
if err != nil {
// ...
}
err = env.Update(func(txn *lmdb.Txn) (err error) { // no deadlock, even if env is resized!
txn.Put(dbi, key, append(v, b...))
})
The developers of LMDB officially recommend against applications changing the memory map size for an open database. It requires careful synchronization by all processes accessing the database file. And, a large memory map will not affect disk usage on operating systems that support sparse files (e.g. Linux, not OS X).
See mdb_env_set_mapsize.
Multi-processing (MapResized)
Using the Handler interface provided by the package MapResizedHandler can be used to automatically resize an enviornment when a lmdb.MapResized error is encountered. Usage of the MapResizedHandler puts important caveats on how one can safely work with transactions. See the function documentation for more detailed information.
When other processes may change an environment's map size it is extremely important to ensure that transactions terminate independent of all other transactions. The MapResized error may be returned at the beginning of any transaction.
See mdb_txn_begin and MDB_MAP_RESIZED.
MapFull ¶
Similar to the MapResizedHandler the MapFullHandler will automatically resize the map and retry transactions when a MapFull error is encountered. Usage of the MapFullHandler puts important caveats on how one can safely work with transactions. See the function documentation for more detailed information.
The caveats on transactions are lessened if lmdb.MapFull is the only error being handled (when multi-processing is not a concern). The only requirement then is that view transactions not depend on the termination of updates transactions.
See mdb_env_set_mapsize and MDB_MAP_FULL.
NoLock ¶
When the lmdb.NoLock flag is set on an environment Env handles all transaction synchronization using Go structures and is an experimental feature. It is unclear what benefits this provides.
Usage of lmdb.NoLock requires that update transactions acquire an exclusive lock on the environment. In such cases it is required that view transactions execute independently of update transactions, a requirement more strict than that from handling MapFull.
See mdb_env_open and MDB_NOLOCK.
Index ¶
- Variables
- type Bag
- type Env
- func (r *Env) BeginTxn(parent *lmdb.Txn, flags uint) (*lmdb.Txn, error)
- func (r *Env) Open(path string, flags uint, mode os.FileMode) error
- func (r *Env) RunTxn(flags uint, op lmdb.TxnOp) (err error)
- func (r *Env) SetMapSize(size int64) error
- func (r *Env) Update(op lmdb.TxnOp) error
- func (r *Env) UpdateLocked(op lmdb.TxnOp) error
- func (r *Env) View(op lmdb.TxnOp) error
- func (r *Env) WithHandler(h Handler) TxnRunner
- type Handler
- type HandlerChain
- type MapFullFunc
- type TxnRunner
Constants ¶
This section is empty.
Variables ¶
var DefaultDelayRepeatResize = time.Millisecond
DefaultDelayRepeatResize is the maximum number of consecutive MapResize will be handled by MapResizedHandler before it stops attempting to handle it and returns MapResize to the caller.
var DefaultRetryResize = 2
DefaultRetryResize is the default number of times to retry a transaction that is returning repeatedly MapResized. This signifies rapid database growth from another process or some bug/corruption in memory.
If DefaultRetryResize is less than zero the transaction will be retried indefinitely.
var ErrTxnRetry = errors.New("lmdbsync: retry failed txn")
ErrTxnRetry is returned by a Handler to have the Env retry the transaction.
Functions ¶
This section is empty.
Types ¶
type Bag ¶
type Bag interface {
// value returns a value associated with key in the Bag.
Value(key interface{}) interface{}
}
Bag is a simple context object for Handlers. Bags are immutable, though by storing references in a Bag the contents themselves may be mutable.
func Background ¶
func Background() Bag
Background returns an empty Bag that can be used in future calls to BagWith.
type Env ¶
type Env struct {
*lmdb.Env
Handlers HandlerChain
// contains filtered or unexported fields
}
Env wraps an *lmdb.Env, receiving all the same methods and proxying some to provide transaction management. Transactions run by an Env handle lmdb.MapResized error transparently through additional synchronization. Additionally, Env is safe to use on environments setting the lmdb.NoLock flag. When in NoLock mode write transactions block all read transactions from running (in addition to blocking other write transactions like a normal lmdb.Env would).
Env proxies several methods to provide synchronization required for safe operation in some scenarios. It is important not byprass proxies and call the methods directly on the underlying lmdb.Env or synchronization may be interfered with. Calling proxied methods directly on the lmdb.Env may result in poor transaction performance or unspecified behavior in from the C library.
func BagEnv ¶
BagEnv returns the Env corresponding to a Bag in the HandleTxnErr method of a Handler.
func NewEnv ¶
NewEnv returns an newly allocated Env that wraps env. If env is nil then lmdb.NewEnv() will be called to allocate an lmdb.Env.
func (*Env) BeginTxn ¶ added in v1.1.1
BeginTxn overrides the r.Env.BeginTxn and always returns an error. An unmanaged transaction.
func (*Env) Open ¶
Open is a proxy for r.Env.Open() that detects the lmdb.NoLock flag to properly manage transaction synchronization.
func (*Env) RunTxn ¶
RunTxn is a proxy for r.Env.RunTxn().
If lmdb.NoLock is set on r.Env then RunTxn will block while other updates are in progress, regardless of flags.
If RunTxn returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.
func (*Env) SetMapSize ¶
SetMapSize is a proxy for r.Env.SetMapSize() that blocks while concurrent transactions are in progress.
func (*Env) Update ¶
Update is a proxy for r.Env.RunTxn().
If lmdb.NoLock is set on r.Env then Update blocks until all other transactions have terminated and blocks all other transactions from running while in progress (including readonly transactions).
If Update returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.
func (*Env) UpdateLocked ¶
UpdateLocked is a proxy for r.Env.RunTxn().
If lmdb.NoLock is set on r.Env then UpdateLocked blocks until all other transactions have terminated and blocks all other transactions from running while in progress (including readonly transactions).
If UpdateLocked returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.
func (*Env) View ¶
View is a proxy for r.Env.RunTxn().
If lmdb.NoLock is set on r.Env then View will block until any running update completes.
If View returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.
func (*Env) WithHandler ¶
WithHandler returns a TxnRunner than handles transaction errors r.Handlers chained with h.
type Handler ¶
Handler can intercept errors returned by a transaction and handle them in an application-specific way, including by resizing the environment and retrying the transaction by returning ErrTxnRetry.
func MapFullHandler ¶
func MapFullHandler(fn MapFullFunc) Handler
MapFullHandler returns a Handler that retries Txns that failed due to MapFull errors by increasing the environment map size according to fn.
A lmdb.TxnOp which is handled by the returned Handler will execute multiple times in the occurrance of a MapFull error.
When MapFullHandler is in use update transactions must not be nested inside view transactions. Resizing the database requires all transactions to terminate first. If any transactions wait for update transactions to complete they may deadlock in the presence of a MapFull error.
func MapResizedHandler ¶
MapResizedHandler returns a Handler than transparently retrie Txns that failed to start due to MapResized errors.
When MapResizeHandler is in use transactions must not be nested inside other transactions. Adopting the new map size requires all transactions to terminate first. If any transactions wait for other transactions to complete they may deadlock in the presence of a MapResized error.
type HandlerChain ¶
type HandlerChain []Handler
HandlerChain is a Handler implementation that iteratively calls each handler in the underlying slice when handling an error.
func (HandlerChain) Append ¶
func (c HandlerChain) Append(h ...Handler) HandlerChain
Append returns a new HandlerChain that will evaluate h in sequence after the Handlers already in C are evaluated. Append does not modify the storage undelying c.
func (HandlerChain) HandleTxnErr ¶
func (c HandlerChain) HandleTxnErr(b Bag, err error) (Bag, error)
HandleTxnErr implements the Handler interface. Each handler in c processes the Bag and error returned by the previous handler.
type MapFullFunc ¶
MapFullFunc is a function for resizing a memory map after it has become full. The function receives the current map size as its argument and returns a new map size. The new size will only be applied if the second return value is true.
type TxnRunner ¶
type TxnRunner interface {
RunTxn(flags uint, op lmdb.TxnOp) error
View(op lmdb.TxnOp) error
Update(op lmdb.TxnOp) error
UpdateLocked(op lmdb.TxnOp) error
WithHandler(h Handler) TxnRunner
}
TxnRunner is an interface for types that can run lmdb transactions. TxnRunner is satisfied by Env.
Source Files