strategy

Documentation

Overview

Package strategy implements various LMDB data insert strategies.

A strategy is responsible for updating the LMDB with a new snapshot or delta. This package contains implementations of insert strategies and a function to pick the best strategy for a given situation.

Index

• Constants
• Variables
• func Append(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error
• func EmptyPut(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error
• func IterPut(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error
• func IterUpdate(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error
• func Put(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error
• func Update(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error
• type Facts
• type Func
  • func Pick(f Facts) (Func, error)
• type Iterator

Constants

const LMDBIntegerKeyFlag = 0x08 // not defined in Go bindings

const LMDBMaxKeySize = 511

Variables

var ErrNotSorted = errors.New("keys not sorted")

ErrNotSorted is returned when keys are found to be not sorted and the strategy requires it.

var ErrSkip = errors.New("skip")

ErrSkip is an error returned by an Iterator to skip a Merge or Clean for an item.

Functions

func Append

func Append(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

Append implements the Append-strategy.

Use the MDB_APPEND flag to directly append items to the database. This is the fastest way to insert items and is great for the very first snapshot load.

Prerequisites:

- Database is empty - Sorted input

Uses: make the very first snapshot load fast.

func EmptyPut

func EmptyPut(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

EmptyPut firsts empties the DBI, then uses Put to insert all the values.

func IterPut

func IterPut(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

IterPut implements the IterPut strategy.

Iter over all existing keys and update all URLs we are processing:

- Add new urls and replace values of existing, just like in the `Put` strategy. - But also remove keys that are not present in the snapshot.

Once we have reached the end of the database, we use `Append` instead of `Put`.

Prerequisites:

- Sorted input

func IterUpdate

func IterUpdate(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

IterUpdate implements the IterUpdate strategy.

Iter over all existing keys and update all URLs we are processing:

- Add new keys and data, just like in the `update` strategy. - But also remove codes from urls that have disappeared for this namespace.

Once we have reached the end of the database, we use `Update` instead of `Append`.

Prerequisites:

- Sorted input

Uses: merging data

func Put

func Put(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

Put implements the Put strategy.

Directly put the entries from the CSV into LMDB, overwriting any existing one.

Prerequisites:

- Single namespace - Sorted input not required

Uses: unsorted first snapshot load (slow for large snapshots).

func Update

func Update(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

Update implements the Update strategy.

- Sorted input not required

Types

type Facts

type Facts struct {
	// Is the database currently completely empty?
	IsEmpty bool
}

Facts are used by Pick to pick a strategy.

type Func

type Func func(txn *lmdb.Txn, dbi lmdb.DBI, it Iterator) error

Func defines the signature of functions implementing a strategy

func Pick

func Pick(f Facts) (Func, error)

Pick picks the best insert strategy based on Facts. FIXME: Trivial choice, remove

type Iterator

type Iterator interface {
	// Next returns the next LMDB key to insert/update. Calling it invalidates
	// earlier byte slides received from any of these methods.
	// It returns io.EOF when no more entries are available.
	Next() (key []byte, err error)
	// Merge takes the existing LMDB value for the current key, merges it
	// with the value we want to insert, and then returns the result.
	// It must never return an empty slice, and instead return nil.
	Merge(oldval []byte) (val []byte, err error)
	// Clean removes all values that refer to the current snapshot from the LMDB
	// value and returns the result.
	// It must never return an empty slice, and instead return nil.
	Clean(oldval []byte) (val []byte, err error)
}

Iterator is the interface expected by the strategy implementations. The Iterator is responsible for iterating of snapshots and deltas, and for the proper serialization and merging of values. Any progress stats needed must be implemented by the Iterator.