store

README

This package holds the storage abstraction needed for FluxDB to function.

Historically, there was:

• One native Bigtable implementation
• Then we had a few implementations
• Then we settled on using kvdb as an abstraction, to use whatever backing KV store.

Documentation

Index

• Variables
• type Batch
• type KVStore
• type OnBlockRef
• type OnTabletRow

Variables

var BreakScan = errors.New("break scan")

BreakScan error can be used on scanning function to notify termination of scanning

var ErrNotFound = errors.New("not found")

Types

type Batch

type Batch interface {
	Flush(ctx context.Context) error
	FlushIfFull(ctx context.Context) error

	// FIXME: Maybe the batch "adder/setter" should not event care about the key and compute
	//        it straight? Since this is per storage engine, it would be a good place since
	//        all saved element would pass through those methods...
	SetABI(key string, value []byte)
	SetRow(key string, value []byte)
	SetLast(key string, value []byte)
	SetIndex(key string, value []byte)

	Reset()
}

type KVStore

type KVStore interface {
	Close() error

	// NewBatch returns the batch implementation suitable for the underlying store.
	//
	// FIXME: For now, we kept the `logger` parameter, not clear the intent was here. Let's
	//        decide if this was required later on when we are close to finish the refactoring.
	NewBatch(logger *zap.Logger) Batch

	FetchABI(ctx context.Context, prefixKey, keyStart, keyEnd string) (rowKey string, rawABI []byte, err error)

	FetchIndex(ctx context.Context, tableKey, prefixKey, keyStart string) (rowKey string, rawIndex []byte, err error)

	HasTabletRow(ctx context.Context, keyPrefix string) (exists bool, err error)

	FetchTabletRow(ctx context.Context, key string, onTabletRow OnTabletRow) error

	FetchTabletRows(ctx context.Context, keys []string, onTabletRow OnTabletRow) error

	ScanTabletRows(ctx context.Context, keyStart, keyEnd string, onTabletRow OnTabletRow) error

	// FetchLastWrittenBlock returns the latest written block reference that was correctly
	// committed to the storage system.
	//
	// If no block was ever written yet, this must return `nil, ErrNotFound`.
	FetchLastWrittenBlock(ctx context.Context, key string) (out bstream.BlockRef, err error)

	ScanLastShardsWrittenBlock(ctx context.Context, keyPrefix string, onBlockRef OnBlockRef) error
}

KVStore represents the abstraction needed by FluxDB to correctly use different underlying KV storage engine.

TODO: For now, most functions receive the actual pre-computed key to fetch or to write.

While we affine the interface, we will see if make it lower-level (i.e. `Get(key)`
directly) or if we keep higher level and defer more job into the implementation
(i.e. removing the key parameters).

type OnBlockRef

type OnBlockRef func(key string, blockRef bstream.BlockRef) error

type OnTabletRow

type OnTabletRow func(key string, value []byte) error