Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EstimateMapBytesSize

func EstimateMapBytesSize(m *Map) int64

EstimateMapBytesSize estimates the size (in bytes) of the results map. It's only an estimate because its impossible to know if some of the references like the series name as well as tags are exclusive to this object or shared with other structures in memory.

Types

type CopyFn

type CopyFn func(ident.ID) ident.ID

CopyFn is the copy key function to execute when copying the key.

type DataBootstrapResult

type DataBootstrapResult interface {
	// Unfulfilled is the unfulfilled time ranges for the bootstrap.
	Unfulfilled() ShardTimeRanges

	// SetUnfulfilled sets the current unfulfilled shard time ranges.
	SetUnfulfilled(unfulfilled ShardTimeRanges)
}

DataBootstrapResult is the result of a bootstrap of series data.

func MergedDataBootstrapResult

func MergedDataBootstrapResult(i, j DataBootstrapResult) DataBootstrapResult

MergedDataBootstrapResult returns a merged result of two bootstrap results. It is a mutating function that mutates the larger result by adding the smaller result to it and then finally returns the mutated result.

func NewDataBootstrapResult

func NewDataBootstrapResult() DataBootstrapResult

NewDataBootstrapResult creates a new result.

type DatabaseSeriesBlocks

type DatabaseSeriesBlocks struct {
	ID     ident.ID
	Tags   ident.Tags
	Blocks block.DatabaseSeriesBlocks
}

DatabaseSeriesBlocks represents a series of blocks and a associated series ID.

type DocumentsBuilderAllocator

type DocumentsBuilderAllocator func() (segment.DocumentsBuilder, error)

DocumentsBuilderAllocator allocates a new DocumentsBuilder type when creating a bootstrap result to return to the index.

func NewDefaultDocumentsBuilderAllocator

func NewDefaultDocumentsBuilderAllocator() DocumentsBuilderAllocator

NewDefaultDocumentsBuilderAllocator returns a default mutable segment allocator.

type EqualsFn

type EqualsFn func(ident.ID, ident.ID) bool

EqualsFn is the equals key function to execute when detecting equality of a key.

type FinalizeFn

type FinalizeFn func(ident.ID)

FinalizeFn is the finalize key function to execute when finished with a key.

type HashFn

type HashFn func(ident.ID) MapHash

HashFn is the hash function to execute when hashing a key.

type IndexBlock

type IndexBlock struct {
	// contains filtered or unexported fields
}

IndexBlock is an index block for a index volume type.

func NewIndexBlock

func NewIndexBlock(
	segments []Segment,
	fulfilled ShardTimeRanges,
) IndexBlock

NewIndexBlock returns a new bootstrap index block result.

func (IndexBlock) Fulfilled

func (b IndexBlock) Fulfilled() ShardTimeRanges

Fulfilled returns the fulfilled time ranges by this index block.

func (IndexBlock) Merged

func (b IndexBlock) Merged(other IndexBlock) IndexBlock

Merged returns a new merged index block, currently it just appends the list of segments from the other index block and the caller merges as they see necessary.

func (IndexBlock) Segments

func (b IndexBlock) Segments() []Segment

Segments returns the segments.

type IndexBlockByVolumeType

type IndexBlockByVolumeType struct {
	// contains filtered or unexported fields
}

IndexBlockByVolumeType contains the bootstrap data structures for an index block by volume type.

func NewIndexBlockByVolumeType

func NewIndexBlockByVolumeType(blockStart time.Time) IndexBlockByVolumeType

NewIndexBlockByVolumeType returns a new bootstrap index blocks by volume type result.

func (IndexBlockByVolumeType) BlockStart

func (b IndexBlockByVolumeType) BlockStart() time.Time

BlockStart returns the block start.

func (IndexBlockByVolumeType) GetBlock

func (b IndexBlockByVolumeType) GetBlock(volumeType persist.IndexVolumeType) (IndexBlock, bool)

GetBlock returns an IndexBlock for volumeType.

func (IndexBlockByVolumeType) Iter

Iter returns the underlying iterable map data.

func (IndexBlockByVolumeType) Merged

Merged returns a new merged index block by volume type. It merges the underlying index blocks together by index volume type.

func (IndexBlockByVolumeType) SetBlock

func (b IndexBlockByVolumeType) SetBlock(volumeType persist.IndexVolumeType, block IndexBlock)

SetBlock sets an IndexBlock for volumeType.

type IndexBootstrapResult

type IndexBootstrapResult interface {
	// IndexResults returns a map of all index block results.
	IndexResults() IndexResults

	// Unfulfilled is the unfulfilled time ranges for the bootstrap.
	Unfulfilled() ShardTimeRanges

	// SetUnfulfilled sets the current unfulfilled shard time ranges.
	SetUnfulfilled(unfulfilled ShardTimeRanges)

	// Add adds an index block result.
	Add(blocks IndexBlockByVolumeType, unfulfilled ShardTimeRanges)

	// NumSeries returns the total number of series across all segments.
	NumSeries() int
}

IndexBootstrapResult is the result of a bootstrap of series index metadata.

func MergedIndexBootstrapResult

func MergedIndexBootstrapResult(i, j IndexBootstrapResult) IndexBootstrapResult

MergedIndexBootstrapResult returns a merged result of two bootstrap results. It is a mutating function that mutates the larger result by adding the smaller result to it and then finally returns the mutated result.

func NewIndexBootstrapResult

func NewIndexBootstrapResult() IndexBootstrapResult

NewIndexBootstrapResult returns a new index bootstrap result.

type IndexBuilder

type IndexBuilder struct {
	// contains filtered or unexported fields
}

IndexBuilder wraps a index segment builder w/ batching.

func NewIndexBuilder

func NewIndexBuilder(builder segment.DocumentsBuilder) *IndexBuilder

NewIndexBuilder creates a wrapped locakble index seg builder.

func (*IndexBuilder) Builder

func (b *IndexBuilder) Builder() segment.DocumentsBuilder

Builder returns the underlying index segment docs builder.

func (*IndexBuilder) FlushBatch

func (b *IndexBuilder) FlushBatch(batch []doc.Document) ([]doc.Document, error)

FlushBatch flushes a batch of documents to the underlying segment builder.

type IndexResults

type IndexResults map[xtime.UnixNano]IndexBlockByVolumeType

IndexResults is a set of index blocks indexed by block start.

func (IndexResults) Add

func (r IndexResults) Add(blocks IndexBlockByVolumeType)

Add will add an index block to the collection, merging if one already exists.

func (IndexResults) AddBlockIfNotExists

func (r IndexResults) AddBlockIfNotExists(
	t time.Time,
	idxopts namespace.IndexOptions,
)

AddBlockIfNotExists adds an index block if it does not already exist to the index results.

func (IndexResults) AddResults

func (r IndexResults) AddResults(other IndexResults)

AddResults will add another set of index results to the collection, merging if index blocks already exists.

func (IndexResults) MarkFulfilled

func (r IndexResults) MarkFulfilled(
	t time.Time,
	fulfilled ShardTimeRanges,
	indexVolumeType persist.IndexVolumeType,
	idxopts namespace.IndexOptions,
) error

MarkFulfilled will mark an index block as fulfilled, either partially or wholly as specified by the shard time ranges passed.

type Map

type Map struct {
	// contains filtered or unexported fields
}

Map uses the genny package to provide a generic hash map that can be specialized by running the following command from this root of the repository: ``` make hashmap-gen pkg=outpkg key_type=Type value_type=Type out_dir=/tmp ``` Or if you would like to use bytes or ident.ID as keys you can use the partially specialized maps to generate your own maps as well: ``` make byteshashmap-gen pkg=outpkg value_type=Type out_dir=/tmp make idhashmap-gen pkg=outpkg value_type=Type out_dir=/tmp ``` This will output to stdout the generated source file to use for your map. It uses linear probing by incrementing the number of the hash created when hashing the identifier if there is a collision. Map is a value type and not an interface to allow for less painful upgrades when adding/removing methods, it is not likely to need mocking so an interface would not be super useful either.

func NewMap

func NewMap(opts MapOptions) *Map

NewMap returns a new byte keyed map.

func (*Map) Contains

func (m *Map) Contains(k ident.ID) bool

Contains returns true if value exists for key, false otherwise, it is shorthand for a call to Get that doesn't return the value.

func (*Map) Delete

func (m *Map) Delete(k ident.ID)

Delete will remove a value set in the map for the specified key.

func (*Map) Get

func (m *Map) Get(k ident.ID) (DatabaseSeriesBlocks, bool)

Get returns a value in the map for an identifier if found.

func (*Map) Iter

func (m *Map) Iter() map[MapHash]MapEntry

Iter provides the underlying map to allow for using a native Go for loop to iterate the map, however callers should only ever read and not write the map.

func (*Map) Len

func (m *Map) Len() int

Len returns the number of map entries in the map.

func (*Map) Reallocate

func (m *Map) Reallocate()

Reallocate will avoid deleting all keys and reallocate a new map, this is useful if you believe you have a large map and will not need to grow back to a similar size.

func (*Map) Reset

func (m *Map) Reset()

Reset will reset the map by simply deleting all keys to avoid allocating a new map.

func (*Map) Set

func (m *Map) Set(k ident.ID, v DatabaseSeriesBlocks)

Set will set the value for an identifier.

func (*Map) SetUnsafe

func (m *Map) SetUnsafe(k ident.ID, v DatabaseSeriesBlocks, opts SetUnsafeOptions)

SetUnsafe will set the value for an identifier with unsafe options for how the map treats the key.

type MapEntry

type MapEntry struct {
	// contains filtered or unexported fields
}

MapEntry is an entry in the map, this is public to support iterating over the map using a native Go for loop.

func (MapEntry) Key

func (e MapEntry) Key() ident.ID

Key returns the map entry key.

func (MapEntry) Value

func (e MapEntry) Value() DatabaseSeriesBlocks

Value returns the map entry value.

type MapHash

type MapHash uint64

MapHash is the hash for a given map entry, this is public to support iterating over the map using a native Go for loop.

type MapOptions

type MapOptions struct {
	InitialSize int
	KeyCopyPool pool.BytesPool
}

MapOptions provides options used when created the map.

type Options

type Options interface {
	// SetClockOptions sets the clock options.
	SetClockOptions(value clock.Options) Options

	// ClockOptions returns the clock options.
	ClockOptions() clock.Options

	// SetInstrumentOptions sets the instrumentation options.
	SetInstrumentOptions(value instrument.Options) Options

	// InstrumentOptions returns the instrumentation options.
	InstrumentOptions() instrument.Options

	// SetDatabaseBlockOptions sets the database block options.
	SetDatabaseBlockOptions(value block.Options) Options

	// DatabaseBlockOptions returns the database block options.
	DatabaseBlockOptions() block.Options

	// SetNewBlocksLen sets the size of a new blocks map size.
	SetNewBlocksLen(value int) Options

	// NewBlocksLen returns the size of a new blocks map size.
	NewBlocksLen() int

	// SetSeriesCachePolicy sets the series cache policy.
	SetSeriesCachePolicy(value series.CachePolicy) Options

	// SeriesCachePolicy returns the series cache policy.
	SeriesCachePolicy() series.CachePolicy

	// SetIndexDocumentsBuilderAllocator sets the index mutable segment allocator.
	SetIndexDocumentsBuilderAllocator(value DocumentsBuilderAllocator) Options

	// IndexDocumentsBuilderAllocator returns the index documents builder allocator.
	IndexDocumentsBuilderAllocator() DocumentsBuilderAllocator
}

Options represents the options for bootstrap results.

func NewOptions

func NewOptions() Options

NewOptions creates new bootstrap options

type Segment

type Segment struct {
	// contains filtered or unexported fields
}

Segment wraps an index segment so we can easily determine whether or not the segment is persisted to disk.

func NewSegment

func NewSegment(segment segment.Segment, persisted bool) Segment

NewSegment returns an index segment w/ persistence metadata.

func (Segment) IsPersisted

func (s Segment) IsPersisted() bool

IsPersisted returns whether or not the underlying segment was persisted to disk.

func (Segment) Segment

func (s Segment) Segment() segment.Segment

Segment returns a segment.

type SetUnsafeOptions

type SetUnsafeOptions struct {
	NoCopyKey     bool
	NoFinalizeKey bool
}

SetUnsafeOptions is a set of options to use when setting a value with the SetUnsafe method.

type ShardResult

type ShardResult interface {
	// IsEmpty returns whether the result is empty.
	IsEmpty() bool

	// BlockAt returns the block at a given time for a given id,
	// or nil if there is no such block.
	BlockAt(id ident.ID, t time.Time) (block.DatabaseBlock, bool)

	// AllSeries returns a map of all series with their associated blocks.
	AllSeries() *Map

	// NumSeries returns the number of distinct series'.
	NumSeries() int64

	// AddBlock adds a data block.
	AddBlock(id ident.ID, tags ident.Tags, block block.DatabaseBlock)

	// AddSeries adds a single series of blocks.
	AddSeries(id ident.ID, tags ident.Tags, rawSeries block.DatabaseSeriesBlocks)

	// AddResult adds a shard result.
	AddResult(other ShardResult)

	// RemoveBlockAt removes a data block at a given timestamp
	RemoveBlockAt(id ident.ID, t time.Time)

	// RemoveSeries removes a single series of blocks.
	RemoveSeries(id ident.ID)

	// Close closes a shard result.
	Close()
}

ShardResult returns the bootstrap result for a shard.

func NewShardResult

func NewShardResult(opts Options) ShardResult

NewShardResult creates a new shard result.

type ShardResults

type ShardResults map[uint32]ShardResult

ShardResults is a map of shards to shard results.

func (ShardResults) AddResults

func (r ShardResults) AddResults(other ShardResults)

AddResults adds other shard results to the current shard results.

func (ShardResults) Equal

func (r ShardResults) Equal(other ShardResults) bool

Equal returns whether another shard results is equal to the current shard results, will not perform a deep equal only a shallow equal of series and their block addresses.

func (ShardResults) NumSeries

func (r ShardResults) NumSeries() int64

NumSeries returns the number of series' across all shards.

type ShardTimeRanges

type ShardTimeRanges interface {
	// Get time ranges for a shard.
	Get(shard uint32) (xtime.Ranges, bool)

	// Set time ranges for a shard.
	Set(shard uint32, ranges xtime.Ranges) ShardTimeRanges

	// GetOrAdd gets or adds time ranges for a shard.
	GetOrAdd(shard uint32) xtime.Ranges

	// AddRanges adds other shard time ranges to the current shard time ranges.
	AddRanges(ranges ShardTimeRanges)

	// Iter returns the underlying map.
	Iter() map[uint32]xtime.Ranges

	Copy() ShardTimeRanges

	// IsSuperset returns whether the current shard time ranges are a
	// superset of the other shard time ranges.
	IsSuperset(other ShardTimeRanges) bool

	// Equal returns whether two shard time ranges are equal.
	Equal(other ShardTimeRanges) bool

	// ToUnfulfilledDataResult will return a result that is comprised of wholly
	// unfufilled time ranges from the set of shard time ranges.
	ToUnfulfilledDataResult() DataBootstrapResult

	// ToUnfulfilledIndexResult will return a result that is comprised of wholly
	// unfufilled time ranges from the set of shard time ranges.
	ToUnfulfilledIndexResult() IndexBootstrapResult

	// Subtract will subtract another range from the current range.
	Subtract(other ShardTimeRanges)

	// MinMax will return the very minimum time as a start and the
	// maximum time as an end in the ranges.
	MinMax() (time.Time, time.Time)

	// MinMaxRange returns the min and max times, and the duration for this range.
	MinMaxRange() (time.Time, time.Time, time.Duration)

	// String returns a description of the time ranges
	String() string

	// SummaryString returns a summary description of the time ranges
	SummaryString() string

	// IsEmpty returns whether the shard time ranges is empty or not.
	IsEmpty() bool

	// Len returns the number of shards
	Len() int
}

ShardTimeRanges is a map of shards to time ranges.

func NewShardTimeRanges

func NewShardTimeRanges() ShardTimeRanges

NewShardTimeRanges returns an empty ShardTimeRanges.

func NewShardTimeRangesFromRange

func NewShardTimeRangesFromRange(start, end time.Time, shards ...uint32) ShardTimeRanges

NewShardTimeRangesFromRange returns a new ShardTimeRanges with provided shards and time range.

func NewShardTimeRangesFromSize

func NewShardTimeRangesFromSize(size int) ShardTimeRanges

NewShardTimeRangesFromSize returns a new ShardTimeRanges with provided shards and time range.