inmem

package
v1.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 20, 2018 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package inmem implements a shared, in-memory index for each database.

The in-memory index is the original index implementation and provides fast access to index data. However, it also forces high memory usage for large datasets and can cause OOM errors.

Index is the shared index structure that provides most of the functionality. However, ShardIndex is a light per-shard wrapper that adapts this original shared index format to the new per-shard format.

Index

Constants

View Source
const IndexName = "inmem"

IndexName is the name of this index.

Variables

This section is empty.

Functions

func NewShardIndex

func NewShardIndex(id uint64, database, path string, seriesIDSet *tsdb.SeriesIDSet, sfile *tsdb.SeriesFile, opt tsdb.EngineOptions) tsdb.Index

NewShardIndex returns a new index for a shard.

Types

type FilterExprs

type FilterExprs map[uint64]influxql.Expr

FilterExprs represents a map of series IDs to filter expressions.

func (FilterExprs) DeleteBoolLiteralTrues

func (fe FilterExprs) DeleteBoolLiteralTrues()

DeleteBoolLiteralTrues deletes all elements whose filter expression is a boolean literal true.

func (FilterExprs) Len

func (fe FilterExprs) Len() int

Len returns the number of elements.

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index is the in memory index of a collection of measurements, time series, and their tags. Exported functions are goroutine safe while un-exported functions assume the caller will use the appropriate locks.

func NewIndex

func NewIndex(database string, sfile *tsdb.SeriesFile) *Index

NewIndex returns a new initialized Index.

func (*Index) Close

func (i *Index) Close() error

func (*Index) CreateMeasurementIndexIfNotExists

func (i *Index) CreateMeasurementIndexIfNotExists(name []byte) *measurement

CreateMeasurementIndexIfNotExists creates or retrieves an in memory index object for the measurement

func (*Index) CreateSeriesListIfNotExists added in v1.5.0

func (i *Index) CreateSeriesListIfNotExists(shardID uint64, seriesIDSet *tsdb.SeriesIDSet, keys, names [][]byte, tagsSlice []models.Tags, opt *tsdb.EngineOptions, ignoreLimits bool) error

CreateSeriesListIfNotExists adds the series for the given measurement to the index and sets its ID or returns the existing series object

func (*Index) Database added in v1.5.0

func (i *Index) Database() string

Database returns the name of the database the index was initialized with.

func (*Index) DiskSizeBytes added in v1.5.0

func (i *Index) DiskSizeBytes() int64

DiskSizeBytes always returns zero bytes, since this is an in-memory index.

func (*Index) DropMeasurement

func (i *Index) DropMeasurement(name []byte) error

DropMeasurement removes the measurement and all of its underlying series from the database index

func (*Index) DropMeasurementIfSeriesNotExist added in v1.5.0

func (i *Index) DropMeasurementIfSeriesNotExist(name []byte) error

DropMeasurementIfSeriesNotExist drops a measurement only if there are no more series for the measurment.

func (*Index) DropSeriesGlobal added in v1.5.0

func (i *Index) DropSeriesGlobal(key []byte, ts int64) error

DropSeriesGlobal removes the series key and its tags from the index.

func (*Index) FieldSet added in v1.5.0

func (i *Index) FieldSet() *tsdb.MeasurementFieldSet

FieldSet returns the assigned fieldset.

func (*Index) ForEachMeasurementName

func (i *Index) ForEachMeasurementName(fn func(name []byte) error) error

ForEachMeasurementName iterates over each measurement name.

func (*Index) ForEachMeasurementTagKey

func (i *Index) ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error

ForEachMeasurementTagKey iterates over all tag keys for a measurement.

func (*Index) HasTagKey

func (i *Index) HasTagKey(name, key []byte) (bool, error)

HasTagKey returns true if tag key exists.

func (*Index) HasTagValue

func (i *Index) HasTagValue(name, key, value []byte) (bool, error)

HasTagValue returns true if tag value exists.

func (*Index) Measurement

func (i *Index) Measurement(name []byte) (*measurement, error)

Measurement returns the measurement object from the index by the name

func (*Index) MeasurementExists

func (i *Index) MeasurementExists(name []byte) (bool, error)

MeasurementExists returns true if the measurement exists.

func (*Index) MeasurementIterator added in v1.5.0

func (i *Index) MeasurementIterator() (tsdb.MeasurementIterator, error)

MeasurementIterator returns an iterator over all measurements in the index. MeasurementIterator does not support authorization.

func (*Index) MeasurementNamesByExpr

func (i *Index) MeasurementNamesByExpr(auth query.Authorizer, expr influxql.Expr) ([][]byte, error)

MeasurementNamesByExpr takes an expression containing only tags and returns a list of matching measurement names.

TODO(edd): Remove authorisation from these methods. There shouldn't need to be any auth passed down into the index.

func (*Index) MeasurementNamesByRegex

func (i *Index) MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)

MeasurementNamesByRegex returns the measurements that match the regex.

func (*Index) MeasurementSeriesIDIterator added in v1.5.0

func (i *Index) MeasurementSeriesIDIterator(name []byte) (tsdb.SeriesIDIterator, error)

func (*Index) MeasurementSeriesKeysByExpr

func (i *Index) MeasurementSeriesKeysByExpr(name []byte, condition influxql.Expr) ([][]byte, error)

func (*Index) MeasurementSeriesKeysByExprIterator added in v1.5.0

func (i *Index) MeasurementSeriesKeysByExprIterator(name []byte, condition influxql.Expr) (tsdb.SeriesIDIterator, error)

func (*Index) MeasurementTagKeyValuesByExpr added in v1.3.2

func (i *Index) MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, keys []string, expr influxql.Expr, keysSorted bool) ([][]string, error)

MeasurementTagKeyValuesByExpr returns a set of tag values filtered by an expression.

See tsm1.Engine.MeasurementTagKeyValuesByExpr for a fuller description of this method.

func (*Index) MeasurementTagKeysByExpr

func (i *Index) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr returns an ordered set of tag keys filtered by an expression.

func (*Index) MeasurementsByName

func (i *Index) MeasurementsByName(names [][]byte) ([]*measurement, error)

MeasurementsByName returns a list of measurements.

func (*Index) MeasurementsSketches

func (i *Index) MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the sketches for the measurements.

func (*Index) Open

func (i *Index) Open() (err error)

func (*Index) Rebuild added in v1.4.0

func (i *Index) Rebuild()

Rebuild recreates the measurement indexes to allow deleted series to be removed and garbage collected.

func (*Index) Series

func (i *Index) Series(key []byte) (*series, error)

Series returns a series by key.

func (*Index) SeriesIDIterator added in v1.5.0

func (i *Index) SeriesIDIterator(opt query.IteratorOptions) (tsdb.SeriesIDIterator, error)

SeriesIDIterator returns an influxql iterator over matching series ids.

func (*Index) SeriesKeys

func (i *Index) SeriesKeys() []string

func (*Index) SeriesSketches

func (i *Index) SeriesSketches() (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the sketches for the series.

func (*Index) SetFieldName

func (i *Index) SetFieldName(measurement []byte, name string)

SetFieldName adds a field name to a measurement.

func (*Index) SetFieldSet

func (i *Index) SetFieldSet(fieldset *tsdb.MeasurementFieldSet)

SetFieldSet sets a shared field set from the engine.

func (*Index) TagKeyCardinality

func (i *Index) TagKeyCardinality(name, key []byte) int

TagKeyCardinality returns the number of values for a measurement/tag key.

func (*Index) TagKeyHasAuthorizedSeries added in v1.4.3

func (i *Index) TagKeyHasAuthorizedSeries(auth query.Authorizer, name []byte, key string) bool

TagKeyHasAuthorizedSeries determines if there exists an authorized series for the provided measurement name and tag key.

func (*Index) TagKeyIterator added in v1.5.0

func (i *Index) TagKeyIterator(name []byte) (tsdb.TagKeyIterator, error)

func (*Index) TagKeySeriesIDIterator added in v1.5.0

func (i *Index) TagKeySeriesIDIterator(name, key []byte) (tsdb.SeriesIDIterator, error)

func (*Index) TagSets

func (i *Index) TagSets(shardSeriesIDs *tsdb.SeriesIDSet, name []byte, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns a list of tag sets.

func (*Index) TagValueIterator added in v1.5.0

func (i *Index) TagValueIterator(name, key []byte) (tsdb.TagValueIterator, error)

TagValueIterator provides an iterator over all the tag values belonging to series with the provided measurement name and tag key.

TagValueIterator does not currently support authorization.

func (*Index) TagValueN

func (i *Index) TagValueN(name, key []byte) int

TagValueN returns the cardinality of a tag value.

func (*Index) TagValueSeriesIDIterator added in v1.5.0

func (i *Index) TagValueSeriesIDIterator(name, key, value []byte) (tsdb.SeriesIDIterator, error)

func (*Index) TagsForSeries

func (i *Index) TagsForSeries(key string) (models.Tags, error)

TagsForSeries returns the tag map for the passed in series

func (*Index) Type

func (i *Index) Type() string

func (*Index) WithLogger

func (i *Index) WithLogger(*zap.Logger)

type ShardIndex

type ShardIndex struct {
	*Index // Shared reference to global database-wide index.
	// contains filtered or unexported fields
}

ShardIndex represents a shim between the TSDB index interface and the shared in-memory index. This is required because per-shard in-memory indexes will grow the heap size too large.

func (*ShardIndex) CreateSeriesIfNotExists

func (idx *ShardIndex) CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error

CreateSeriesIfNotExists creates the provided series on the index if it is not already present.

func (*ShardIndex) CreateSeriesListIfNotExists

func (idx *ShardIndex) CreateSeriesListIfNotExists(keys, names [][]byte, tagsSlice []models.Tags) error

CreateSeriesListIfNotExists creates a list of series if they doesn't exist in bulk.

func (*ShardIndex) DropMeasurementIfSeriesNotExist added in v1.5.0

func (idx *ShardIndex) DropMeasurementIfSeriesNotExist(name []byte) error

DropMeasurementIfSeriesNotExist drops a measurement only if there are no more series for the measurment.

func (*ShardIndex) DropSeries added in v1.5.0

func (idx *ShardIndex) DropSeries(seriesID uint64, _ []byte, _ bool) error

DropSeries removes the provided series id from the local bitset that tracks series in this shard only.

func (*ShardIndex) InitializeSeries

func (idx *ShardIndex) InitializeSeries(keys, names [][]byte, tags []models.Tags) error

InitializeSeries is called during start-up. This works the same as CreateSeriesListIfNotExists except it ignore limit errors.

func (*ShardIndex) SeriesIDSet added in v1.5.0

func (idx *ShardIndex) SeriesIDSet() *tsdb.SeriesIDSet

SeriesIDSet returns the bitset associated with the series ids.

func (*ShardIndex) SeriesN added in v1.5.0

func (idx *ShardIndex) SeriesN() int64

SeriesN returns the number of unique non-tombstoned series local to this shard.

func (*ShardIndex) TagSets

func (idx *ShardIndex) TagSets(name []byte, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns a list of tag sets based on series filtering.

type TagFilter

type TagFilter struct {
	Op    influxql.Token
	Key   string
	Value string
	Regex *regexp.Regexp
}

TagFilter represents a tag filter when looking up other tags or measurements.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL