storage

package
v0.0.0-...-ea98ea7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 10, 2021 License: Apache-2.0 Imports: 36 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	NotFetch      FetchDataOption = 1
	OnlyFetchTime                 = iota
	FetchAll
)

Variables

View Source 
var (
	// MetricsWithDroppedLabels is the number of metrics with at least a single dropped label
	MetricsWithDroppedLabels uint64

	// TooLongLabelNames is the number of too long label names
	TooLongLabelNames uint64

	// TooLongLabelValues is the number of too long label values
	TooLongLabelValues uint64
)
View Source 
var ErrDeadlineExceeded = fmt.Errorf("deadline exceeded")

ErrDeadlineExceeded is returned when the request times out.

Functions

func MarshalBlock 

func MarshalBlock(dst []byte, b *Block) []byte

MarshalBlock marshals b to dst.

b.MarshalData must be called on b before calling MarshalBlock.

func MarshalMetricNameRaw 

func MarshalMetricNameRaw(dst []byte, labels []Label) []byte

MarshalMetricNameRaw marshals labels to dst and returns the result.

The result must be unmarshaled with SpanName.unmarshalRaw

func MarshalMetricRow 

func MarshalMetricRow(dst []byte, metricNameRaw []byte, timestamp int64, value []byte) []byte

MarshalMetricRow marshals SpanRow data to dst and returns the result.

func PutSpanName 

func PutSpanName(mn *SpanName)

PutSpanName returns mn to the pool.

func RegexpCacheMisses 

func RegexpCacheMisses() uint64

RegexpCacheMisses returns the number of cache misses for regexp cache.

func RegexpCacheRequests 

func RegexpCacheRequests() uint64

RegexpCacheRequests returns the number of requests to regexp cache.

func RegexpCacheSize 

func RegexpCacheSize() int

RegexpCacheSize returns the number of cached regexps for tag filters.

func SetBigMergeWorkersCount 

func SetBigMergeWorkersCount(n int)

SetBigMergeWorkersCount sets the maximum number of concurrent mergers for big blocks.

The function must be called before opening or creating any storage.

func SetFinalMergeDelay 

func SetFinalMergeDelay(delay time.Duration)

SetFinalMergeDelay sets the delay before doing final merge for partitions without newly ingested data.

This function may be called only before Storage initialization.

func SetMaxLabelsPerTimeseries 

func SetMaxLabelsPerTimeseries(maxLabels int)

SetMaxLabelsPerTimeseries sets the limit on the number of labels per each time series.

Superfouos labels are dropped.

func SetSmallMergeWorkersCount 

func SetSmallMergeWorkersCount(n int)

SetSmallMergeWorkersCount sets the maximum number of concurrent mergers for small blocks.

The function must be called before opening or creating any storage.

func UnmarshalBlock 

func UnmarshalBlock(dst *Block, src []byte) ([]byte, error)

UnmarshalBlock unmarshal Block from src to dst.

dst.UnmarshalData isn't called on the block.

Types

type Block 

type Block struct {
	// contains filtered or unexported fields
}

Block represents a block of time series values for a single TraceID.

func (*Block) AppendRowsWithTimeRangeFilter 

func (b *Block) AppendRowsWithTimeRangeFilter(dstTimestamps []int64, dstValues [][]byte, tr ScanRange) ([]int64, [][]byte)

AppendRowsWithTimeRangeFilter filters samples from b according to tr and appends them to dst*.

It is expected that UnmarshalData has been already called on b.

func (*Block) CopyFrom 

func (b *Block) CopyFrom(src *Block)

CopyFrom copies src to b.

func (*Block) Init 

func (b *Block) Init(traceID TraceID, timestamps []int64, values [][]byte)

Init initializes b with the given tsid, timestamps, values and scale.

func (*Block) MarshalData 

func (b *Block) MarshalData(timestampsBlockOffset, valuesBlockOffset uint64) ([]byte, []byte, []byte)

MarshalData marshals the block into binary representation.

func (*Block) MarshalPortable 

func (b *Block) MarshalPortable(dst []byte) []byte

MarshalPortable marshals b to dst, so it could be portably migrated to other VictoriaMetrics instance.

The marshaled value must be unmarshaled with UnmarshalPortable function.

func (*Block) Reset 

func (b *Block) Reset()

Reset resets b.

func (*Block) RowsCount 

func (b *Block) RowsCount() int

RowsCount returns the number of rows in the block.

func (*Block) TraceID 

func (b *Block) TraceID() *TraceID

func (*Block) UnmarshalData 

func (b *Block) UnmarshalData() error

UnmarshalData unmarshals block data.

func (*Block) UnmarshalPortable 

func (b *Block) UnmarshalPortable(src []byte) ([]byte, error)

UnmarshalPortable unmarshals block from src to b and returns the remaining tail.

It is assumed that the block has been marshaled with MarshalPortable.

type BlockRef 

type BlockRef struct {
	// contains filtered or unexported fields
}

BlockRef references a Block.

BlockRef is valid only until the corresponding Search is valid, i.e. it becomes invalid after Search.MustClose is called.

func (*BlockRef) MustReadBlock 

func (br *BlockRef) MustReadBlock(dst *Block, fetchData uint8)

MustReadBlock reads block from br to dst.

if fetchData is false, then only block header is read, otherwise all the data is read.

type FetchDataOption 

type FetchDataOption = uint8

type IndexDBMetrics 

type IndexDBMetrics struct {
	TagCacheSize      uint64
	TagCacheSizeBytes uint64
	TagCacheRequests  uint64
	TagCacheMisses    uint64

	UselessTagFiltersCacheSize      uint64
	UselessTagFiltersCacheSizeBytes uint64
	UselessTagFiltersCacheRequests  uint64
	UselessTagFiltersCacheMisses    uint64

	IndexDBRefCount uint64

	NewTimeseriesCreated    uint64
	MissingTSIDsForMetricID uint64

	RecentHourMetricIDsSearchCalls uint64
	RecentHourMetricIDsSearchHits  uint64
	DateMetricIDsSearchCalls       uint64
	DateMetricIDsSearchHits        uint64

	DateRangeSearchCalls uint64
	DateRangeSearchHits  uint64

	MissingMetricNamesForMetricID uint64

	IndexBlocksWithMetricIDsProcessed      uint64
	IndexBlocksWithMetricIDsIncorrectOrder uint64

	mergeset.TableMetrics
}

IndexDBMetrics contains essential metrics for indexDB.

type Label 

type Label struct {
	Name  []byte
	Value []byte
}

Label is a timeseries label

type Metrics 

type Metrics struct {
	RowsAddedTotal uint64

	TooSmallTimestampRows uint64
	TooBigTimestampRows   uint64

	AddRowsConcurrencyLimitReached uint64
	AddRowsConcurrencyLimitTimeout uint64
	AddRowsConcurrencyDroppedRows  uint64
	AddRowsConcurrencyCapacity     uint64
	AddRowsConcurrencyCurrent      uint64

	SearchTraceIDsConcurrencyLimitReached uint64
	SearchTraceIDsConcurrencyLimitTimeout uint64
	SearchTraceIDsConcurrencyCapacity     uint64
	SearchTraceIDsConcurrencyCurrent      uint64

	SearchDelays uint64

	SlowRowInserts         uint64
	SlowPerDayIndexInserts uint64
	SlowMetricNameLoads    uint64

	TimestampsBlocksMerged uint64
	TimestampsBytesSaved   uint64

	MetricNameCacheSize       uint64
	MetricNameCacheSizeBytes  uint64
	MetricNameCacheRequests   uint64
	MetricNameCacheMisses     uint64
	MetricNameCacheCollisions uint64

	MinuteMetricIDCacheSize      uint64
	MinuteMetricIDCacheSizeBytes uint64

	IndexDBMetrics IndexDBMetrics
	TableMetrics   TableMetrics
}

Metrics contains essential metrics for the Storage.

func (*Metrics) Reset 

func (m *Metrics) Reset()

Reset resets m.

type ScanRange 

type ScanRange struct {
	MinTimestamp int64
	MaxTimestamp int64
	Limit        int
}

ScanRange is time range.

func (*ScanRange) String 

func (tr *ScanRange) String() string
type Search struct {
	// BlockRef is updated with each Search.NextMetricBlock call.
	BlockRef *BlockRef
	// contains filtered or unexported fields
}

Search is a search for time series.

func (*Search) Error 

func (s *Search) Error() error

Error returns the last error from s.

func (*Search) Init 

func (s *Search) Init(storage *Storage, tfss []*TagFilters, tr ScanRange, deadline uint64) int

Init initializes s from the given storage, tfss and tr.

MustClose must be called when the search is done.

Init returns the upper bound on the number of found time series.

func (*Search) MustClose 

func (s *Search) MustClose()

MustClose closes the Search.

func (*Search) NextMetricBlock 

func (s *Search) NextMetricBlock() bool

NextMetricBlock proceeds to the next MetricBlockRef.

type SearchQuery 

type SearchQuery struct {
	MinTimestamp int64
	MaxTimestamp int64
	TagFilterss  [][]TagFilter
	Limit        int
	Forward      bool
	FetchData    FetchDataOption
}

SearchQuery is used for sending search queries from vmselect to vmstorage.

func (*SearchQuery) Marshal 

func (sq *SearchQuery) Marshal(dst []byte) []byte

Marshal appends marshaled sq to dst and returns the result.

func (*SearchQuery) String 

func (sq *SearchQuery) String() string

String returns string representation of the search query.

func (*SearchQuery) Unmarshal 

func (sq *SearchQuery) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals sq from src and returns the tail.

type SpanName 

type SpanName struct {
	TraceID []byte

	// Tags are optional. They must be sorted by tag Key for canonical view.
	// Use sortTags method.
	Tags []Tag
}

SpanName reperesents a metric name.

func GetSpanName 

func GetSpanName() *SpanName

GetSpanName returns a SpanName from pool.

func (*SpanName) AddTag 

func (mn *SpanName) AddTag(key, value []byte)

AddTag adds new tag to mn with the given key and value.

func (*SpanName) CopyFrom 

func (mn *SpanName) CopyFrom(src *SpanName)

CopyFrom copies src to mn.

func (*SpanName) GetTagValue 

func (mn *SpanName) GetTagValue(tagKey string) []byte

GetTagValue returns tag value for the given tagKey.

func (*SpanName) Marshal 

func (mn *SpanName) Marshal(dst []byte) []byte

Marshal appends marshaled mn to dst and returns the result.

Tags must be sorted before calling this function.

func (*SpanName) RemoveTag 

func (mn *SpanName) RemoveTag(tagKey string)

RemoveTag removes a tag with the given tagKey

func (*SpanName) RemoveTagsIgnoring 

func (mn *SpanName) RemoveTagsIgnoring(ignoringTags []string)

RemoveTagsIgnoring removes all the tags included in ignoringTags.

func (*SpanName) RemoveTagsOn 

func (mn *SpanName) RemoveTagsOn(onTags []string)

RemoveTagsOn removes all the tags not included to onTags.

func (*SpanName) Reset 

func (mn *SpanName) Reset()

Reset resets the mn.

func (*SpanName) ResetMetricGroup 

func (mn *SpanName) ResetMetricGroup()

ResetMetricGroup resets mn.TraceID

func (*SpanName) SetTags 

func (mn *SpanName) SetTags(addTags []string, src *SpanName)

SetTags sets tags from src with keys matching addTags.

func (*SpanName) String 

func (mn *SpanName) String() string

String returns user-readable representation of the metric name.

Use this function only for debug logging.

func (*SpanName) Unmarshal 

func (mn *SpanName) Unmarshal(src []byte) error

Unmarshal unmarshals mn from src.

type SpanRow 

type SpanRow struct {
	// SpanNameRaw contains raw metric name, which must be decoded
	// with SpanName.unmarshalRaw.
	SpanNameRaw []byte

	Timestamp int64
	Value     []byte
}

SpanRow is a metric to insert into storage.

func (*SpanRow) CopyFrom 

func (mr *SpanRow) CopyFrom(src *SpanRow)

CopyFrom copies src to mr.

func (*SpanRow) Marshal 

func (mr *SpanRow) Marshal(dst []byte) []byte

Marshal appends marshaled mr to dst and returns the result.

func (*SpanRow) String 

func (mr *SpanRow) String() string

String returns string representation of the mr.

func (*SpanRow) Unmarshal 

func (mr *SpanRow) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals mr from src and returns the remaining tail from src.

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage represents TSDB storage.

func OpenStorage 

func OpenStorage(path string, retentionMsecs int64) (*Storage, error)

OpenStorage opens storage on the given path with the given retentionMsecs.

func (*Storage) AddRows 

func (s *Storage) AddRows(mrs []SpanRow) error

AddRows adds the given mrs to s.

func (*Storage) ForceMergePartitions 

func (s *Storage) ForceMergePartitions(partitionNamePrefix string) error

ForceMergePartitions force-merges partitions in s with names starting from the given partitionNamePrefix.

Partitions are merged sequentially in order to reduce load on the system.

func (*Storage) MustClose 

func (s *Storage) MustClose()

MustClose closes the storage.

func (*Storage) RetentionMonths 

func (s *Storage) RetentionMonths() int

RetentionMonths returns retention months for s.

func (*Storage) SearchTagEntries 

func (s *Storage) SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]TagEntry, error)

SearchTagEntries returns a list of (tagName -> tagValues).

func (*Storage) SearchTagKeys 

func (s *Storage) SearchTagKeys(maxTagKeys int, deadline uint64) ([]string, error)

SearchTagKeys searches for tag keys.

func (*Storage) SearchTagValues 

func (s *Storage) SearchTagValues(tagKey []byte, maxTagValues int, deadline uint64) ([]string, error)

SearchTagValues searches for tag values.

func (*Storage) SearchTraceIDs 

func (s *Storage) SearchTraceIDs(tfss []*TagFilters, tr ScanRange, deadline uint64) ([]TraceID, error)

SearchTraceIDs returns sorted TraceIDs for the given tfss and the given tr.

func (*Storage) UpdateMetrics 

func (s *Storage) UpdateMetrics(m *Metrics)

UpdateMetrics updates m with metrics from s.

type TableMetrics 

type TableMetrics struct {
	PartitionsRefCount uint64
	// contains filtered or unexported fields
}

TableMetrics contains essential metrics for the table.

type Tag 

type Tag struct {
	Key   []byte
	Value []byte
}

Tag represents a (key, value) tag for metric.

func (*Tag) Equal 

func (tag *Tag) Equal(t *Tag) bool

Equal returns true if tag equals t

func (*Tag) Marshal 

func (tag *Tag) Marshal(dst []byte) []byte

Marshal appends marshaled tag to dst and returns the result.

func (*Tag) Reset 

func (tag *Tag) Reset()

Reset resets the tag.

func (*Tag) Unmarshal 

func (tag *Tag) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals tag from src and returns the remaining data from src.

type TagEntry 

type TagEntry struct {
	// Key is tagName
	Key string

	// Values contains all the values for Key.
	Values []string
}

TagEntry contains (tagName -> tagValues) mapping

type TagFilter 

type TagFilter struct {
	Key        []byte
	Value      []byte
	IsNegative bool
	IsRegexp   bool
}

TagFilter represents a single tag filter from SearchQuery.

func (*TagFilter) Marshal 

func (tf *TagFilter) Marshal(dst []byte) []byte

Marshal appends marshaled tf to dst and returns the result.

func (*TagFilter) String 

func (tf *TagFilter) String() string

String returns string representation of tf.

func (*TagFilter) Unmarshal 

func (tf *TagFilter) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals tf from src and returns the tail.

type TagFilters 

type TagFilters struct {
	// contains filtered or unexported fields
}

TagFilters represents filters used for filtering tags.

func NewTagFilters 

func NewTagFilters() *TagFilters

NewTagFilters returns new TagFilters for the given.

func (*TagFilters) Add 

func (tfs *TagFilters) Add(key, value []byte, isNegative, isRegexp bool) error

Add adds the given tag filter to tfs.

TraceID must be encoded with nil key.

Finalize must be called after tfs is constructed.

func (*TagFilters) Finalize 

func (tfs *TagFilters) Finalize() []*TagFilters

Finalize finalizes tfs and may return complementary TagFilters, which must be added to the resulting set of tag filters.

func (*TagFilters) Reset 

func (tfs *TagFilters) Reset()

Reset resets the tf

func (*TagFilters) String 

func (tfs *TagFilters) String() string

String returns human-readable value for tfs.

type TraceID 

type TraceID = uint128.Uint128

func UnmarshalTraceID 

func UnmarshalTraceID(src []byte) (TraceID, []byte, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.