chunkenc

package
v0.44.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 14, 2023 License: Apache-2.0 Imports: 8 Imported by: 122

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsValidEncoding added in v0.39.0 

func IsValidEncoding(e Encoding) bool

IsValidEncoding returns true for supported encodings.

Types

type Appender 

type Appender interface {
	Append(int64, float64)
	AppendHistogram(t int64, h *histogram.Histogram)
	AppendFloatHistogram(t int64, h *histogram.FloatHistogram)
}

Appender adds sample pairs to a chunk.

type Chunk 

type Chunk interface {
	// Bytes returns the underlying byte slice of the chunk.
	Bytes() []byte

	// Encoding returns the encoding type of the chunk.
	Encoding() Encoding

	// Appender returns an appender to append samples to the chunk.
	Appender() (Appender, error)

	// The iterator passed as argument is for re-use.
	// Depending on implementation, the iterator can
	// be re-used or a new iterator can be allocated.
	Iterator(Iterator) Iterator

	// NumSamples returns the number of samples in the chunk.
	NumSamples() int

	// Compact is called whenever a chunk is expected to be complete (no more
	// samples appended) and the underlying implementation can eventually
	// optimize the chunk.
	// There's no strong guarantee that no samples will be appended once
	// Compact() is called. Implementing this function is optional.
	Compact()
}

Chunk holds a sequence of sample pairs that can be iterated over and appended to.

func FromData 

func FromData(e Encoding, d []byte) (Chunk, error)

FromData returns a chunk from a byte slice of chunk data. This is there so that users of the library can easily create chunks from bytes.

func NewEmptyChunk added in v0.40.0 

func NewEmptyChunk(e Encoding) (Chunk, error)

NewEmptyChunk returns an empty chunk for the given encoding.

type CounterResetHeader added in v0.40.0 

type CounterResetHeader byte

CounterResetHeader defines the first 2 bits of the chunk header. 

const (
	// CounterReset means there was definitely a counter reset that resulted in this chunk.
	CounterReset CounterResetHeader = 0b10000000
	// NotCounterReset means there was definitely no counter reset when cutting this chunk.
	NotCounterReset CounterResetHeader = 0b01000000
	// GaugeType means this chunk contains a gauge histogram, where counter resets do not happen.
	GaugeType CounterResetHeader = 0b11000000
	// UnknownCounterReset means we cannot say if this chunk was created due to a counter reset or not.
	// An explicit counter reset detection needs to happen during query time.
	UnknownCounterReset CounterResetHeader = 0b00000000
)

type Encoding 

type Encoding uint8

Encoding is the identifier for a chunk encoding. 

const (
	EncNone Encoding = iota
	EncXOR
	EncHistogram
	EncFloatHistogram
)

The different available chunk encodings.

func (Encoding) String 

func (e Encoding) String() string

type FloatHistogramAppender added in v0.42.0 

type FloatHistogramAppender struct {
	// contains filtered or unexported fields
}

FloatHistogramAppender is an Appender implementation for float histograms.

func (*FloatHistogramAppender) Append added in v0.42.0 

func (a *FloatHistogramAppender) Append(int64, float64)

Append implements Appender. This implementation panics because normal float samples must never be appended to a histogram chunk.

func (*FloatHistogramAppender) AppendFloatHistogram added in v0.42.0 

func (a *FloatHistogramAppender) AppendFloatHistogram(t int64, h *histogram.FloatHistogram)

AppendFloatHistogram appends a float histogram to the chunk. The caller must ensure that the histogram is properly structured, e.g. the number of buckets used corresponds to the number conveyed by the span structures. First call Appendable() and act accordingly!

func (*FloatHistogramAppender) AppendHistogram added in v0.42.0 

func (a *FloatHistogramAppender) AppendHistogram(int64, *histogram.Histogram)

AppendHistogram implements Appender. This implementation panics because integer histogram samples must never be appended to a float histogram chunk.

func (*FloatHistogramAppender) Appendable added in v0.42.0 

func (a *FloatHistogramAppender) Appendable(h *histogram.FloatHistogram) (
	positiveInserts, negativeInserts []Insert,
	okToAppend, counterReset bool,
)

Appendable returns whether the chunk can be appended to, and if so whether any recoding needs to happen using the provided inserts (in case of any new buckets, positive or negative range, respectively). If the sample is a gauge histogram, AppendableGauge must be used instead.

The chunk is not appendable in the following cases:

  • The schema has changed.
  • The threshold for the zero bucket has changed.
  • Any buckets have disappeared.
  • There was a counter reset in the count of observations or in any bucket, including the zero bucket.
  • The last sample in the chunk was stale while the current sample is not stale.

The method returns an additional boolean set to true if it is not appendable because of a counter reset. If the given sample is stale, it is always ok to append. If counterReset is true, okToAppend is always false.

func (*FloatHistogramAppender) AppendableGauge added in v0.42.0 

func (a *FloatHistogramAppender) AppendableGauge(h *histogram.FloatHistogram) (
	positiveInserts, negativeInserts []Insert,
	backwardPositiveInserts, backwardNegativeInserts []Insert,
	positiveSpans, negativeSpans []histogram.Span,
	okToAppend bool,
)

AppendableGauge returns whether the chunk can be appended to, and if so whether:

  1. Any recoding needs to happen to the chunk using the provided inserts (in case of any new buckets, positive or negative range, respectively).
  2. Any recoding needs to happen for the histogram being appended, using the backward inserts (in case of any missing buckets, positive or negative range, respectively).

This method must be only used for gauge histograms.

The chunk is not appendable in the following cases:

  • The schema has changed.
  • The threshold for the zero bucket has changed.
  • The last sample in the chunk was stale while the current sample is not stale.

func (*FloatHistogramAppender) GetCounterResetHeader added in v0.42.0 

func (a *FloatHistogramAppender) GetCounterResetHeader() CounterResetHeader

func (*FloatHistogramAppender) NumSamples added in v0.42.0 

func (a *FloatHistogramAppender) NumSamples() int

func (*FloatHistogramAppender) Recode added in v0.42.0 

func (a *FloatHistogramAppender) Recode(
	positiveInserts, negativeInserts []Insert,
	positiveSpans, negativeSpans []histogram.Span,
) (Chunk, Appender)

Recode converts the current chunk to accommodate an expansion of the set of (positive and/or negative) buckets used, according to the provided inserts, resulting in the honoring of the provided new positive and negative spans. To continue appending, use the returned Appender rather than the receiver of this method.

func (*FloatHistogramAppender) RecodeHistogramm added in v0.42.0 

func (a *FloatHistogramAppender) RecodeHistogramm(
	fh *histogram.FloatHistogram,
	pBackwardInter, nBackwardInter []Insert,
)

RecodeHistogramm converts the current histogram (in-place) to accommodate an expansion of the set of (positive and/or negative) buckets used.

type FloatHistogramChunk added in v0.42.0 

type FloatHistogramChunk struct {
	// contains filtered or unexported fields
}

FloatHistogramChunk holds encoded sample data for a sparse, high-resolution float histogram.

Each sample has multiple "fields", stored in the following way (raw = store number directly, delta = store delta to the previous number, dod = store delta of the delta to the previous number, xor = what we do for regular sample values): 

field →    ts    count zeroCount sum []posbuckets []negbuckets
sample 1   raw   raw   raw       raw []raw        []raw
sample 2   delta xor   xor       xor []xor        []xor
sample >2  dod   xor   xor       xor []xor        []xor

func NewFloatHistogramChunk added in v0.42.0 

func NewFloatHistogramChunk() *FloatHistogramChunk

NewFloatHistogramChunk returns a new chunk with float histogram encoding.

func (*FloatHistogramChunk) Appender added in v0.42.0 

func (c *FloatHistogramChunk) Appender() (Appender, error)

Appender implements the Chunk interface.

func (*FloatHistogramChunk) Bytes added in v0.42.0 

func (c *FloatHistogramChunk) Bytes() []byte

Bytes returns the underlying byte slice of the chunk.

func (*FloatHistogramChunk) Compact added in v0.42.0 

func (c *FloatHistogramChunk) Compact()

Compact implements the Chunk interface.

func (*FloatHistogramChunk) Encoding added in v0.42.0 

func (c *FloatHistogramChunk) Encoding() Encoding

Encoding returns the encoding type.

func (*FloatHistogramChunk) GetCounterResetHeader added in v0.42.0 

func (c *FloatHistogramChunk) GetCounterResetHeader() CounterResetHeader

GetCounterResetHeader returns the info about the first 2 bits of the chunk header.

func (*FloatHistogramChunk) Iterator added in v0.42.0 

func (c *FloatHistogramChunk) Iterator(it Iterator) Iterator

Iterator implements the Chunk interface.

func (*FloatHistogramChunk) Layout added in v0.42.0 

func (c *FloatHistogramChunk) Layout() (
	schema int32, zeroThreshold float64,
	negativeSpans, positiveSpans []histogram.Span,
	err error,
)

Layout returns the histogram layout. Only call this on chunks that have at least one sample.

func (*FloatHistogramChunk) NumSamples added in v0.42.0 

func (c *FloatHistogramChunk) NumSamples() int

NumSamples returns the number of samples in the chunk.

func (*FloatHistogramChunk) SetCounterResetHeader added in v0.42.0 

func (c *FloatHistogramChunk) SetCounterResetHeader(h CounterResetHeader)

SetCounterResetHeader sets the counter reset header.

type HistogramAppender added in v0.40.0 

type HistogramAppender struct {
	// contains filtered or unexported fields
}

HistogramAppender is an Appender implementation for sparse histograms.

func (*HistogramAppender) Append added in v0.40.0 

func (a *HistogramAppender) Append(int64, float64)

Append implements Appender. This implementation panics because normal float samples must never be appended to a histogram chunk.

func (*HistogramAppender) AppendFloatHistogram added in v0.42.0 

func (a *HistogramAppender) AppendFloatHistogram(int64, *histogram.FloatHistogram)

AppendFloatHistogram implements Appender. This implementation panics because float histogram samples must never be appended to a histogram chunk.

func (*HistogramAppender) AppendHistogram added in v0.40.0 

func (a *HistogramAppender) AppendHistogram(t int64, h *histogram.Histogram)

AppendHistogram appends a histogram to the chunk. The caller must ensure that the histogram is properly structured, e.g. the number of buckets used corresponds to the number conveyed by the span structures. First call Appendable() and act accordingly!

func (*HistogramAppender) Appendable added in v0.40.0 

func (a *HistogramAppender) Appendable(h *histogram.Histogram) (
	positiveInserts, negativeInserts []Insert,
	okToAppend, counterReset bool,
)

Appendable returns whether the chunk can be appended to, and if so whether any recoding needs to happen using the provided inserts (in case of any new buckets, positive or negative range, respectively). If the sample is a gauge histogram, AppendableGauge must be used instead.

The chunk is not appendable in the following cases:

  • The schema has changed.
  • The threshold for the zero bucket has changed.
  • Any buckets have disappeared.
  • There was a counter reset in the count of observations or in any bucket, including the zero bucket.
  • The last sample in the chunk was stale while the current sample is not stale.

The method returns an additional boolean set to true if it is not appendable because of a counter reset. If the given sample is stale, it is always ok to append. If counterReset is true, okToAppend is always false.

func (*HistogramAppender) AppendableGauge added in v0.42.0 

func (a *HistogramAppender) AppendableGauge(h *histogram.Histogram) (
	positiveInserts, negativeInserts []Insert,
	backwardPositiveInserts, backwardNegativeInserts []Insert,
	positiveSpans, negativeSpans []histogram.Span,
	okToAppend bool,
)

AppendableGauge returns whether the chunk can be appended to, and if so whether:

  1. Any recoding needs to happen to the chunk using the provided inserts (in case of any new buckets, positive or negative range, respectively).
  2. Any recoding needs to happen for the histogram being appended, using the backward inserts (in case of any missing buckets, positive or negative range, respectively).

This method must be only used for gauge histograms.

The chunk is not appendable in the following cases:

  • The schema has changed.
  • The threshold for the zero bucket has changed.
  • The last sample in the chunk was stale while the current sample is not stale.

func (*HistogramAppender) GetCounterResetHeader added in v0.42.0 

func (a *HistogramAppender) GetCounterResetHeader() CounterResetHeader

func (*HistogramAppender) NumSamples added in v0.42.0 

func (a *HistogramAppender) NumSamples() int

func (*HistogramAppender) Recode added in v0.40.0 

func (a *HistogramAppender) Recode(
	positiveInserts, negativeInserts []Insert,
	positiveSpans, negativeSpans []histogram.Span,
) (Chunk, Appender)

Recode converts the current chunk to accommodate an expansion of the set of (positive and/or negative) buckets used, according to the provided inserts, resulting in the honoring of the provided new positive and negative spans. To continue appending, use the returned Appender rather than the receiver of this method.

func (*HistogramAppender) RecodeHistogram added in v0.42.0 

func (a *HistogramAppender) RecodeHistogram(
	h *histogram.Histogram,
	pBackwardInserts, nBackwardInserts []Insert,
)

RecodeHistogram converts the current histogram (in-place) to accommodate an expansion of the set of (positive and/or negative) buckets used.

type HistogramChunk added in v0.40.0 

type HistogramChunk struct {
	// contains filtered or unexported fields
}

HistogramChunk holds encoded sample data for a sparse, high-resolution histogram.

Each sample has multiple "fields", stored in the following way (raw = store number directly, delta = store delta to the previous number, dod = store delta of the delta to the previous number, xor = what we do for regular sample values): 

field →    ts    count zeroCount sum []posbuckets []negbuckets
sample 1   raw   raw   raw       raw []raw        []raw
sample 2   delta delta delta     xor []delta      []delta
sample >2  dod   dod   dod       xor []dod        []dod

func NewHistogramChunk added in v0.40.0 

func NewHistogramChunk() *HistogramChunk

NewHistogramChunk returns a new chunk with histogram encoding of the given size.

func (*HistogramChunk) Appender added in v0.40.0 

func (c *HistogramChunk) Appender() (Appender, error)

Appender implements the Chunk interface.

func (*HistogramChunk) Bytes added in v0.40.0 

func (c *HistogramChunk) Bytes() []byte

Bytes returns the underlying byte slice of the chunk.

func (*HistogramChunk) Compact added in v0.40.0 

func (c *HistogramChunk) Compact()

Compact implements the Chunk interface.

func (*HistogramChunk) Encoding added in v0.40.0 

func (c *HistogramChunk) Encoding() Encoding

Encoding returns the encoding type.

func (*HistogramChunk) GetCounterResetHeader added in v0.40.0 

func (c *HistogramChunk) GetCounterResetHeader() CounterResetHeader

GetCounterResetHeader returns the info about the first 2 bits of the chunk header.

func (*HistogramChunk) Iterator added in v0.40.0 

func (c *HistogramChunk) Iterator(it Iterator) Iterator

Iterator implements the Chunk interface.

func (*HistogramChunk) Layout added in v0.40.0 

func (c *HistogramChunk) Layout() (
	schema int32, zeroThreshold float64,
	negativeSpans, positiveSpans []histogram.Span,
	err error,
)

Layout returns the histogram layout. Only call this on chunks that have at least one sample.

func (*HistogramChunk) NumSamples added in v0.40.0 

func (c *HistogramChunk) NumSamples() int

NumSamples returns the number of samples in the chunk.

func (*HistogramChunk) SetCounterResetHeader added in v0.40.0 

func (c *HistogramChunk) SetCounterResetHeader(h CounterResetHeader)

SetCounterResetHeader sets the counter reset header.

type Insert added in v0.42.0 

type Insert struct {
	// contains filtered or unexported fields
}

An Insert describes how many new buckets have to be inserted before processing the pos'th bucket from the original slice.

type Iterator 

type Iterator interface {
	// Next advances the iterator by one and returns the type of the value
	// at the new position (or ValNone if the iterator is exhausted).
	Next() ValueType
	// Seek advances the iterator forward to the first sample with a
	// timestamp equal or greater than t. If the current sample found by a
	// previous `Next` or `Seek` operation already has this property, Seek
	// has no effect. If a sample has been found, Seek returns the type of
	// its value. Otherwise, it returns ValNone, after which the iterator is
	// exhausted.
	Seek(t int64) ValueType
	// At returns the current timestamp/value pair if the value is a float.
	// Before the iterator has advanced, the behaviour is unspecified.
	At() (int64, float64)
	// AtHistogram returns the current timestamp/value pair if the value is
	// a histogram with integer counts. Before the iterator has advanced,
	// the behaviour is unspecified.
	AtHistogram() (int64, *histogram.Histogram)
	// AtFloatHistogram returns the current timestamp/value pair if the
	// value is a histogram with floating-point counts. It also works if the
	// value is a histogram with integer counts, in which case a
	// FloatHistogram copy of the histogram is returned. Before the iterator
	// has advanced, the behaviour is unspecified.
	AtFloatHistogram() (int64, *histogram.FloatHistogram)
	// AtT returns the current timestamp.
	// Before the iterator has advanced, the behaviour is unspecified.
	AtT() int64
	// Err returns the current error. It should be used only after the
	// iterator is exhausted, i.e. `Next` or `Seek` have returned ValNone.
	Err() error
}

Iterator is a simple iterator that can only get the next value. Iterator iterates over the samples of a time series, in timestamp-increasing order.

func MockSeriesIterator 

func MockSeriesIterator(timestamps []int64, values []float64) Iterator

MockSeriesIterator returns an iterator for a mock series with custom timeStamps and values.

func NewNopIterator 

func NewNopIterator() Iterator

NewNopIterator returns a new chunk iterator that does not hold any data.

type Pool 

type Pool interface {
	Put(Chunk) error
	Get(e Encoding, b []byte) (Chunk, error)
}

Pool is used to create and reuse chunk references to avoid allocations.

func NewPool 

func NewPool() Pool

NewPool returns a new pool.

type ValueType added in v0.40.0 

type ValueType uint8

ValueType defines the type of a value an Iterator points to. 

const (
	ValNone           ValueType = iota // No value at the current position.
	ValFloat                           // A simple float, retrieved with At.
	ValHistogram                       // A histogram, retrieve with AtHistogram, but AtFloatHistogram works, too.
	ValFloatHistogram                  // A floating-point histogram, retrieve with AtFloatHistogram.
)

Possible values for ValueType.

func (ValueType) ChunkEncoding added in v0.40.0 

func (v ValueType) ChunkEncoding() Encoding

func (ValueType) String added in v0.40.0 

func (v ValueType) String() string

type XORChunk 

type XORChunk struct {
	// contains filtered or unexported fields
}

XORChunk holds XOR encoded sample data.

func NewXORChunk 

func NewXORChunk() *XORChunk

NewXORChunk returns a new chunk with XOR encoding of the given size.

func (*XORChunk) Appender 

func (c *XORChunk) Appender() (Appender, error)

Appender implements the Chunk interface. It is not valid to call Appender() multiple times concurrently or to use multiple Appenders on the same chunk.

func (*XORChunk) Bytes 

func (c *XORChunk) Bytes() []byte

Bytes returns the underlying byte slice of the chunk.

func (*XORChunk) Compact 

func (c *XORChunk) Compact()

Compact implements the Chunk interface.

func (*XORChunk) Encoding 

func (c *XORChunk) Encoding() Encoding

Encoding returns the encoding type.

func (*XORChunk) Iterator 

func (c *XORChunk) Iterator(it Iterator) Iterator

Iterator implements the Chunk interface. Iterator() must not be called concurrently with any modifications to the chunk, but after it returns you can use an Iterator concurrently with an Appender or other Iterators.

func (*XORChunk) NumSamples 

func (c *XORChunk) NumSamples() int

NumSamples returns the number of samples in the chunk.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.