# sampler

package
Version: v0.0.0-...-24d4c76 Latest Latest

Go to latest
Published: Oct 16, 2018 License: BSD-3-Clause

## Documentation ¶

### Overview ¶

Package sampler contains all the logic of the agent-side trace sampling

Currently implementation is based on the scoring of the "signature" of each trace Based on the score, we get a sample rate to apply to the given trace

Current score implementation is super-simple, it is a counter with polynomial decay per signature. We increment it for each incoming trace then we periodically divide the score by two every X seconds. Right after the division, the score is an approximation of the number of received signatures over X seconds. It is different from the scoring in the Agent.

Since the sampling can happen at different levels (client, agent, server) or depending on different rules, we have to track the sample rate applied at previous steps. This way, sampling twice at 50% can result in an effective 25% sampling. The rate is stored as a metric in the trace root.

#### Package sampler contains all the logic of the agent-side trace sampling ¶

Currently implementation is based on the scoring of the "signature" of each trace Based on the score, we get a sample rate to apply to the given trace

Current score implementation is super-simple, it is a counter with polynomial decay per signature. We increment it for each incoming trace then we periodically divide the score by two every X seconds. Right after the division, the score is an approximation of the number of received signatures over X seconds. It is different from the scoring in the Agent.

Since the sampling can happen at different levels (client, agent, server) or depending on different rules, we have to track the sample rate applied at previous steps. This way, sampling twice at 50% can result in an effective 25% sampling. The rate is stored as a metric in the trace root.

#### Package sampler contains all the logic of the agent-side trace sampling ¶

Currently implementation is based on the scoring of the "signature" of each trace Based on the score, we get a sample rate to apply to the given trace

Current score implementation is super-simple, it is a counter with polynomial decay per signature. We increment it for each incoming trace then we periodically divide the score by two every X seconds. Right after the division, the score is an approximation of the number of received signatures over X seconds. It is different from the scoring in the Agent.

Since the sampling can happen at different levels (client, agent, server) or depending on different rules, we have to track the sample rate applied at previous steps. This way, sampling twice at 50% can result in an effective 25% sampling. The rate is stored as a metric in the trace root.

### Constants ¶

View Source
```const (
// SamplingPriorityKey is the key of the sampling priority value in the metrics map of the root span
SamplingPriorityKey = "_sampling_priority_v1"
)```
View Source
```const (
// with the number of traces contained in the payload.
)```

### Variables ¶

This section is empty.

### Functions ¶

#### func CalcPreSampleRate ¶

`func CalcPreSampleRate(maxUserAvg, currentUserAvg, currentRate float64) (float64, error)`

CalcPreSampleRate gives the new sample rate to apply for a given max user CPU average. It takes the current sample rate and user CPU average as those parameters both have an influence on the result.

#### func GetTraceAppliedSampleRate ¶

`func GetTraceAppliedSampleRate(root *model.Span) float64`

GetTraceAppliedSampleRate gets the sample rate the sample rate applied earlier in the pipeline.

#### func SampleByRate ¶

`func SampleByRate(traceID uint64, sampleRate float64) bool`

SampleByRate tells if a trace (from its ID) with a given rate should be sampled Use Knuth multiplicative hashing to leverage imbalanced traceID generators

#### func SetTraceAppliedSampleRate ¶

`func SetTraceAppliedSampleRate(root *model.Span, sampleRate float64)`

SetTraceAppliedSampleRate sets the currently applied sample rate in the trace data to allow chained up sampling.

### Types ¶

#### type Backend ¶

```type Backend interface {
// Run runs the blocking execution of the backend main loop.
Run()

// Stop stops the backend main loop.
Stop()

// CountSample counts that 1 trace is going through the sampler.
CountSample()

// CountSignature counts that 1 signature is going through the sampler.
CountSignature(signature Signature)

// GetTotalScore returns the TPS (Traces Per Second) of all traces ingested.
GetTotalScore() float64

// GetSampledScore returns the TPS of all traces sampled.
GetSampledScore() float64

// GetUpperSampledScore is similar to GetSampledScore, but with the upper approximation.
GetUpperSampledScore() float64

// GetSignatureScore returns the TPS of traces ingested of a given signature.
GetSignatureScore(signature Signature) float64

// GetSignatureScores returns the TPS of traces ingested for all signatures.
GetSignatureScores() map[Signature]float64

// GetCardinality returns the number of different signatures seen.
GetCardinality() int64
}```

Backend stores and counts traces and signatures ingested by a sampler.

#### type Engine ¶

```type Engine interface {
// Run the sampler.
Run()
// Stop the sampler.
Stop()
// Sample a trace.
Sample(trace model.Trace, root *model.Span, env string) bool
// GetState returns information about the sampler.
GetState() interface{}
// GetType returns the type of the sampler.
GetType() EngineType
}```

Engine is a common basic interface for sampler engines.

#### type EngineType ¶

`type EngineType int`

EngineType represents the type of a sampler engine.

```const (
// NormalScoreEngineType is the type of the ScoreEngine sampling non-error traces.
NormalScoreEngineType EngineType = iota
// ErrorsScoreEngineType is the type of the ScoreEngine sampling error traces.
// PriorityEngineType is type of the priority sampler engine type.
PriorityEngineType
)```

#### type InternalState ¶

```type InternalState struct {
Offset      float64
Slope       float64
Cardinality int64
InTPS       float64
OutTPS      float64
MaxTPS      float64
}```

InternalState exposes all the main internal settings of the score sampler

#### type MemoryBackend ¶

```type MemoryBackend struct {
// contains filtered or unexported fields
}```

MemoryBackend storing any state required to run the sampling algorithms.

Current implementation is only based on counters with polynomial decay. Its bias with steady counts is 1 * decayFactor. The stored scores represent approximation of the real count values (with a countScaleFactor factor).

#### func NewMemoryBackend ¶

`func NewMemoryBackend(decayPeriod time.Duration, decayFactor float64) *MemoryBackend`

NewMemoryBackend returns an initialized Backend.

#### func (*MemoryBackend) CountSample ¶

`func (b *MemoryBackend) CountSample()`

CountSample counts a trace sampled by the sampler.

#### func (*MemoryBackend) CountSignature ¶

`func (b *MemoryBackend) CountSignature(signature Signature)`

CountSignature counts an incoming signature.

#### func (*MemoryBackend) GetCardinality ¶

`func (b *MemoryBackend) GetCardinality() int64`

GetCardinality returns the number of different signatures seen recently.

#### func (*MemoryBackend) GetSampledScore ¶

`func (b *MemoryBackend) GetSampledScore() float64`

GetSampledScore returns the global score of all sampled traces.

#### func (*MemoryBackend) GetSignatureScore ¶

`func (b *MemoryBackend) GetSignatureScore(signature Signature) float64`

GetSignatureScore returns the score of a signature. It is normalized to represent a number of signatures per second.

#### func (*MemoryBackend) GetSignatureScores ¶

`func (b *MemoryBackend) GetSignatureScores() map[Signature]float64`

GetSignatureScores returns the scores for all signatures. It is normalized to represent a number of signatures per second.

#### func (*MemoryBackend) GetTotalScore ¶

`func (b *MemoryBackend) GetTotalScore() float64`

GetTotalScore returns the global score of all sampled traces.

#### func (*MemoryBackend) GetUpperSampledScore ¶

`func (b *MemoryBackend) GetUpperSampledScore() float64`

GetUpperSampledScore returns a certain upper bound of the global count of all sampled traces.

#### func (*MemoryBackend) Run ¶

`func (b *MemoryBackend) Run()`

Run runs and block on the Sampler main loop.

#### func (*MemoryBackend) Stop ¶

`func (b *MemoryBackend) Stop()`

Stop stops the main Run loop.

#### type PreSampler ¶

```type PreSampler struct {
// contains filtered or unexported fields
}```

PreSampler tries to tell wether we should keep a payload, even before fully processing it. Its only clues are the unparsed payload and the HTTP headers. It should remain very light and fast.

#### func NewPreSampler ¶

`func NewPreSampler() *PreSampler`

NewPreSampler returns an initialized presampler

#### func (*PreSampler) Rate ¶

`func (ps *PreSampler) Rate() float64`

Rate returns the current target pre-sample rate, thread-safe. The target pre-sample rate is the value set with SetRate, ideally this is the sample rate, but depending on what is received, the real rate might defer.

#### func (*PreSampler) RealRate ¶

`func (ps *PreSampler) RealRate() float64`

RealRate returns the current real pre-sample rate, thread-safe. This is the value obtained by counting what was kept and dropped.

#### func (*PreSampler) Run ¶

`func (ps *PreSampler) Run()`

Run runs and block on the Sampler main loop

#### func (*PreSampler) Sample ¶

`func (ps *PreSampler) Sample(req *http.Request) bool`

Sample tells wether a given request should be kept (true means: "yes, keep it"). Calling this alters the statistics, it affects the result of RealRate() so only call it once per payload.

#### func (*PreSampler) SetError ¶

`func (ps *PreSampler) SetError(err error)`

SetError set the pre-sample error, thread-safe.

#### func (*PreSampler) SetRate ¶

`func (ps *PreSampler) SetRate(rate float64)`

SetRate set the pre-sample rate, thread-safe.

#### func (*PreSampler) Stats ¶

`func (ps *PreSampler) Stats() *PreSamplerStats`

Stats returns a copy of the currrent pre-sampler stats.

#### func (*PreSampler) Stop ¶

`func (ps *PreSampler) Stop()`

Stop stops the main Run loop

#### type PreSamplerStats ¶

```type PreSamplerStats struct {
// Rate is the target pre-sampling rate.
Rate float64
// Error is the last error got when trying to calc the pre-sampling rate.
// Stored as a string as this is easier to marshal & publish in JSON.
Error string
// RecentTracesSeen is the number of traces that passed by.
RecentTracesSeen float64
// RecentTracesDropped is the number of traces that were dropped.
RecentTracesDropped float64
}```

PreSamplerStats contains pre-sampler data. The public content might be interesting for statistics, logging.

#### func (*PreSamplerStats) RealRate ¶

`func (stats *PreSamplerStats) RealRate() float64`

RealRate calcuates the current real pre-sample rate from the stats data. If no data is available, returns the target rate.

#### type PriorityEngine ¶

```type PriorityEngine struct {
// Sampler is the underlying sampler used by this engine, sharing logic among various engines.
Sampler *Sampler
// contains filtered or unexported fields
}```

PriorityEngine is the main component of the sampling logic

#### func NewPriorityEngine ¶

`func NewPriorityEngine(extraRate float64, maxTPS float64, rateByService *config.RateByService) *PriorityEngine`

NewPriorityEngine returns an initialized Sampler

#### func (*PriorityEngine) GetState ¶

`func (s *PriorityEngine) GetState() interface{}`

GetState collects and return internal statistics and coefficients for indication purposes It returns an interface{}, as other samplers might return other informations.

#### func (*PriorityEngine) GetType ¶

`func (s *PriorityEngine) GetType() EngineType`

GetType return the type of the sampler engine

#### func (*PriorityEngine) Run ¶

`func (s *PriorityEngine) Run()`

Run runs and block on the Sampler main loop

#### func (*PriorityEngine) Sample ¶

`func (s *PriorityEngine) Sample(trace model.Trace, root *model.Span, env string) bool`

Sample counts an incoming trace and tells if it is a sample which has to be kept

#### func (*PriorityEngine) Stop ¶

`func (s *PriorityEngine) Stop()`

Stop stops the main Run loop

#### type Sampler ¶

```type Sampler struct {
// Storage of the state of the sampler
Backend Backend
// contains filtered or unexported fields
}```

Sampler is the main component of the sampling logic

`func (s *Sampler) AdjustScoring()`

AdjustScoring modifies sampler coefficients to fit better the `maxTPS` condition

#### func (*Sampler) GetAllCountScores ¶

`func (s *Sampler) GetAllCountScores() map[Signature]float64`

GetAllCountScores scores all signatures based on their recent throughput The score value can be seeing as the sample rate if the count were the only factor Since other factors can intervene (such as extra global sampling), its value can be larger than 1

#### func (*Sampler) GetAllSignatureSampleRates ¶

`func (s *Sampler) GetAllSignatureSampleRates() map[Signature]float64`

GetAllSignatureSampleRates gives the sample rate to apply to all signatures. For now, only based on count score.

#### func (*Sampler) GetCountScore ¶

`func (s *Sampler) GetCountScore(signature Signature) float64`

GetCountScore scores any signature based on its recent throughput The score value can be seeing as the sample rate if the count were the only factor Since other factors can intervene (such as extra global sampling), its value can be larger than 1

#### func (*Sampler) GetDefaultCountScore ¶

`func (s *Sampler) GetDefaultCountScore() float64`

GetDefaultCountScore returns a default score when not knowing the signature for real. Since other factors can intervene (such as extra global sampling), its value can be larger than 1

#### func (*Sampler) GetDefaultSampleRate ¶

`func (s *Sampler) GetDefaultSampleRate() float64`

GetDefaultSampleRate gives the sample rate to apply to an unknown signature. For now, only based on count score.

#### func (*Sampler) GetMaxTPSSampleRate ¶

`func (s *Sampler) GetMaxTPSSampleRate() float64`

GetMaxTPSSampleRate returns an extra sample rate to apply if we are above maxTPS.

#### func (*Sampler) GetSampleRate ¶

`func (s *Sampler) GetSampleRate(trace model.Trace, root *model.Span, signature Signature) float64`

GetSampleRate returns the sample rate to apply to a trace.

#### func (*Sampler) GetSignatureSampleRate ¶

`func (s *Sampler) GetSignatureSampleRate(signature Signature) float64`

GetSignatureSampleRate gives the sample rate to apply to any signature. For now, only based on count score.

#### func (*Sampler) GetState ¶

`func (s *Sampler) GetState() InternalState`

GetState collects and return internal statistics and coefficients for indication purposes

#### func (*Sampler) Run ¶

`func (s *Sampler) Run()`

Run runs and block on the Sampler main loop

`func (s *Sampler) RunAdjustScoring()`

#### func (*Sampler) SetSignatureCoefficients ¶

`func (s *Sampler) SetSignatureCoefficients(offset float64, slope float64)`

SetSignatureCoefficients updates the internal scoring coefficients used by the signature scoring

#### func (*Sampler) Stop ¶

`func (s *Sampler) Stop()`

Stop stops the main Run loop

#### func (*Sampler) UpdateExtraRate ¶

`func (s *Sampler) UpdateExtraRate(extraRate float64)`

UpdateExtraRate updates the extra sample rate

#### func (*Sampler) UpdateMaxTPS ¶

`func (s *Sampler) UpdateMaxTPS(maxTPS float64)`

UpdateMaxTPS updates the max TPS limit

#### type ScoreEngine ¶

```type ScoreEngine struct {
// Sampler is the underlying sampler used by this engine, sharing logic among various engines.
Sampler *Sampler
// contains filtered or unexported fields
}```

ScoreEngine is the main component of the sampling logic

`func NewErrorsEngine(extraRate float64, maxTPS float64) *ScoreEngine`

NewErrorsEngine returns an initialized Sampler dedicate to errors. It behaves just like the the normal ScoreEngine except for its GetType method (useful for reporting).

#### func NewScoreEngine ¶

`func NewScoreEngine(extraRate float64, maxTPS float64) *ScoreEngine`

NewScoreEngine returns an initialized Sampler

#### func (*ScoreEngine) GetState ¶

`func (s *ScoreEngine) GetState() interface{}`

GetState collects and return internal statistics and coefficients for indication purposes It returns an interface{}, as other samplers might return other informations.

#### func (*ScoreEngine) GetType ¶

`func (s *ScoreEngine) GetType() EngineType`

GetType returns the type of the sampler

#### func (*ScoreEngine) Run ¶

`func (s *ScoreEngine) Run()`

Run runs and block on the Sampler main loop

#### func (*ScoreEngine) Sample ¶

`func (s *ScoreEngine) Sample(trace model.Trace, root *model.Span, env string) bool`

Sample counts an incoming trace and tells if it is a sample which has to be kept

#### func (*ScoreEngine) Stop ¶

`func (s *ScoreEngine) Stop()`

Stop stops the main Run loop

#### type Signature ¶

`type Signature uint64`

Signature is a simple representation of trace, used to identify simlar traces