Back to godoc.org
github.com/stripe/veneur/samplers

package samplers

v13.1.0+incompatible
Latest Go to latest
Published: Jan 7, 2020 | License: MIT | Module: github.com/stripe/veneur

Index

Variables

var AggregatesLookup = map[string]Aggregate{
	"min":    AggregateMin,
	"max":    AggregateMax,
	"median": AggregateMedian,
	"avg":    AggregateAverage,
	"count":  AggregateCount,
	"sum":    AggregateSum,
	"hmean":  AggregateHarmonicMean,
}

func ParseEvent

func ParseEvent(packet []byte) (*ssf.SSFSample, error)

ParseEvent parses a DogStatsD event packet and returns an SSF sample or an error on failure. To facilitate the many Datadog-specific values that are present in a DogStatsD event but not in an SSF sample, a series of special tags are set as defined in protocol/dogstatsd/protocol.go. Any sink that wants to consume these events will then need to implement FlushOtherSamples and unwind these special tags into whatever is appropriate for that sink.

func ParseTagSliceToMap

func ParseTagSliceToMap(tags []string) map[string]string

ParseTagSliceToMap handles splitting a slice of string tags on `:` and creating a map from the parts.

func ValidMetric

func ValidMetric(sample UDPMetric) bool

ValidMetric takes in an SSF sample and determines if it is valid or not.

type Aggregate

type Aggregate int
const (
	AggregateMin Aggregate = 1 << iota
	AggregateMax
	AggregateMedian
	AggregateAverage
	AggregateCount
	AggregateSum
	AggregateHarmonicMean
)

type Counter

type Counter struct {
	Name string
	Tags []string
	// contains filtered or unexported fields
}

Counter is an accumulator

func NewCounter

func NewCounter(Name string, Tags []string) *Counter

NewCounter generates and returns a new Counter.

func (*Counter) Combine

func (c *Counter) Combine(other []byte) error

Combine merges the values seen with another set (marshalled as a byte slice)

func (*Counter) Export

func (c *Counter) Export() (JSONMetric, error)

Export converts a Counter into a JSONMetric which reports the rate.

func (*Counter) Flush

func (c *Counter) Flush(interval time.Duration) []InterMetric

Flush generates an InterMetric from the current state of this Counter.

func (*Counter) GetName

func (c *Counter) GetName() string

GetName returns the name of the counter.

func (*Counter) Merge

func (c *Counter) Merge(v *metricpb.CounterValue)

Merge adds the value from the input CounterValue to this one.

func (*Counter) Metric

func (c *Counter) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Counter for forwarding.

func (*Counter) Sample

func (c *Counter) Sample(sample float64, sampleRate float32)

Sample adds a sample to the counter.

type DerivedMetricsProcessor

type DerivedMetricsProcessor interface {
	SendSample(sample *ssf.SSFSample) error
}

DerivedMetricsProcessor processes any metric created from events or service checks into the worker channels for flushing

type Gauge

type Gauge struct {
	Name string
	Tags []string
	// contains filtered or unexported fields
}

Gauge retains whatever the last value was.

func NewGauge

func NewGauge(Name string, Tags []string) *Gauge

NewGauge generates an empty (valueless) Gauge

func (*Gauge) Combine

func (g *Gauge) Combine(other []byte) error

Combine is pretty naïve for Gauges, as it just overwrites the value.

func (*Gauge) Export

func (g *Gauge) Export() (JSONMetric, error)

Export converts a Gauge into a JSONMetric.

func (*Gauge) Flush

func (g *Gauge) Flush() []InterMetric

Flush generates an InterMetric from the current state of this gauge.

func (*Gauge) GetName

func (g *Gauge) GetName() string

GetName returns the name of the gauge.

func (*Gauge) Merge

func (g *Gauge) Merge(v *metricpb.GaugeValue)

Merge sets the value of this Gauge to the value of the other.

func (*Gauge) Metric

func (g *Gauge) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Gauge for forwarding.

func (*Gauge) Sample

func (g *Gauge) Sample(sample float64, sampleRate float32)

Sample takes on whatever value is passed in as a sample.

type Histo

type Histo struct {
	Name  string
	Tags  []string
	Value *tdigest.MergingDigest
	// these values are computed from only the samples that came through this
	// veneur instance, ignoring any histograms merged from elsewhere
	// we separate them because they're easy to aggregate on the backend without
	// loss of granularity, and having host-local information on them might be
	// useful
	LocalWeight        float64
	LocalMin           float64
	LocalMax           float64
	LocalSum           float64
	LocalReciprocalSum float64
}

Histo is a collection of values that generates max, min, count, and percentiles over time.

func NewHist

func NewHist(Name string, Tags []string) *Histo

NewHist generates a new Histo and returns it.

func (*Histo) Combine

func (h *Histo) Combine(other []byte) error

Combine merges the values of a histogram with another histogram (marshalled as a byte slice)

func (*Histo) Export

func (h *Histo) Export() (JSONMetric, error)

Export converts a Histogram into a JSONMetric

func (*Histo) Flush

func (h *Histo) Flush(interval time.Duration, percentiles []float64, aggregates HistogramAggregates, global bool) []InterMetric

Flush generates InterMetrics for the current state of the Histo. percentiles indicates what percentiles should be exported from the histogram.

func (*Histo) GetName

func (h *Histo) GetName() string

GetName returns the name of the Histo.

func (*Histo) Merge

func (h *Histo) Merge(v *metricpb.HistogramValue)

Merge merges the t-digests of the two histograms and mutates the state of this one.

func (*Histo) Metric

func (h *Histo) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Histo for forwarding.

func (*Histo) Sample

func (h *Histo) Sample(sample float64, sampleRate float32)

Sample adds the supplied value to the histogram.

type HistogramAggregates

type HistogramAggregates struct {
	Value Aggregate
	Count int
}

type InterMetric

type InterMetric struct {
	Name      string
	Timestamp int64
	Value     float64
	Tags      []string
	Type      MetricType
	Message   string
	HostName  string

	// Sinks, if non-nil, indicates which metric sinks a metric
	// should be inserted into. If nil, that means the metric is
	// meant to go to every sink.
	Sinks RouteInformation
}

InterMetric represents a metric that has been completed and is ready for flushing by sinks.

type InvalidMetrics

type InvalidMetrics interface {
	error

	// Samples returns any samples that couldn't be parsed or validated.
	Samples() []*ssf.SSFSample
}

InvalidMetrics is an error type returned if any metric could not be parsed.

type JSONMetric

type JSONMetric struct {
	MetricKey
	Tags []string `json:"tags"`
	// the Value is an internal representation of the metric's contents, eg a
	// gob-encoded histogram or hyperloglog.
	Value []byte `json:"value"`
}

JSONMetric is used to represent a metric that can be remarshaled with its internal state intact. It is used to send metrics from one Veneur to another.

type MetricKey

type MetricKey struct {
	Name       string `json:"name"`
	Type       string `json:"type"`
	JoinedTags string `json:"tagstring"` // tags in deterministic order, joined with commas
}

MetricKey is a struct used to key the metrics into the worker's map. All fields must be comparable types.

func NewMetricKeyFromMetric

func NewMetricKeyFromMetric(m *metricpb.Metric) MetricKey

NewMetricKeyFromMetric initializes a MetricKey from the protobuf-compatible metricpb.Metric

func (MetricKey) String

func (m MetricKey) String() string

ToString returns a string representation of this MetricKey

type MetricScope

type MetricScope int

MetricScope describes where the metric will be emitted.

const (
	MixedScope MetricScope = iota
	LocalOnly
	GlobalOnly
)

func ScopeFromPB

func ScopeFromPB(scope metricpb.Scope) MetricScope

ScopeFromPB creates an internal MetricScope type from the protobuf Scope type.

func (MetricScope) ToPB

func (m MetricScope) ToPB() metricpb.Scope

ToPB maps the metric scope to a protobuf Scope type.

type MetricType

type MetricType int

MetricType defines what kind of metric this is, so that we or our upstream sinks can do the right thing with it.

const (
	// CounterMetric is a counter
	CounterMetric MetricType = iota
	// GaugeMetric is a gauge
	GaugeMetric
	// StatusMetric is a status (synonymous with a service check)
	StatusMetric
)

func (MetricType) String

func (i MetricType) String() string

type RouteInformation

type RouteInformation map[string]struct{}

RouteInformation is a key-only map indicating sink names that are supposed to receive a metric. A nil RouteInformation value corresponds to the "every sink" value; an entry in a non-nil RouteInformation means that the key should receive the metric.

func (RouteInformation) RouteTo

func (ri RouteInformation) RouteTo(name string) bool

RouteTo returns true if the named sink should receive a metric according to the route table. A nil route table causes any sink to be eligible for the metric.

type Set

type Set struct {
	Name string
	Tags []string
	Hll  *hyperloglog.Sketch
}

Set is a list of unique values seen.

func NewSet

func NewSet(Name string, Tags []string) *Set

NewSet generates a new Set and returns it

func (*Set) Combine

func (s *Set) Combine(other []byte) error

Combine merges the values seen with another set (marshalled as a byte slice)

func (*Set) Export

func (s *Set) Export() (JSONMetric, error)

Export converts a Set into a JSONMetric which reports the Tags in the set.

func (*Set) Flush

func (s *Set) Flush() []InterMetric

Flush generates an InterMetric for the state of this Set.

func (*Set) GetName

func (s *Set) GetName() string

GetName returns the name of the set.

func (*Set) Merge

func (s *Set) Merge(v *metricpb.SetValue) error

Merge combines the HyperLogLog with that of the input Set. Since the HyperLogLog is marshalled in the value, it unmarshals it first.

func (*Set) Metric

func (s *Set) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Set for forwarding.

func (*Set) Sample

func (s *Set) Sample(sample string)

Sample checks if the supplied value has is already in the filter. If not, it increments the counter!

type SplitBytes

type SplitBytes struct {
	// contains filtered or unexported fields
}

SplitBytes iterates over a byte buffer, returning chunks split by a given delimiter byte. It does not perform any allocations, and does not modify the buffer it is given. It is not safe for use by concurrent goroutines.

sb := NewSplitBytes(buf, '\n')
for sb.Next() {
    fmt.Printf("%q\n", sb.Chunk())
}

The sequence of chunks returned by SplitBytes is equivalent to calling bytes.Split, except without allocating an intermediate slice.

func NewSplitBytes

func NewSplitBytes(buf []byte, delim byte) *SplitBytes

NewSplitBytes initializes a SplitBytes struct with the provided buffer and delimiter.

func (*SplitBytes) Chunk

func (sb *SplitBytes) Chunk() []byte

Chunk returns the current chunk.

func (*SplitBytes) Next

func (sb *SplitBytes) Next() bool

Next advances SplitBytes to the next chunk, returning true if a new chunk actually exists and false otherwise.

type StatusCheck

type StatusCheck struct {
	InterMetric
}

StatusCheck retains whatever the last value was.

func NewStatusCheck

func NewStatusCheck(Name string, Tags []string) *StatusCheck

NewStatusCheck generates an empty (valueless) StatusCheck

func (*StatusCheck) Combine

func (s *StatusCheck) Combine(other []byte) error

Combine is pretty naïve for StatusChecks, as it just overwrites the value.

func (*StatusCheck) Export

func (s *StatusCheck) Export() (JSONMetric, error)

Export converts a StatusCheck into a JSONMetric.

func (*StatusCheck) Flush

func (s *StatusCheck) Flush() []InterMetric

Flush generates an InterMetric from the current state of this status check.

func (*StatusCheck) Sample

func (s *StatusCheck) Sample(sample float64, sampleRate float32, message string, hostname string)

Sample takes on whatever value is passed in as a sample.

type UDPMetric

type UDPMetric struct {
	MetricKey
	Digest     uint32
	Value      interface{}
	SampleRate float32
	Tags       []string
	Scope      MetricScope
	Timestamp  int64
	Message    string
	HostName   string
}

UDPMetric is a representation of the sample provided by a client. The tag list should be deterministically ordered.

func ConvertIndicatorMetrics

func ConvertIndicatorMetrics(span *ssf.SSFSpan, indicatorTimerName, objectiveTimerName string) (metrics []UDPMetric, err error)

ConvertIndicatorMetrics takes a span that may be an "indicator" span and returns metrics that can be determined from that span. Currently, it converts the span to two timer metrics for the duration of the span. One timer (the "indicator") is tagged with the span's service and error-ness. The other timer (the "objective") is tagged with the span's service, error-ness, and name. The name can be overridden with the ssf_objective tag.

func ConvertMetrics

func ConvertMetrics(m *ssf.SSFSpan) ([]UDPMetric, error)

ConvertMetrics examines an SSF message, parses and returns a new array containing any metrics contained in the message. If any parse error occurs in processing any of the metrics, ExtractMetrics collects them into the error type InvalidMetrics and returns this error alongside any valid metrics that could be parsed.

func ConvertSpanUniquenessMetrics

func ConvertSpanUniquenessMetrics(span *ssf.SSFSpan, rate float32) ([]UDPMetric, error)

ConvertSpanUniquenessMetrics takes a trace span and computes uniqueness metrics about it, returning UDPMetrics sampled at rate. Currently, the only metric returned is a Set counting the unique names per indicator span/service.

func ParseMetric

func ParseMetric(packet []byte) (*UDPMetric, error)

ParseMetric converts the incoming packet from Datadog DogStatsD Datagram format in to a Metric. http://docs.datadoghq.com/guides/dogstatsd/#datagram-format

func ParseMetricSSF

func ParseMetricSSF(metric *ssf.SSFSample) (UDPMetric, error)

ParseMetricSSF converts an incoming SSF packet to a Metric.

func ParseServiceCheck

func ParseServiceCheck(packet []byte) (*UDPMetric, error)

ParseServiceCheck parses a packet that represents a service status check and returns a UDPMetric or an error on failure. The UDPMetric struct has explicit fields for each value of a service status check and does not require overloading magical tags for conversion.

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
f or F : Jump to identifier