statsdinfluxdb

package module
v1.0.2-0...-a0d399c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 9, 2019 License: MIT Imports: 11 Imported by: 0

README

Build Status Documentation Go Report Card codecov License FOSSA Status

Go statsd client library with zero allocation overhead, great performance and automatic reconnects.

Client has zero memory allocation per metric sent:

  • ring of buffers, each buffer is UDP packet
  • buffer is taken from the pool, filled with metrics, passed on to the network delivery and returned to the pool
  • buffer is flushed either when it is full or when flush period comes (e.g. every 100ms)
  • separate goroutines handle network operations: sending UDP packets and reconnecting UDP socket
  • when metric is serialized, zero allocation operations are used to avoid reflect and temporary buffers

Zero memory allocation

As metrics could be sent by the application at very high rate (e.g. hundreds of metrics per one request), it is important that sending metrics doesn't cause any additional GC or CPU pressure. go-statsd is using buffer pools and it tries to avoid allocations while building statsd packets.

Reconnecting to statsd

With modern container-based platforms with dynamic DNS statsd server might change its address when container gets rescheduled. As statsd packets are delivered over UDP, there's no easy way for the client to figure out that packets are going nowhere. go-statsd supports configurable reconnect interval which forces DNS resolution.

While client is reconnecting, metrics are still processed and buffered.

Dropping metrics

When buffer pool is exhausted, go-statsd starts dropping packets. Number of dropped packets is reported via Client.GetLostPackets() and every minute logged using log.Printf(). Usually packets should never be dropped, if that happens it's usually signal of enormous metric volume.

Stastd server

Any statsd-compatible server should work well with go-statsd, statsite works exceptionally well as it has great performance and low memory footprint even with huge number of metrics.

Usage

Initialize client instance with options, one client per application is usually enough:

client := statsd.NewClient("localhost:8125",
    statsd.MaxPacketSize(1400),
    statsd.MetricPrefix("web."))

Send metrics as events happen in the application, metrics will be packed together and delivered to statsd server:

start := time.Now()
client.Incr("requests.http", 1)
// ...
client.PrecisionTiming("requests.route.api.latency", time.Since(start))

Shutdown client during application shutdown to flush all the pending metrics:

client.Close()

Tagging

Metrics could be tagged to support aggregation on TSDB side. go-statsd supports tags in InfluxDB and Datadog formats. Format and default tags (applied to every metric) are passed as options to the client initialization:

client := statsd.NewClient("localhost:8125",
    statsd.TagStyle(TagFormatDatadog),
    statsd.DefaultTags(statsd.StringTag("app", "billing")))

For every metric sent, tags could be added as the last argument(s) to the function call:

client.Incr("request", 1,
    statsd.StringTag("procotol", "http"), statsd.IntTag("port", 80))

Benchmark

Benchmark comparing several clients:

Benchmark results:

BenchmarkAlexcesaro-12    	 5000000	       333 ns/op	       0 B/op	       0 allocs/op
BenchmarkGoStatsd-12      	10000000	       230 ns/op	      23 B/op	       0 allocs/op
BenchmarkCactus-12        	 3000000	       604 ns/op	       5 B/op	       0 allocs/op
BenchmarkG2s-12           	  200000	      7499 ns/op	     576 B/op	      21 allocs/op
BenchmarkQuipo-12         	 1000000	      1048 ns/op	     384 B/op	       7 allocs/op
BenchmarkUnix4ever-12        1000000	      1695 ns/op	     408 B/op	      18 allocs/op

Origins

Ideas were borrowed from the following stastd clients:

Talks

I gave a talk about design and optimizations which went into go-statsd at Gophercon Russia 2018: slides, source.

License

License is MIT License.

FOSSA Status

Documentation

Overview

Package statsd implements high-performance, zero-allocation statsd client.

Go statsd client library with zero allocation overhead, great performance and automatic reconnects.

With statsd architecture aggregation is performed on statsd server side (e.g. using high-performance servers like statsite), so application emits many metrics per user action. Performance of statsd client library is critical to introduce as little overhead as possible.

StatsdClient has zero memory allocation per metric being sent, architecture is the following:

  • there's ring of buffers, each buffer is UDP packet
  • buffer is taken from the pool, filled with metrics, passed on to the network delivery and passed back to the pool
  • buffer is flushed either when it is full or when flush period comes (e.g. every 100ms)
  • separate goroutine is handling network operations: sending UDP packets and reconnecting UDP socket (to handle statsd DNS address change)
  • when metric is serialized, zero allocation methods are used to avoid `reflect` and memory allocation

Ideas were borrowed from the following stastd clients:

Usage

Initialize client instance with options, one client per application is usually enough: 

client := statsd.NewStatsdClient("localhost:8125",
    statsd.MaxPacketSize(1400),
    statsd.MetricPrefix("web."))

Send metrics as events happen in the application, metrics will be packed together and delivered to statsd server: 

start := time.Now()
client.Incr("requests.http", 1)
...
client.PrecisionTiming("requests.route.api.latency", time.Since(start))

Shutdown client during application shutdown to flush all the pending metrics: 

client.Close()

Tagging

Metrics could be tagged to support aggregation on TSDB side. go-statsd supports tags in InfluxDB and Datadog formats. Format and default tags (applied to every metric) are passed as options to the client initialization: 

client := statsd.NewStatsdClient("localhost:8125",
    statsd.TagStyle(StatsdTagFormatDatadog),
    statsd.DefaultTags(statsd.StringTag("app", "billing")))

For every metric sent, tags could be added as the last argument(s) to the function call: 

client.Incr("request", 1,
    statsd.StringTag("protocol", "http"), statsd.IntTag("port", 80))

Index

Constants

View Source 
const (
	DefaultMaxPacketSize     = 1432
	DefaultMetricPrefix      = ""
	DefaultFlushInterval     = 100 * time.Millisecond
	DefaultReconnectInterval = time.Duration(0)
	DefaultReportInterval    = time.Minute
	DefaultRetryTimeout      = 5 * time.Second
	DefaultStatsdLogPrefix   = "[STATSD] "
	DefaultInfluxDBLogPrefix = "[INFLUXDB] "
	DefaultBufPoolCapacity   = 20
	DefaultSendQueueCapacity = 10
	DefaultSendLoopCount     = 1
)

Default settings

View Source 
const (
	TagPlacementName = iota
	TagPlacementSuffix
)

StatsdTag placement constants

Variables

View Source 
var (
	// StatsdTagFormatInfluxDB is format for InfluxDB StatsD telegraf plugin
	//
	// Docs: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd
	StatsdTagFormatInfluxDB = &StatsdTagFormat{
		Placement:         TagPlacementName,
		FirstSeparator:    ",",
		OtherSeparator:    ',',
		KeyValueSeparator: '=',
	}

	// StatsdTagFormatDatadog is format for DogStatsD (Datadog Agent)
	//
	// Docs: https://docs.datadoghq.com/developers/dogstatsd/#datagram-format
	StatsdTagFormatDatadog = &StatsdTagFormat{
		Placement:         TagPlacementSuffix,
		FirstSeparator:    "|#",
		OtherSeparator:    ',',
		KeyValueSeparator: ':',
	}
)

Functions

This section is empty.

Types

type ClientOptions 

type ClientOptions struct {
	// Addr is statsd server address in "host:port" format
	Addr string

	// MetricPrefix is metricPrefix to prepend to every metric being sent
	//
	// If not set defaults to empty string
	MetricPrefix string

	// MaxPacketSize is maximum UDP packet size
	//
	// Safe value is 1432 bytes, if your network supports jumbo frames,
	// this value could be raised up to 8960 bytes
	MaxPacketSize int

	// FlushInterval controls flushing incomplete UDP packets which makes
	// sure metric is not delayed longer than FlushInterval
	//
	// Default value is 100ms, setting FlushInterval to zero disables flushing
	FlushInterval time.Duration

	// ReconnectInterval controls UDP socket reconnects
	//
	// Reconnecting is important to follow DNS changes, e.g. in
	// dynamic container environments like K8s where statsd server
	// instance might be relocated leading to new IP address.
	//
	// By default reconnects are disabled
	ReconnectInterval time.Duration

	// RetryTimeout controls how often client should attempt reconnecting
	// to statsd server on failure
	//
	// Default value is 5 seconds
	RetryTimeout time.Duration

	// ReportInterval instructs client to report number of packets lost
	// each interval via Logger
	//
	// By default lost packets are reported every minute, setting to zero
	// disables reporting
	ReportInterval time.Duration

	// Logger is used by statsd client to report errors and lost packets
	//
	// If not set, default logger to stderr with metricPrefix `[STATSD] ` is being used
	Logger SomeLogger

	// BufPoolCapacity controls size of pre-allocated buffer cache
	//
	// Each buffer is MaxPacketSize. Cache allows to avoid allocating
	// new buffers during high load
	//
	// Default value is DefaultBufPoolCapacity
	BufPoolCapacity int

	// SendQueueCapacity controls length of the queue of packet ready to be sent
	//
	// Packets might stay in the queue during short load bursts or while
	// client is reconnecting to statsd
	//
	// Default value is DefaultSendQueueCapacity
	SendQueueCapacity int

	// SendLoopCount controls number of goroutines sending UDP packets
	//
	// Default value is 1, so packets are sent from single goroutine, this
	// value might need to be bumped under high load
	SendLoopCount int

	// StatsdTagFormat controls formatting of StatsD tags
	//
	// If tags are not used, value of this setting isn't used.
	//
	// There are two predefined formats: for InfluxDB and Datadog, default
	// format is InfluxDB tag format.
	TagFormat *StatsdTagFormat

	// DefaultTags is a list of tags to be applied to every metric
	DefaultTags []StatsdTag
}

ClientOptions are statsd client settings

type InfluxDBClient 

type InfluxDBClient struct {
	// contains filtered or unexported fields
}

InfluxDBClient implements an InfluxDB line protocol client

func NewInfluxDBClient 

func NewInfluxDBClient(addr string, options ...Option) *InfluxDBClient

NewInfluxDBClient returns a newly constructed UDP client for the InfluxDB line protocol

func (*InfluxDBClient) Close 

func (c *InfluxDBClient) Close() error

Close closes the InfluxDB client connection

func (*InfluxDBClient) Send 

func (c *InfluxDBClient) Send(measurement string, tags []InfluxDBTag, fields []InfluxDBField)

Send sends a non-timestamped metric

func (*InfluxDBClient) SendWithTimestamp 

func (c *InfluxDBClient) SendWithTimestamp(measurement string, tags []InfluxDBTag, fields []InfluxDBField, ts time.Time)

SendWithTimestamp sends a timestamped metric

type InfluxDBField 

type InfluxDBField struct {
	// contains filtered or unexported fields
}

InfluxDBField represents a typed key=value field pair

func BoolField 

func BoolField(name string, value bool) InfluxDBField

BoolField constructs a bool-typed InfluxDBField

func FloatField 

func FloatField(name string, value float64) InfluxDBField

FloatField constructs a float-typed InfluxDBField

func IntField 

func IntField(name string, value int64) InfluxDBField

IntField constructs an int-typed InfluxDBField

func StringField 

func StringField(name string, value string) InfluxDBField

StringField constructs a string-typed InfluxDBField

func (*InfluxDBField) Append 

func (f *InfluxDBField) Append(buf []byte) []byte

Append appends the representation of the InfluxDBField to the given buffer

type InfluxDBTag 

type InfluxDBTag struct {
	Name  string
	Value string
}

InfluxDB tag represents a key=value tag pair

type Option 

type Option func(c *ClientOptions)

Option is type for option transport

func BufPoolCapacity 

func BufPoolCapacity(capacity int) Option

BufPoolCapacity controls size of pre-allocated buffer cache

Each buffer is MaxPacketSize. Cache allows to avoid allocating new buffers during high load

Default value is DefaultBufPoolCapacity

func DefaultTags 

func DefaultTags(tags ...StatsdTag) Option

DefaultTags defines a list of tags to be applied to every metric

func FlushInterval 

func FlushInterval(interval time.Duration) Option

FlushInterval controls flushing incomplete UDP packets which makes sure metric is not delayed longer than FlushInterval

Default value is 100ms, setting FlushInterval to zero disables flushing

func Logger 

func Logger(logger SomeLogger) Option

Logger is used by statsd client to report errors and lost packets

If not set, default logger to stderr with metricPrefix `[STATSD] ` is being used

func MaxPacketSize 

func MaxPacketSize(packetSize int) Option

MaxPacketSize control maximum UDP packet size

Default value is DefaultMaxPacketSize

func MetricPrefix 

func MetricPrefix(prefix string) Option

MetricPrefix is metricPrefix to prepend to every metric being sent

Usually metrics are prefixed with app name, e.g. `app.`. To avoid providing this metricPrefix for every metric being collected, and to enable shared libraries to collect metric under app name, use MetricPrefix to set global metricPrefix for all the app metrics, e.g. `MetricPrefix("app".)`.

If not set defaults to empty string

func ReconnectInterval 

func ReconnectInterval(interval time.Duration) Option

ReconnectInterval controls UDP socket reconnects

Reconnecting is important to follow DNS changes, e.g. in dynamic container environments like K8s where statsd server instance might be relocated leading to new IP address.

By default reconnects are disabled

func ReportInterval 

func ReportInterval(interval time.Duration) Option

ReportInterval instructs client to report number of packets lost each interval via Logger

By default lost packets are reported every minute, setting to zero disables reporting

func RetryTimeout 

func RetryTimeout(timeout time.Duration) Option

RetryTimeout controls how often client should attempt reconnecting to statsd server on failure

Default value is 5 seconds

func SendLoopCount 

func SendLoopCount(threads int) Option

SendLoopCount controls number of goroutines sending UDP packets

Default value is 1, so packets are sent from single goroutine, this value might need to be bumped under high load

func SendQueueCapacity 

func SendQueueCapacity(capacity int) Option

SendQueueCapacity controls length of the queue of packet ready to be sent

Packets might stay in the queue during short load bursts or while client is reconnecting to statsd

Default value is DefaultSendQueueCapacity

func TagStyle 

func TagStyle(style *StatsdTagFormat) Option

TagStyle controls formatting of StatsD tags

There are two predefined formats: for InfluxDB and Datadog, default format is InfluxDB tag format.

type SomeLogger 

type SomeLogger interface {
	Printf(fmt string, args ...interface{})
}

SomeLogger defines logging interface that allows using 3rd party loggers (e.g. github.com/sirupsen/logrus) with this Statsd client.

type StatsdClient 

type StatsdClient struct {
	// contains filtered or unexported fields
}

StatsdClient implements statsd client

func NewStatsdClient 

func NewStatsdClient(addr string, options ...Option) *StatsdClient

NewStatsdClient creates new statsd client and starts background processing

StatsdClient connects to statsd server at addr ("host:port")

StatsdClient settings could be controlled via functions of type Option

func (*StatsdClient) CloneWithPrefix 

func (c *StatsdClient) CloneWithPrefix(prefix string) *StatsdClient

CloneWithPrefix returns a clone of the original client with different metricPrefix.

func (*StatsdClient) CloneWithPrefixExtension 

func (c *StatsdClient) CloneWithPrefixExtension(extension string) *StatsdClient

CloneWithPrefixExtension returns a clone of the original client with the original prefixed extended with the specified string.

func (*StatsdClient) Close 

func (c *StatsdClient) Close() error

Close stops the client and all its clones. Calling it on a clone has the same effect as calling it on the original client - it is stopped with all its clones.

func (*StatsdClient) Decr 

func (c *StatsdClient) Decr(stat string, count int64, tags ...StatsdTag)

Decr decrements a counter metri

Often used to note a particular event

func (*StatsdClient) FDecr 

func (c *StatsdClient) FDecr(stat string, count float64, tags ...StatsdTag)

FDecr decrements a float counter metric

func (*StatsdClient) FGauge 

func (c *StatsdClient) FGauge(stat string, value float64, tags ...StatsdTag)

FGauge sends a floating point value for a gauge

func (*StatsdClient) FGaugeDelta 

func (c *StatsdClient) FGaugeDelta(stat string, value float64, tags ...StatsdTag)

FGaugeDelta sends a floating point change for a gauge

func (*StatsdClient) FIncr 

func (c *StatsdClient) FIncr(stat string, count float64, tags ...StatsdTag)

FIncr increments a float counter metric

func (*StatsdClient) Gauge 

func (c *StatsdClient) Gauge(stat string, value int64, tags ...StatsdTag)

Gauge sets or updates constant value for the interval

Gauges are a constant data type. They are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again. If you specify delta to be true, that specifies that the gauge should be updated, not set. Due to the underlying protocol, you can't explicitly set a gauge to a negative number without first setting it to zero.

func (*StatsdClient) GaugeDelta 

func (c *StatsdClient) GaugeDelta(stat string, value int64, tags ...StatsdTag)

GaugeDelta sends a change for a gauge

func (*StatsdClient) GetLostPackets 

func (c *StatsdClient) GetLostPackets() int64

GetLostPackets returns number of packets lost during client lifecycle

func (*StatsdClient) Incr 

func (c *StatsdClient) Incr(stat string, count int64, tags ...StatsdTag)

Incr increments a counter metric

Often used to note a particular event, for example incoming web request.

func (*StatsdClient) PrecisionTiming 

func (c *StatsdClient) PrecisionTiming(stat string, delta time.Duration, tags ...StatsdTag)

PrecisionTiming track a duration event, the time delta has to be a duration

Usually request processing time, time to run database query, etc. are used with this metric type.

func (*StatsdClient) SetAdd 

func (c *StatsdClient) SetAdd(stat string, value string, tags ...StatsdTag)

SetAdd adds unique element to a set

Statsd server will provide cardinality of the set over aggregation period.

func (*StatsdClient) Timing 

func (c *StatsdClient) Timing(stat string, delta int64, tags ...StatsdTag)

Timing tracks a duration event, the time delta must be given in milliseconds

type StatsdTag 

type StatsdTag struct {
	// contains filtered or unexported fields
}

StatsdTag is metric-specific tag

func Int64Tag 

func Int64Tag(name string, value int64) StatsdTag

Int64Tag creates StatsdTag with integer value

func IntTag 

func IntTag(name string, value int) StatsdTag

IntTag creates StatsdTag with integer value

func StringTag 

func StringTag(name, value string) StatsdTag

StringTag creates StatsdTag with string value

func (StatsdTag) Append 

func (tag StatsdTag) Append(buf []byte, style *StatsdTagFormat) []byte

Append formats tag and appends it to the buffer

type StatsdTagFormat 

type StatsdTagFormat struct {
	// FirstSeparator is put after metric name and before first tag
	FirstSeparator string
	// Placement specifies part of the message to append tags to
	Placement byte
	// OtherSeparator separates 2nd and subsequent tags from each other
	OtherSeparator byte
	// KeyValueSeparator separates tag name and tag value
	KeyValueSeparator byte
}

StatsdTagFormat controls tag formatting style

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.