statsd

package module
v1.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2024 License: MIT Imports: 8 Imported by: 47

README

Build Status Documentation Go Report Card codecov License FOSSA Status

Go statsd client library with zero allocation overhead, great performance and automatic reconnects.

Client has zero memory allocation per metric sent:

  • ring of buffers, each buffer is UDP packet
  • buffer is taken from the pool, filled with metrics, passed on to the network delivery and returned to the pool
  • buffer is flushed either when it is full or when flush period comes (e.g. every 100ms)
  • separate goroutines handle network operations: sending UDP packets and reconnecting UDP socket
  • when metric is serialized, zero allocation operations are used to avoid reflect and temporary buffers

Zero memory allocation

As metrics could be sent by the application at very high rate (e.g. hundreds of metrics per one request), it is important that sending metrics doesn't cause any additional GC or CPU pressure. go-statsd is using buffer pools and it tries to avoid allocations while building statsd packets.

Reconnecting to statsd

With modern container-based platforms with dynamic DNS statsd server might change its address when container gets rescheduled. As statsd packets are delivered over UDP, there's no easy way for the client to figure out that packets are going nowhere. go-statsd supports configurable reconnect interval which forces DNS resolution.

While client is reconnecting, metrics are still processed and buffered.

Dropping metrics

When buffer pool is exhausted, go-statsd starts dropping packets. Number of dropped packets is reported via Client.GetLostPackets() and every minute logged using log.Printf(). Usually packets should never be dropped, if that happens it's usually signal of enormous metric volume.

Stastd server

Any statsd-compatible server should work well with go-statsd, statsite works exceptionally well as it has great performance and low memory footprint even with huge number of metrics.

Usage

Initialize client instance with options, one client per application is usually enough:

client := statsd.NewClient("localhost:8125",
    statsd.MaxPacketSize(1400),
    statsd.MetricPrefix("web."))

Send metrics as events happen in the application, metrics will be packed together and delivered to statsd server:

start := time.Now()
client.Incr("requests.http", 1)
// ...
client.PrecisionTiming("requests.route.api.latency", time.Since(start))

Shutdown client during application shutdown to flush all the pending metrics:

client.Close()

Tagging

Metrics could be tagged to support aggregation on TSDB side. go-statsd supports tags in InfluxDB , Datadog and Graphite formats. Format and default tags (applied to every metric) are passed as options to the client initialization:

client := statsd.NewClient("localhost:8125",
    statsd.TagStyle(TagFormatDatadog),
    statsd.DefaultTags(statsd.StringTag("app", "billing")))

For every metric sent, tags could be added as the last argument(s) to the function call:

client.Incr("request", 1,
    statsd.StringTag("procotol", "http"), statsd.IntTag("port", 80))

Benchmark

Benchmark comparing several clients:

Benchmark results:

BenchmarkAlexcesaro-12    	 5000000	       333 ns/op	       0 B/op	       0 allocs/op
BenchmarkGoStatsd-12      	10000000	       230 ns/op	      23 B/op	       0 allocs/op
BenchmarkCactus-12        	 3000000	       604 ns/op	       5 B/op	       0 allocs/op
BenchmarkG2s-12           	  200000	      7499 ns/op	     576 B/op	      21 allocs/op
BenchmarkQuipo-12         	 1000000	      1048 ns/op	     384 B/op	       7 allocs/op
BenchmarkUnix4ever-12        1000000	      1695 ns/op	     408 B/op	      18 allocs/op

Origins

Ideas were borrowed from the following stastd clients:

Talks

I gave a talk about design and optimizations which went into go-statsd at Gophercon Russia 2018: slides, source.

License

License is MIT License.

FOSSA Status

Documentation

Overview

Package statsd implements high-performance, zero-allocation statsd client.

Go statsd client library with zero allocation overhead, great performance and automatic reconnects.

With statsd architecture aggregation is performed on statsd server side (e.g. using high-performance servers like statsite), so application emits many metrics per user action. Performance of statsd client library is critical to introduce as little overhead as possible.

Client has zero memory allocation per metric being sent, architecture is the following:

  • there's ring of buffers, each buffer is UDP packet
  • buffer is taken from the pool, filled with metrics, passed on to the network delivery and passed back to the pool
  • buffer is flushed either when it is full or when flush period comes (e.g. every 100ms)
  • separate goroutine is handling network operations: sending UDP packets and reconnecting UDP socket (to handle statsd DNS address change)
  • when metric is serialized, zero allocation methods are used to avoid `reflect` and memory allocation

Ideas were borrowed from the following stastd clients:

Usage

Initialize client instance with options, one client per application is usually enough: 

client := statsd.NewClient("localhost:8125",
    statsd.MaxPacketSize(1400),
    statsd.MetricPrefix("web."))

Send metrics as events happen in the application, metrics will be packed together and delivered to statsd server: 

start := time.Now()
client.Incr("requests.http", 1)
...
client.PrecisionTiming("requests.route.api.latency", time.Since(start))

Shutdown client during application shutdown to flush all the pending metrics: 

client.Close()

Tagging

Metrics could be tagged to support aggregation on TSDB side. go-statsd supports tags in InfluxDB and Datadog formats. Format and default tags (applied to every metric) are passed as options to the client initialization: 

client := statsd.NewClient("localhost:8125",
    statsd.TagStyle(TagFormatDatadog),
    statsd.DefaultTags(statsd.StringTag("app", "billing")))

For every metric sent, tags could be added as the last argument(s) to the function call: 

client.Incr("request", 1,
    statsd.StringTag("protocol", "http"), statsd.IntTag("port", 80))

Index

Constants

View Source 
const (
	DefaultMaxPacketSize     = 1432
	DefaultMetricPrefix      = ""
	DefaultFlushInterval     = 100 * time.Millisecond
	DefaultReconnectInterval = time.Duration(0)
	DefaultReportInterval    = time.Minute
	DefaultRetryTimeout      = 5 * time.Second
	DefaultLogPrefix         = "[STATSD] "
	DefaultBufPoolCapacity   = 20
	DefaultSendQueueCapacity = 10
	DefaultSendLoopCount     = 1
	DefaultNetwork           = "udp"
)

Default settings

View Source 
const (
	TagPlacementName = iota
	TagPlacementSuffix
)

Tag placement constants

Variables

View Source 
var (
	// TagFormatInfluxDB is format for InfluxDB StatsD telegraf plugin
	//
	// Docs: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd
	TagFormatInfluxDB = &TagFormat{
		Placement:         TagPlacementName,
		FirstSeparator:    ",",
		OtherSeparator:    ',',
		KeyValueSeparator: []byte{'='},
	}

	// TagFormatDatadog is format for DogStatsD (Datadog Agent)
	//
	// Docs: https://docs.datadoghq.com/developers/dogstatsd/#datagram-format
	TagFormatDatadog = &TagFormat{
		Placement:         TagPlacementSuffix,
		FirstSeparator:    "|#",
		OtherSeparator:    ',',
		KeyValueSeparator: []byte{':'},
	}

	// TagFormatGraphite is format for Graphite
	//
	// Docs: https://graphite.readthedocs.io/en/latest/tags.html
	TagFormatGraphite = &TagFormat{
		Placement:         TagPlacementName,
		FirstSeparator:    ";",
		OtherSeparator:    ';',
		KeyValueSeparator: []byte{'='},
	}

	// TagFormatOkmeter is format for Okmeter agent
	//
	// Docs: https://okmeter.io/misc/docs#statsd-plugin-config
	TagFormatOkmeter = &TagFormat{
		Placement:         TagPlacementName,
		FirstSeparator:    ".",
		OtherSeparator:    '.',
		KeyValueSeparator: []byte("_is_"),
	}
)

Functions

This section is empty.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client implements statsd client

func NewClient 

func NewClient(addr string, options ...Option) *Client

NewClient creates new statsd client and starts background processing

Client connects to statsd server at addr ("host:port")

Client settings could be controlled via functions of type Option

func (*Client) CloneWithPrefix added in v1.2.0 

func (c *Client) CloneWithPrefix(prefix string) *Client

CloneWithPrefix returns a clone of the original client with different metricPrefix.

func (*Client) CloneWithPrefixExtension added in v1.2.0 

func (c *Client) CloneWithPrefixExtension(extension string) *Client

CloneWithPrefixExtension returns a clone of the original client with the original prefixed extended with the specified string.

func (*Client) Close 

func (c *Client) Close() error

Close stops the client and all its clones. Calling it on a clone has the same effect as calling it on the original client - it is stopped with all its clones.

func (*Client) Decr 

func (c *Client) Decr(stat string, count int64, tags ...Tag)

Decr decrements a counter metri

Often used to note a particular event

func (*Client) FDecr added in v1.2.1 

func (c *Client) FDecr(stat string, count float64, tags ...Tag)

FDecr decrements a float counter metric

func (*Client) FGauge 

func (c *Client) FGauge(stat string, value float64, tags ...Tag)

FGauge sends a floating point value for a gauge

func (*Client) FGaugeDelta 

func (c *Client) FGaugeDelta(stat string, value float64, tags ...Tag)

FGaugeDelta sends a floating point change for a gauge

func (*Client) FIncr added in v1.2.1 

func (c *Client) FIncr(stat string, count float64, tags ...Tag)

FIncr increments a float counter metric

func (*Client) Gauge 

func (c *Client) Gauge(stat string, value int64, tags ...Tag)

Gauge sets or updates constant value for the interval

Gauges are a constant data type. They are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again. If you specify delta to be true, that specifies that the gauge should be updated, not set. Due to the underlying protocol, you can't explicitly set a gauge to a negative number without first setting it to zero.

func (*Client) GaugeDelta 

func (c *Client) GaugeDelta(stat string, value int64, tags ...Tag)

GaugeDelta sends a change for a gauge

func (*Client) GetLostPackets 

func (c *Client) GetLostPackets() int64

GetLostPackets returns number of packets lost during client lifecycle

func (*Client) Incr 

func (c *Client) Incr(stat string, count int64, tags ...Tag)

Incr increments a counter metric

Often used to note a particular event, for example incoming web request.

func (*Client) PrecisionTiming 

func (c *Client) PrecisionTiming(stat string, delta time.Duration, tags ...Tag)

PrecisionTiming track a duration event, the time delta has to be a duration

Usually request processing time, time to run database query, etc. are used with this metric type.

func (*Client) SetAdd 

func (c *Client) SetAdd(stat string, value string, tags ...Tag)

SetAdd adds unique element to a set

Statsd server will provide cardinality of the set over aggregation period.

func (*Client) Timing 

func (c *Client) Timing(stat string, delta int64, tags ...Tag)

Timing tracks a duration event, the time delta must be given in milliseconds

type ClientOptions 

type ClientOptions struct {
	// Addr is statsd server address in "host:port" format
	Addr string

	// AddrNetwork is network type for the address. Defaults to udp.
	AddrNetwork string

	// MetricPrefix is metricPrefix to prepend to every metric being sent
	//
	// If not set defaults to empty string
	MetricPrefix string

	// MaxPacketSize is maximum UDP packet size
	//
	// Safe value is 1432 bytes, if your network supports jumbo frames,
	// this value could be raised up to 8960 bytes
	MaxPacketSize int

	// FlushInterval controls flushing incomplete UDP packets which makes
	// sure metric is not delayed longer than FlushInterval
	//
	// Default value is 100ms, setting FlushInterval to zero disables flushing
	FlushInterval time.Duration

	// ReconnectInterval controls UDP socket reconnects
	//
	// Reconnecting is important to follow DNS changes, e.g. in
	// dynamic container environments like K8s where statsd server
	// instance might be relocated leading to new IP address.
	//
	// By default reconnects are disabled
	ReconnectInterval time.Duration

	// RetryTimeout controls how often client should attempt reconnecting
	// to statsd server on failure
	//
	// Default value is 5 seconds
	RetryTimeout time.Duration

	// ReportInterval instructs client to report number of packets lost
	// each interval via Logger
	//
	// By default lost packets are reported every minute, setting to zero
	// disables reporting
	ReportInterval time.Duration

	// Logger is used by statsd client to report errors and lost packets
	//
	// If not set, default logger to stderr with metricPrefix `[STATSD] ` is being used
	Logger SomeLogger

	// BufPoolCapacity controls size of pre-allocated buffer cache
	//
	// Each buffer is MaxPacketSize. Cache allows to avoid allocating
	// new buffers during high load
	//
	// Default value is DefaultBufPoolCapacity
	BufPoolCapacity int

	// SendQueueCapacity controls length of the queue of packet ready to be sent
	//
	// Packets might stay in the queue during short load bursts or while
	// client is reconnecting to statsd
	//
	// Default value is DefaultSendQueueCapacity
	SendQueueCapacity int

	// SendLoopCount controls number of goroutines sending UDP packets
	//
	// Default value is 1, so packets are sent from single goroutine, this
	// value might need to be bumped under high load
	SendLoopCount int

	// TagFormat controls formatting of StatsD tags
	//
	// If tags are not used, value of this setting isn't used.
	//
	// There are two predefined formats: for InfluxDB and Datadog, default
	// format is InfluxDB tag format.
	TagFormat *TagFormat

	// DefaultTags is a list of tags to be applied to every metric
	DefaultTags []Tag
}

ClientOptions are statsd client settings

type Option 

type Option func(c *ClientOptions)

Option is type for option transport

func BufPoolCapacity 

func BufPoolCapacity(capacity int) Option

BufPoolCapacity controls size of pre-allocated buffer cache

Each buffer is MaxPacketSize. Cache allows to avoid allocating new buffers during high load

Default value is DefaultBufPoolCapacity

func DefaultTags added in v1.1.0 

func DefaultTags(tags ...Tag) Option

DefaultTags defines a list of tags to be applied to every metric

func FlushInterval 

func FlushInterval(interval time.Duration) Option

FlushInterval controls flushing incomplete UDP packets which makes sure metric is not delayed longer than FlushInterval

Default value is 100ms, setting FlushInterval to zero disables flushing

func Logger 

func Logger(logger SomeLogger) Option

Logger is used by statsd client to report errors and lost packets

If not set, default logger to stderr with metricPrefix `[STATSD] ` is being used

func MaxPacketSize 

func MaxPacketSize(packetSize int) Option

MaxPacketSize control maximum UDP packet size

Default value is DefaultMaxPacketSize

func MetricPrefix 

func MetricPrefix(prefix string) Option

MetricPrefix is metricPrefix to prepend to every metric being sent

Usually metrics are prefixed with app name, e.g. `app.`. To avoid providing this metricPrefix for every metric being collected, and to enable shared libraries to collect metric under app name, use MetricPrefix to set global metricPrefix for all the app metrics, e.g. `MetricPrefix("app.")`.

If not set defaults to empty string

func Network added in v1.3.4 

func Network(network string) Option

Network sets the network to use Dialing the statsd server

func ReconnectInterval 

func ReconnectInterval(interval time.Duration) Option

ReconnectInterval controls UDP socket reconnects

Reconnecting is important to follow DNS changes, e.g. in dynamic container environments like K8s where statsd server instance might be relocated leading to new IP address.

By default reconnects are disabled

func ReportInterval 

func ReportInterval(interval time.Duration) Option

ReportInterval instructs client to report number of packets lost each interval via Logger

By default lost packets are reported every minute, setting to zero disables reporting

func RetryTimeout 

func RetryTimeout(timeout time.Duration) Option

RetryTimeout controls how often client should attempt reconnecting to statsd server on failure

Default value is 5 seconds

func SendLoopCount 

func SendLoopCount(threads int) Option

SendLoopCount controls number of goroutines sending UDP packets

Default value is 1, so packets are sent from single goroutine, this value might need to be bumped under high load

func SendQueueCapacity 

func SendQueueCapacity(capacity int) Option

SendQueueCapacity controls length of the queue of packet ready to be sent

Packets might stay in the queue during short load bursts or while client is reconnecting to statsd

Default value is DefaultSendQueueCapacity

func TagStyle added in v1.1.0 

func TagStyle(style *TagFormat) Option

TagStyle controls formatting of StatsD tags

There are two predefined formats: for InfluxDB and Datadog, default format is InfluxDB tag format.

type SomeLogger added in v1.1.1 

type SomeLogger interface {
	Printf(fmt string, args ...interface{})
}

SomeLogger defines logging interface that allows using 3rd party loggers (e.g. github.com/sirupsen/logrus) with this Statsd client.

type Tag added in v1.1.0 

type Tag struct {
	// contains filtered or unexported fields
}

Tag is metric-specific tag

func Int64Tag added in v1.1.0 

func Int64Tag(name string, value int64) Tag

Int64Tag creates Tag with integer value

func IntTag added in v1.1.0 

func IntTag(name string, value int) Tag

IntTag creates Tag with integer value

func StringTag added in v1.1.0 

func StringTag(name, value string) Tag

StringTag creates Tag with string value

func (Tag) Append added in v1.1.0 

func (tag Tag) Append(buf []byte, style *TagFormat) []byte

Append formats tag and appends it to the buffer

type TagFormat added in v1.1.0 

type TagFormat struct {
	// FirstSeparator is put after metric name and before first tag
	FirstSeparator string
	// Placement specifies part of the message to append tags to
	Placement byte
	// OtherSeparator separates 2nd and subsequent tags from each other
	OtherSeparator byte
	// KeyValueSeparator separates tag name and tag value
	KeyValueSeparator []byte
}

TagFormat controls tag formatting style

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.