dogdirect

README

(data)dogdirect

Directly send metrics to DataDog via the HTTP API using golang.

What Problem Are We Solving?

Using DataDog's SasS offering to upload time series data when you can't use the official DataDog agent.

In general, you should use the official DataDog agent. It's great. Use it. But, sometimes you can't or won't run the official DataDog agent. This could be because:

• You are on a constrained environment with low resources
• You don't want alien code running in your environment
• You don't want to run agent process along side your application.
• You don't want to run dedicated container to run the DataDog agent (per cluster or per availability zone).

So if you can't (or won't run the datadog), package provides a simple interface for counters and gauges and uploads data using the HTTP API (the same one the DataDog agent uses).

What does this do?

• Stores your metrics locally supporting both counters and gauges.
• Timinig and Histograms are supported but the descriptive statistics are hardwired and not configurable (but easy to add).
• Allows setting a global namespace
• Allows setting global tags (applied to every metric)
• Uploads your metrics to DataDog every 15 seconds

What doesn't this do?

• Anything that's not a guage or counter or histogram: logs, traces, service checks, events.
• Error handling is probably not awesome. Pull requests welcome.

This is not an official client

This is not an official client. This project is not sponsored or blessed by DataDog.

References and Credits

Offical HTTP API Documentation

THe offical HTTP API documentation on metrics

We store metric points at the 1 second resolution, but we’d prefer if you only submitted points every 15 seconds. Any metrics with fractions of a second timestamps gets rounded to the nearest second, and if any points have the same timestamp, the latest point overwrites the previous ones.

https://docs.datadoghq.com/api/?lang=bash#post-timeseries-points

datadog-go

In particular, much of this code is based on statsd.go. I have mixed feelings about the "watcher" being glued into the main object but seems to work for now.

Interestingly, the buffered implimentation doesn't consolidate anything. If you do 100 increments of a single stat, it will send 100 statsd messages.

datadog-agent

The golang datadog agent has some nuggest, especially the metrics. Unfortunately the metrics package can't be used standalone as it glues in a bunch of extraneous stuff specific to DataDog (logging, configuration). That's too bad.

strip/veneue

stripe/veneue is a statds server on sterioids. It uses the HTTP API as well.

See the datadog section and the http.go for details.

Documentation

Index

• Constants
• type API
  • func NewAPI(apikey string, appkey string, timeout time.Duration) API
  • func (a API) AddHostTags(host string, source string, tags []string) error
  • func (a API) AddPoints(metrics []*Metric) error
• type Client
  • func New(hostname string, api API) *Client
  • func (c *Client) Close() error
  • func (c *Client) Count(name string, value float64, tags []string) error
  • func (c *Client) Decr(name string, tags []string) error
  • func (c *Client) Flush() error
  • func (c *Client) Gauge(name string, value float64, tags []string) error
  • func (c *Client) Histogram(name string, val float64, tags []string) error
  • func (c *Client) Incr(name string, tags []string) error
  • func (c *Client) Snapshot() *Client
  • func (c *Client) Timing(name string, val time.Duration, tags []string) error
• type ExactHistogram
  • func NewExactHistogram(points int, tags []string) *ExactHistogram
  • func (he *ExactHistogram) Add(val float64)
  • func (he *ExactHistogram) Flush() HistogramResult
• type FlushCloser
• type HistogramResult
• type HostTagger
  • func NewHostTagger(api API, hostname string, tags []string) *HostTagger
  • func (ht *HostTagger) Close() error
  • func (ht *HostTagger) Flush() error
• type Metric
  • func NewMetric(name string, mtype string, tags []string) *Metric
• type MultiTask
  • func (mt *MultiTask) Close() error
  • func (mt *MultiTask) Flush() error
• type Periodic
  • func NewPeriodic(client FlushCloser, duration time.Duration) *Periodic
  • func (p *Periodic) Close() error
  • func (p *Periodic) Flush() error

Constants

const (
	TypeGauge = "gauge"
	TypeRate  = "rate"
	TypeCount = "count"
)

todo: make proper type

Types

type API

type API struct {
	// contains filtered or unexported fields
}

func NewAPI

func NewAPI(apikey string, appkey string, timeout time.Duration) API

func (API) AddHostTags

func (a API) AddHostTags(host string, source string, tags []string) error

AddHostTags adds a host tags to a given host. If host is new or doesn't have metrics yet, this call will fail.

func (API) AddPoints

func (a API) AddPoints(metrics []*Metric) error

type Client

type Client struct {
	Series []*Metric `json:"series"` // raw data

	sync.Mutex
	// contains filtered or unexported fields
}

Client is the main datastructure of metrics to upload

func New

func New(hostname string, api API) *Client

New creates a new datadog metrics client

func (*Client) Close

func (c *Client) Close() error

Close the client connection.

func (*Client) Count

func (c *Client) Count(name string, value float64, tags []string) error

Count represents a count of events

func (*Client) Decr

func (c *Client) Decr(name string, tags []string) error

Decr subtracts one event, same as Count(name, -1)

func (*Client) Flush

func (c *Client) Flush() error

Flush forces a flush of the pending commands in the buffer

func (*Client) Gauge

func (c *Client) Gauge(name string, value float64, tags []string) error

Gauge represents an observation

func (*Client) Histogram

func (c *Client) Histogram(name string, val float64, tags []string) error

Histogram records a value that will be used in aggregate

func (*Client) Incr

func (c *Client) Incr(name string, tags []string) error

Incr adds one event count, same as Count(name, 1)

func (*Client) Snapshot

func (c *Client) Snapshot() *Client

Snapshot makes a copy of the data and resets everything locally

func (*Client) Timing

func (c *Client) Timing(name string, val time.Duration, tags []string) error

Timing records a duration

type ExactHistogram

type ExactHistogram struct {
	// contains filtered or unexported fields
}

ExactHistogram is the dumbest way possible to compute various descriptive statistics

It keeps all data, does a sort, and the figures out various stats.
That said for 1000 elements, it takes under 1/20 of a millisecond to compute.

Also the "sort" method is what datadog's agent does, so it can't be too painful.

func NewExactHistogram

func NewExactHistogram(points int, tags []string) *ExactHistogram

NewExactHistogram creates a new object

func (*ExactHistogram) Add

func (he *ExactHistogram) Add(val float64)

Add adds a data point

func (*ExactHistogram) Flush

func (he *ExactHistogram) Flush() HistogramResult

Flush needs to be renamed, but computes the data

type FlushCloser

type FlushCloser interface {
	Flush() error
	Close() error
}

FlushCloser is similar to a io.WriteCloser

type HistogramResult

type HistogramResult struct {
	Count  float64
	Min    float64
	Max    float64
	Avg    float64
	Median float64
	P95    float64
}

HistogramResult returns some descriptive statistics Add other stats as needed

type HostTagger

type HostTagger struct {
	// contains filtered or unexported fields
}

func NewHostTagger

func NewHostTagger(api API, hostname string, tags []string) *HostTagger

func (*HostTagger) Close

func (ht *HostTagger) Close() error

func (*HostTagger) Flush

func (ht *HostTagger) Flush() error

type Metric

type Metric struct {
	Name     string        `json:"metric"`
	Value    [1][2]float64 `json:"points"`
	Type     string        `json:"type"`
	Hostname string        `json:"host,omitempty"`
	Tags     []string      `json:"tags,omitempty"`
	Interval int           `json:"interval,omitempty"`
}

Metric is a data structure that represents the JSON that Datadog wants when posting to the API

func NewMetric

func NewMetric(name string, mtype string, tags []string) *Metric

NewMetric creates a new metric

type MultiTask

type MultiTask []FlushCloser

MultiTask is a sequence of FlushClosers

It operates in serial, although the underlyding implimentation
can work in parallel.  See Periodic below

func (*MultiTask) Close

func (mt *MultiTask) Close() error

Close writes out any buffered data and shutdown client

func (*MultiTask) Flush

func (mt *MultiTask) Flush() error

Flush writes out all data and returns first error if any

type Periodic

type Periodic struct {
	// contains filtered or unexported fields
}

Periodic handles flushing data periodically, also satifies the FlushCloser interface so it can be used in MutliTask

func NewPeriodic

func NewPeriodic(client FlushCloser, duration time.Duration) *Periodic

NewPeriodic task create a new ticket to flush data at regular intervals

func (*Periodic) Close

func (p *Periodic) Close() error

Close stops the ticket and closes the client

func (*Periodic) Flush

func (p *Periodic) Flush() error

Flush causes data to be written out