telegraf

package module
v0.1.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 24, 2015 License: MIT Imports: 14 Imported by: 0

README

Telegraf - A native agent for InfluxDB Circle CI

Telegraf is an agent written in Go for collecting metrics from the system it's running on or from other services and writing them into InfluxDB.

Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics from well known services (like Hadoop, or Postgres, or Redis) and third party APIs (like Mailchimp, AWS CloudWatch, or Google Analytics).

We'll eagerly accept pull requests for new plugins and will manage the set of plugins that Telegraf supports. See the bottom of this doc for instructions on writing new plugins.

Quickstart

  • Build from source or download telegraf:
Linux packages for Debian/Ubuntu and RHEL/CentOS:

NOTE: version 0.1.4+ has introduced some breaking changes! A 0.1.4+ telegraf agent is NOT backwards-compatible with a config file from 0.1.3 and below. That being said, the difference is not huge, see below for an example on how to setup the new config file.

As well, due to a breaking change to the InfluxDB integer line-protocol, there are some InfluxDB compatibility requirements:

  • InfluxDB 0.9.3+ (including nightly builds) requires Telegraf 0.1.5+
  • InfluxDB 0.9.2 and prior requires Telegraf 0.1.4

Telegraf 0.1.5

Telegraf 0.1.4

OSX via Homebrew:
brew update
brew install telegraf
From Source:

Telegraf manages dependencies via godep, which gets installed via the Makefile. Assuming you have your GOPATH setup, make build should be enough to gather dependencies and build telegraf.

How to use it:
  • Run telegraf -sample-config > telegraf.toml to create an initial configuration
  • Edit the configuration to match your needs
  • Run telegraf -config telegraf.toml -test to output one full measurement sample to STDOUT
  • Run telegraf -config telegraf.toml to gather and send metrics to configured outputs.
  • Run telegraf -config telegraf.toml -filter system:swap to enable only the system & swap plugins defined in the config.

Telegraf Options

Telegraf has a few options you can configure under the agent section of the config. If you don't see an agent section run telegraf -sample-config > telegraf.toml to create a valid initial configuration:

  • hostname: The hostname is passed as a tag. By default this will be the value retured by hostname on the machine running Telegraf. You can override that value here.
  • interval: How ofter to gather metrics. Uses a simple number + unit parser, ie "10s" for 10 seconds or "5m" for 5 minutes.
  • debug: Set to true to gather and send metrics to STDOUT as well as InfluxDB.

Plugin Options

There are 5 configuration options that are configurable per plugin:

  • pass: An array of strings that is used to filter metrics generated by the current plugin. Each string in the array is tested as a prefix against metric names and if it matches, the metric is emitted.
  • drop: The inverse of pass, if a metric name matches, it is not emitted.
  • tagpass: tag names and arrays of strings that are used to filter metrics by the current plugin. Each string in the array is tested as an exact match against the tag name, and if it matches the metric is emitted.
  • tagdrop: The inverse of tagpass. If a tag matches, the metric is not emitted. This is tested on metrics that have passed the tagpass test.
  • interval: How often to gather this metric. Normal plugins use a single global interval, but if one particular plugin should be run less or more often, you can configure that here.
Plugin Configuration Examples

This is a full working config that will output CPU data to an InfluxDB instance at 192.168.59.103:8086, tagging measurements with dc="denver-1". It will output measurements at a 10s interval and will collect totalcpu & percpu data.

[outputs]
[outputs.influxdb]
url = "http://192.168.59.103:8086" # required.
database = "telegraf" # required.

[tags]
dc = "denver-1"

[agent]
interval = "10s"

# PLUGINS
[cpu]
percpu = true
totalcpu = true

Below is how to configure tagpass parameters (added in 0.1.4)

# Don't collect CPU data for cpu6 & cpu7
[cpu.tagdrop]
cpu = [ "cpu6", "cpu7" ]

[disk]
[disk.tagpass]
# tagpass conditions are OR, not AND.
# If the (filesystem is ext4 or xfs) OR (the path is /opt or /home)
# then the metric passes
fstype = [ "ext4", "xfs" ]
path = [ "/opt", "/home" ]

Supported Plugins

Telegraf currently has support for collecting metrics from:

  • disque
  • elasticsearch
  • exec (generic executable JSON-gathering plugin)
  • haproxy
  • httpjson (generic JSON-emitting http service plugin)
  • kafka_consumer
  • leofs
  • lustre2
  • memcached
  • mongodb
  • mysql
  • nginx
  • postgresql
  • prometheus
  • rabbitmq
  • redis
  • rethinkdb
  • system (mem, CPU, load, etc.)

We'll be adding support for many more over the coming months. Read on if you want to add support for another service or third-party API.

Plugins

This section is for developers that want to create new collection plugins. Telegraf is entirely plugin driven. This interface allows for operators to pick and chose what is gathered as well as makes it easy for developers to create new ways of generating metrics.

Plugin authorship is kept as simple as possible to promote people to develop and submit new plugins.

Guidelines

  • A plugin must conform to the plugins.Plugin interface.
  • Telegraf promises to run each plugin's Gather function serially. This means developers don't have to worry about thread safety within these functions.
  • Each generated metric automatically has the name of the plugin that generated it prepended. This is to keep plugins honest.
  • Plugins should call plugins.Add in their init function to register themselves. See below for a quick example.
  • To be available within Telegraf itself, plugins must add themselves to the github.com/influxdb/telegraf/plugins/all/all.go file.
  • The SampleConfig function should return valid toml that describes how the plugin can be configured. This is include in telegraf -sample-config.
  • The Description function should say in one line what this plugin does.
Plugin interface
type Plugin interface {
	SampleConfig() string
	Description() string
	Gather(Accumulator) error
}

type Accumulator interface {
	Add(measurement string, value interface{}, tags map[string]string)
	AddValuesWithTime(measurement string,
		values map[string]interface{},
		tags map[string]string,
		timestamp time.Time)
}
Accumulator

The way that a plugin emits metrics is by interacting with the Accumulator.

The Add function takes 3 arguments:

  • measurement: A string description of the metric. For instance bytes_read or faults.
  • value: A value for the metric. This accepts 5 different types of value:
    • int: The most common type. All int types are accepted but favor using int64 Useful for counters, etc.
    • float: Favor float64, useful for gauges, percentages, etc.
    • bool: true or false, useful to indicate the presence of a state. light_on, etc.
    • string: Typically used to indicate a message, or some kind of freeform information.
    • time.Time: Useful for indicating when a state last occurred, for instance light_on_since.
  • tags: This is a map of strings to strings to describe the where or who about the metric. For instance, the net plugin adds a tag named "interface" set to the name of the network interface, like "eth0".

The AddValuesWithTime allows multiple values for a point to be passed. The values used are the same type profile as value above. The timestamp argument allows a point to be registered as having occurred at an arbitrary time.

Let's say you've written a plugin that emits metrics about processes on the current host.


type Process struct {
	CPUTime float64
	MemoryBytes int64
	PID int
}

func Gather(acc plugins.Accumulator) error {
	for _, process := range system.Processes() {
		tags := map[string]string {
			"pid": fmt.Sprintf("%d", process.Pid),
		}

		acc.Add("cpu", process.CPUTime, tags)
		acc.Add("memory", process.MemoryBytes, tags)
	}
}
Example
package simple

// simple.go

import "github.com/influxdb/telegraf/plugins"

type Simple struct {
	Ok bool
}

func (s *Simple) Description() string {
	return "a demo plugin"
}

func (s *Simple) SampleConfig() string {
	return "ok = true # indicate if everything is fine"
}

func (s *Simple) Gather(acc plugins.Accumulator) error {
	if s.Ok {
		acc.Add("state", "pretty good", nil)
	} else {
		acc.Add("state", "not great", nil)
	}

	return nil
}

func init() {
	plugins.Add("simple", func() plugins.Plugin { return &Simple{} })
}

Testing

Execute short tests:

execute make test-short

Execute long tests:

As Telegraf collects metrics from several third-party services it becomes a difficult task to mock each service as some of them have complicated protocols which would take some time to replicate.

To overcome this situation we've decided to use docker containers to provide a fast and reproducible environment to test those services which require it. For other situations (i.e: https://github.com/influxdb/telegraf/blob/master/plugins/redis/redis_test.go ) a simple mock will suffice.

To execute Telegraf tests follow these simple steps:

  • Install docker compose following these instructions
    • mac users should be able to simply do brew install boot2docker and brew install docker-compose
  • execute make test
Unit test troubleshooting:

Try cleaning up your test environment by executing make test-cleanup and re-running

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func PrintSampleConfig

func PrintSampleConfig()

PrintSampleConfig prints the sample config!

Types

type Agent

type Agent struct {

	// Interval at which to gather information
	Interval Duration

	// Run in debug mode?
	Debug    bool
	Hostname string

	Config *Config
	// contains filtered or unexported fields
}

Agent runs telegraf and collects data based on the given config

func NewAgent

func NewAgent(config *Config) (*Agent, error)

NewAgent returns an Agent struct based off the given Config

func (*Agent) Close added in v0.1.4

func (a *Agent) Close() error

Close closes the connection to all configured outputs

func (*Agent) Connect

func (a *Agent) Connect() error

Connect connects to all configured outputs

func (*Agent) LoadOutputs added in v0.1.4

func (a *Agent) LoadOutputs() ([]string, error)

LoadOutputs loads the agent's outputs

func (*Agent) LoadPlugins

func (a *Agent) LoadPlugins(pluginsFilter string) ([]string, error)

LoadPlugins loads the agent's plugins

func (*Agent) Run

func (a *Agent) Run(shutdown chan struct{}) error

Run runs the agent daemon, gathering every Interval

func (*Agent) Test

func (a *Agent) Test() error

Test verifies that we can 'Gather' from all plugins with their configured Config struct

func (*Agent) TestAllPlugins

func (a *Agent) TestAllPlugins() error

TestAllPlugins verifies that we can 'Gather' from all plugins with the default configuration

type BatchPoints

type BatchPoints struct {
	client.BatchPoints

	Debug bool

	Prefix string

	Config *ConfiguredPlugin
	// contains filtered or unexported fields
}

BatchPoints is used to send a batch of data in a single write from telegraf to influx

func (*BatchPoints) Add

func (bp *BatchPoints) Add(measurement string, val interface{}, tags map[string]string)

Add adds a measurement

func (*BatchPoints) AddValuesWithTime

func (bp *BatchPoints) AddValuesWithTime(
	measurement string,
	values map[string]interface{},
	tags map[string]string,
	timestamp time.Time,
)

AddValuesWithTime adds a measurement with a provided timestamp

type Config

type Config struct {
	Tags map[string]string
	// contains filtered or unexported fields
}

Config specifies the URL/user/password for the database that telegraf will be logging to, as well as all the plugins that the user has specified

func LoadConfig

func LoadConfig(path string) (*Config, error)

LoadConfig loads the given config file and returns a *Config pointer

func (*Config) ApplyAgent

func (c *Config) ApplyAgent(v interface{}) error

ApplyAgent loads the toml config into the given interface

func (*Config) ApplyOutput added in v0.1.4

func (c *Config) ApplyOutput(name string, v interface{}) error

ApplyOutput loads the toml config into the given interface

func (*Config) ApplyPlugin

func (c *Config) ApplyPlugin(name string, v interface{}) (*ConfiguredPlugin, error)

ApplyPlugin takes defined plugin names and applies them to the given interface, returning a ConfiguredPlugin object in the end that can be inserted into a runningPlugin by the agent.

func (*Config) ListTags

func (c *Config) ListTags() string

ListTags returns a string of tags specified in the config, line-protocol style

func (*Config) Outputs added in v0.1.4

func (c *Config) Outputs() map[string]*ast.Table

Outputs returns the configured outputs as a map of name -> output toml

func (*Config) OutputsDeclared added in v0.1.4

func (c *Config) OutputsDeclared() []string

OutputsDeclared returns the name of all outputs declared in the config.

func (*Config) Plugins

func (c *Config) Plugins() map[string]*ast.Table

Plugins returns the configured plugins as a map of name -> plugin toml

func (*Config) PluginsDeclared

func (c *Config) PluginsDeclared() []string

PluginsDeclared returns the name of all plugins declared in the config.

type ConfiguredPlugin

type ConfiguredPlugin struct {
	Name string

	Drop []string
	Pass []string

	TagDrop []TagFilter
	TagPass []TagFilter

	Interval time.Duration
}

ConfiguredPlugin containing a name, interval, and drop/pass prefix lists Also lists the tags to filter

func (*ConfiguredPlugin) ShouldPass

func (cp *ConfiguredPlugin) ShouldPass(measurement string, tags map[string]string) bool

ShouldPass returns true if the metric should pass, false if should drop

type Duration

type Duration struct {
	time.Duration
}

Duration just wraps time.Duration

func (*Duration) UnmarshalTOML

func (d *Duration) UnmarshalTOML(b []byte) error

UnmarshalTOML parses the duration from the TOML config file

type TagFilter added in v0.1.4

type TagFilter struct {
	Name   string
	Filter []string
}

TagFilter is the name of a tag, and the values on which to filter

Directories

Path Synopsis
Godeps
_workspace/src/github.com/Shopify/sarama
Package sarama provides client libraries for the Kafka 0.8 protocol.
Package sarama provides client libraries for the Kafka 0.8 protocol.
_workspace/src/github.com/Shopify/sarama/mocks
Package mocks provides mocks that can be used for testing applications that use Sarama.
Package mocks provides mocks that can be used for testing applications that use Sarama.
+build go1.3
_workspace/src/github.com/boltdb/bolt
Package bolt implements a low-level key/value store in pure Go.
Package bolt implements a low-level key/value store in pure Go.
_workspace/src/github.com/cenkalti/backoff
Package backoff implements backoff algorithms for retrying operations.
Package backoff implements backoff algorithms for retrying operations.
_workspace/src/github.com/eapache/go-resiliency/breaker
Package breaker implements the circuit-breaker resiliency pattern for Go.
Package breaker implements the circuit-breaker resiliency pattern for Go.
_workspace/src/github.com/eapache/queue
Package queue provides a fast, ring-buffer queue based on the version suggested by Dariusz Górecki.
Package queue provides a fast, ring-buffer queue based on the version suggested by Dariusz Górecki.
_workspace/src/github.com/fsouza/go-dockerclient
Package docker provides a client for the Docker remote API.
Package docker provides a client for the Docker remote API.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/parsers
Package parsers provides helper functions to parse and validate different type of string.
Package parsers provides helper functions to parse and validate different type of string.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/pools
Package pools provides a collection of pools which provide various data types with buffers.
Package pools provides a collection of pools which provide various data types with buffers.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ulimit
Package ulimit provides structure and helper function to parse and represent resource limits (Rlimit and Ulimit, its human friendly version).
Package ulimit provides structure and helper function to parse and represent resource limits (Rlimit and Ulimit, its human friendly version).
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/units
Package units provides helper function to parse and print size and time units in human-readable format.
Package units provides helper function to parse and print size and time units in human-readable format.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/gorilla/context
Package context stores values shared during a request lifetime.
Package context stores values shared during a request lifetime.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/gorilla/mux
Package gorilla/mux implements a request router and dispatcher.
Package gorilla/mux implements a request router and dispatcher.
_workspace/src/github.com/fsouza/go-dockerclient/testing
Package testing provides a fake implementation of the Docker API, useful for testing purpose.
Package testing provides a fake implementation of the Docker API, useful for testing purpose.
_workspace/src/github.com/go-sql-driver/mysql
Go MySQL Driver - A MySQL-Driver for Go's database/sql package The driver should be used via the database/sql package: import "database/sql" import _ "github.com/go-sql-driver/mysql" db, err := sql.Open("mysql", "user:password@/dbname") See https://github.com/go-sql-driver/mysql#usage for details
Go MySQL Driver - A MySQL-Driver for Go's database/sql package The driver should be used via the database/sql package: import "database/sql" import _ "github.com/go-sql-driver/mysql" db, err := sql.Open("mysql", "user:password@/dbname") See https://github.com/go-sql-driver/mysql#usage for details
_workspace/src/github.com/gogo/protobuf/proto
Package proto converts data structures to and from the wire format of protocol buffers.
Package proto converts data structures to and from the wire format of protocol buffers.
_workspace/src/github.com/gogo/protobuf/proto/proto3_proto
Package proto3_proto is a generated protocol buffer package.
Package proto3_proto is a generated protocol buffer package.
_workspace/src/github.com/golang/protobuf/proto
Package proto converts data structures to and from the wire format of protocol buffers.
Package proto converts data structures to and from the wire format of protocol buffers.
_workspace/src/github.com/golang/protobuf/proto/proto3_proto
Package proto3_proto is a generated protocol buffer package.
Package proto3_proto is a generated protocol buffer package.
_workspace/src/github.com/golang/snappy
Package snappy implements the snappy block-based compression format.
Package snappy implements the snappy block-based compression format.
_workspace/src/github.com/gonuts/go-shellquote
Shellquote provides utilities for joining/splitting strings using sh's word-splitting rules.
Shellquote provides utilities for joining/splitting strings using sh's word-splitting rules.
_workspace/src/github.com/hashicorp/go-msgpack/codec
High Performance, Feature-Rich Idiomatic Go encoding library for msgpack and binc .
High Performance, Feature-Rich Idiomatic Go encoding library for msgpack and binc .
_workspace/src/github.com/influxdb/influxdb/influxql
Package influxql implements a parser for the InfluxDB query language.
Package influxql implements a parser for the InfluxDB query language.
_workspace/src/github.com/influxdb/influxdb/meta/internal
Package internal is a generated protocol buffer package.
Package internal is a generated protocol buffer package.
_workspace/src/github.com/influxdb/influxdb/tsdb
Package tsdb implements a durable time series database.
Package tsdb implements a durable time series database.
_workspace/src/github.com/influxdb/influxdb/tsdb/engine/wal
Package WAL implements a write ahead log optimized for write throughput that can be put in front of the database index.
Package WAL implements a write ahead log optimized for write throughput that can be put in front of the database index.
_workspace/src/github.com/influxdb/influxdb/tsdb/internal
Package internal is a generated protocol buffer package.
Package internal is a generated protocol buffer package.
_workspace/src/github.com/lib/pq
Package pq is a pure Go Postgres driver for the database/sql package.
Package pq is a pure Go Postgres driver for the database/sql package.
_workspace/src/github.com/lib/pq/listen_example
Below you will find a self-contained Go program which uses the LISTEN / NOTIFY mechanism to avoid polling the database while waiting for more work to arrive.
Below you will find a self-contained Go program which uses the LISTEN / NOTIFY mechanism to avoid polling the database while waiting for more work to arrive.
_workspace/src/github.com/lib/pq/oid
Package oid contains OID constants as defined by the Postgres server.
Package oid contains OID constants as defined by the Postgres server.
_workspace/src/github.com/matttproud/golang_protobuf_extensions/pbutil
Package pbutil provides record length-delimited Protocol Buffer streaming.
Package pbutil provides record length-delimited Protocol Buffer streaming.
_workspace/src/github.com/prometheus/client_golang/extraction
Package extraction decodes Prometheus clients' data streams for consumers.
Package extraction decodes Prometheus clients' data streams for consumers.
_workspace/src/github.com/prometheus/client_golang/model
Package model contains core representation of Prometheus client primitives.
Package model contains core representation of Prometheus client primitives.
_workspace/src/github.com/prometheus/client_golang/text
Package text contains helper functions to parse and create text-based exchange formats.
Package text contains helper functions to parse and create text-based exchange formats.
_workspace/src/github.com/prometheus/client_model/go
Package io_prometheus_client is a generated protocol buffer package.
Package io_prometheus_client is a generated protocol buffer package.
_workspace/src/github.com/samuel/go-zookeeper/zk
Package zk is a native Go client library for the ZooKeeper orchestration service.
Package zk is a native Go client library for the ZooKeeper orchestration service.
_workspace/src/github.com/stretchr/objx
objx - Go package for dealing with maps, slices, JSON and other data.
objx - Go package for dealing with maps, slices, JSON and other data.
_workspace/src/github.com/stretchr/testify/assert
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
_workspace/src/github.com/stretchr/testify/mock
Provides a system by which it is possible to mock your objects and verify calls are happening as expected.
Provides a system by which it is possible to mock your objects and verify calls are happening as expected.
_workspace/src/github.com/stretchr/testify/require
Alternative testing tools which stop test execution if test failed.
Alternative testing tools which stop test execution if test failed.
_workspace/src/golang.org/x/crypto/bcrypt
Package bcrypt implements Provos and Mazières's bcrypt adaptive hashing algorithm.
Package bcrypt implements Provos and Mazières's bcrypt adaptive hashing algorithm.
_workspace/src/golang.org/x/crypto/blowfish
Package blowfish implements Bruce Schneier's Blowfish encryption algorithm.
Package blowfish implements Bruce Schneier's Blowfish encryption algorithm.
_workspace/src/gopkg.in/dancannon/gorethink.v1
Package gorethink implements a Go driver for RethinkDB Current version: v1.0.0 (RethinkDB v2.0) For more in depth information on how to use RethinkDB check out the API docs at http://rethinkdb.com/api
Package gorethink implements a Go driver for RethinkDB Current version: v1.0.0 (RethinkDB v2.0) For more in depth information on how to use RethinkDB check out the API docs at http://rethinkdb.com/api
_workspace/src/gopkg.in/mgo.v2
Package mgo offers a rich MongoDB driver for Go.
Package mgo offers a rich MongoDB driver for Go.
_workspace/src/gopkg.in/mgo.v2/bson
Package bson is an implementation of the BSON specification for Go: http://bsonspec.org It was created as part of the mgo MongoDB driver for Go, but is standalone and may be used on its own without the driver.
Package bson is an implementation of the BSON specification for Go: http://bsonspec.org It was created as part of the mgo MongoDB driver for Go, but is standalone and may be used on its own without the driver.
_workspace/src/gopkg.in/mgo.v2/internal/scram
Pacakage scram implements a SCRAM-{SHA-1,etc} client per RFC5802.
Pacakage scram implements a SCRAM-{SHA-1,etc} client per RFC5802.
_workspace/src/gopkg.in/mgo.v2/testserver
WARNING: This package was replaced by mgo.v2/dbtest.
WARNING: This package was replaced by mgo.v2/dbtest.
_workspace/src/gopkg.in/mgo.v2/txn
The txn package implements support for multi-document transactions.
The txn package implements support for multi-document transactions.
cmd
all
all
lustre2
Lustre 2.x telegraf plugin Lustre (http://lustre.org/) is an open-source, parallel file system for HPC environments.
Lustre 2.x telegraf plugin Lustre (http://lustre.org/) is an open-source, parallel file system for HPC environments.
system/ps/common
gopsutil is a port of psutil(http://pythonhosted.org/psutil/).
gopsutil is a port of psutil(http://pythonhosted.org/psutil/).
system/ps/disk
Package binary implements simple translation between numbers and byte sequences and encoding and decoding of varints.
Package binary implements simple translation between numbers and byte sequences and encoding and decoding of varints.
system/ps/process
Package binary implements simple translation between numbers and byte sequences and encoding and decoding of varints.
Package binary implements simple translation between numbers and byte sequences and encoding and decoding of varints.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL