grafsy

README

Description

This is a very light proxy for graphite metrics with additional features:

• Taking metrics from network (see configuration) or from file directly
• Buffering metrics if Graphite itself is down
• Function of summing/averaging metrics with a special prefix (see configuration)
• Filtering 'bad' metrics, which are not passing check against regexp
• Periodical sending to Graphite server to avoid traffic pikes

This is a representation of the Grafsy as a Black box

As you can see on diagram host2 lost connection to Graphite. With Grafsy it is completely safe, because it will retry to deliver metrics over and over until it succeed or limits will be reached

This is a simplified representation of internal components

Also I recommend you to see the presentation https://prezi.com/giwit3kyy0vu/grafsy/

Releases

We are using release-based workflow with tags. Please do not add in your CI master branch. Use latest tag. For jenkins it might look like this (+refs/tags/*:refs/remotes/uw/tags/* and */tags/*):

Please look at releases

Configuration

There is a config file which must be located under /etc/grafsy/grafsy.toml
But you can redefine it with option -c
Most of the time you need to use default (recommended) configuration of grafsy, but you can always modify params:

Base

• supervisor - supervisor manager which is used to run Grafsy. e.g. systemd or supervisord. Default is none
• clientSendInterval - the interval, after which client will send data to graphite. In seconds
• metricsPerSecond - maximum amount of metrics which can be processed per second
  In case of problems with connection/amount of metrics, this configuration will take save up to maxMetrics*clientSendInterval metrics in retryDir
  Also these 2 params are exactly allocating memory
• allowedMetrics - regexp of allowed metric. Every metric which is not passing check against regexp will be removed
• log - main log file
• hostname - alias to use instead of os.Hostname() result

Sending and cache

• carbonAddrs - array of carbon metrics receivers.
• connectTimeout - timeout for connecting to carbonAddrs. Timeout for writing metrics themselves will be clientSendInterval-connectTimeout-1. Default 7. In seconds
• localBind - local address:port for local daemon
• metricDir - directory, in which developers or admins can write any file with metrics
• useACL - enables ACL for metricDir to let grafsy read files there with any permissions. Default is false
• retryDir - data, which was not sent will be buffered in this directory per carbon server

Aggregation

• sumPrefix - prefix for metric to sum. Do not forget to include it in allowedMetrics if you change it
• avgPrefix - prefix for metric to calculate average. Do not forget to include it in allowedMetrics if you change it
• minPrefix - prefix for metric to find minimal value. Do not forget to include it in allowedMetrics if you change it
• maxPrefix - prefix for metric to find maximum value. Do not forget to include it in allowedMetrics if you change it
• aggrInterval - summing up interval for metrics with all prefixes. In seconds
• aggrPerSecond - amount of aggregations which grafsy performs per second. If grafsy receives more metrics than aggrPerSecond * aggrInterval - rest will be dropped

Monitoring

• monitoringPath - full path for metrics, send by grafsy itself. "HOSTNAME" will be replaced with os.Hostname() result from GO.
  If os.Hostname() returns result with dots in it - they will be replaced with _.
  You can define your own path. If it does not contain magic "HOSTNAME" word, it will be preserved.
  At the end of your path grafsy will append grafsy.{sent,dropped,got...}
  E.g servers.HOSTNAME.software or servers.my-awesome-hostname
  Default is "HOSTNAME"

Overwrite

Grafsy can overwrite metric name. It might be very useful if you have a software, which has hardcoded path. E.g., PowerDNS 3. You can specify as many overwrites as you want. Each of them must be in separate section:

[[overwrite]]
replaceWhatRegexp = "^(SUM|AVG|MIN|MAX).pdns"
replaceWith = "servers.HOSTNAME.software.pdns"
[[overwrite]]
replaceWhatRegexp = "^pdns"
replaceWith = "servers.HOSTNAME.software.pdns"

This will ask Grafsy to replace all kinds of metric starting with pdns or aggregation prefixes ^(SUM|AVG|MIN|MAX).pdns to servers.HOSTNAME.software.pdns where HOSTNAME will be replaced with os.Hostname() output

Installation

• Install go https://golang.org/doc/install
• Make a proper structure of directories: mkdir -p /opt/go/src /opt/go/bin /opt/go/pkg
• Setup g GOPATH variable: export GOPATH=/opt/go
• Clone this project to src: go get github.com/innogames/grafsy
• Fetch dependencies: cd /opt/go/github.com/innogames/grafsy && go get ./...
• Compile project: go install github.com/innogames/grafsy/grafsy
• Copy config file: mkdir /etc/grafsy && cp /opt/go/src/github.com/innogames/grafsy/grafsy.toml /etc/grafsy/
• Change your settings, e.g. carbonAddrs
• Run it /opt/go/bin/grafsy

Godocs

https://godoc.org/github.com/innogames/grafsy

Documentation

Index

• type Client
  • func (c Client) Run()
• type Config
  • func (conf *Config) GenerateLocalConfig() (*LocalConfig, error)
  • func (conf *Config) LoadConfig(configFile string) error
• type LocalConfig
• type Monitoring
  • func (m *Monitoring) Increase(metric *int, value int)
  • func (m *Monitoring) Run()
• type Server
  • func (s *Server) Run()

Types

type Client

type Client struct {
	// User config.
	Conf *Config

	// Local config.
	Lc *LocalConfig

	// Pointer to Monitoring structure.
	Mon *Monitoring
	// contains filtered or unexported fields
}

Client is a class wich sends metrics to the carbon receivers

func (Client) Run

func (c Client) Run()

Run a client, which: 1) Make monitoring and main channels per carbon server 2) Launchs go routine per carbon server 3) Copy metrics from monitoring and main channels to the carbon server specific Should be run in separate goroutine.

type Config

type Config struct {
	// Supervisor manager which is used to run Grafsy. e.g. systemd.
	// Default is none.
	Supervisor string

	// The interval, after which client will send data to graphite. In seconds.
	ClientSendInterval int

	// Maximum amount of metrics which can be processed per second.
	// In case of problems with connection/amount of metrics,
	// this configuration will take save up to maxMetrics*clientSendInterval metrics in.
	MetricsPerSecond int

	// Real Carbon servers to which client will send all data
	CarbonAddrs []string

	// Timeout for connecting to graphiteAddr.
	// Timeout for writing metrics themselves will be clientSendInterval-connectTimeout-1.
	// Default 7. In seconds.
	ConnectTimeout int

	// Local address:port for local daemon.
	LocalBind string

	// Main log file.
	Log string

	// Directory, in which developers/admins... can write any file with metrics.
	MetricDir string

	// Enables ACL for metricDir to let grafsy read files there with any permissions.
	// Default is false.
	UseACL bool

	// Data, which was not sent will be buffered in this directory.
	RetryDir string

	// Prefix for metric to sum.
	// Do not forget to include it in allowedMetrics if you change it.
	SumPrefix string

	// Prefix for metric to calculate average.
	// Do not forget to include it in allowedMetrics if you change it.
	AvgPrefix string

	// Prefix for metric to find minimal value.
	// Do not forget to include it in allowedMetrics if you change it.
	MinPrefix string

	// Prefix for metric to find maximum value.
	// Do not forget to include it in allowedMetrics if you change it.
	MaxPrefix string

	// Summing up interval for metrics with all prefixes. In seconds.
	AggrInterval int

	// Amount of aggregations which grafsy performs per second.
	// If grafsy receives more metrics than aggrPerSecond*aggrInterval - rest will be dropped.
	AggrPerSecond int

	// Alias to use instead of os.Hostname() result
	Hostname string

	// Full path for metrics, send by grafsy itself.
	// "HOSTNAME" will be replaced with os.Hostname() result from GO.
	// Default is "HOSTNAME"
	MonitoringPath string

	// Regexp of allowed metric.
	// Every metric which is not passing check against regexp will be removed.
	AllowedMetrics string

	// List of metrics to overwrite
	Overwrite []struct {
		// Regexp of metric to replace from config
		ReplaceWhatRegexp string

		// New metric part
		ReplaceWith string
	}
}

Config is the main config specified by user.

func (*Config) GenerateLocalConfig

func (conf *Config) GenerateLocalConfig() (*LocalConfig, error)

GenerateLocalConfig generates LocalConfig with all needed for running server variables based on Config.

func (*Config) LoadConfig

func (conf *Config) LoadConfig(configFile string) error

LoadConfig loads a configFile to a Config structure.

type LocalConfig

type LocalConfig struct {
	// contains filtered or unexported fields
}

LocalConfig is generated based on Config.

type Monitoring

type Monitoring struct {
	// User config.
	Conf *Config

	// Local config.
	Lc *LocalConfig
	// contains filtered or unexported fields
}

Monitoring structure. Based on this self-monitoring will be sent to Graphite.

func (*Monitoring) Increase

func (m *Monitoring) Increase(metric *int, value int)

Increase metric value in the thread safe way

func (*Monitoring) Run

func (m *Monitoring) Run()

Run monitoring. Should be run in separate goroutine.

type Server

type Server struct {
	// User config.
	Conf *Config

	// Local config.
	Lc *LocalConfig

	// Pointer to Monitoring structure.
	Mon *Monitoring
}

The Server class to receive a data

func (*Server) Run

func (s *Server) Run()

Run server. Should be run in separate goroutine.