snitch

README

Snitch

This tool allows to parse log files and send statistics to statsd endpoint

Startup options

• {string} config - configuration file
• {string} statsd - statsd endpoint
• {string} prefix - statsd global key prefix, e.g. balancer.
• {int} buffer - buffer interval for metrics, 0 default: one metric - one request

Configuration file

Snith has a particular configuration structure in yml:

sources:
- source: name of source
  file: ./test/log/test2.log
  noFollow: false
  mustExists: false
  reOpen: true
  prefix: "balancer.%HOST%"
  delimiter: "\t"
  keys:
    - key: All.$3.$6
      count: true
      timing: $4
      delimiter: " : "
    - key: All.$3.
      timing: $10
      delimiter: " : "

where is:

• {string} source - source name

• {string} file - where the log file is

• {boolean} noFollow - means no follow new lines that are written in the log file, if true

• {boolean} mustExists - log file have to be existed, if true

• {boolean} reOpen - re-open file, if true (e.g. log rotation)

• {string} prefix - statsd key prefix, %HOST% is used as substitution a hostname

• {string} delimiter - column delimiter in log files, it is reasonable to use \t delimiter in log files

• {[]key} keys - list of keys, which would be send to statsd

  • {string} key - statsd metric key, $N - means column position
  • {boolean} count - boolean, means
  • {string} timing - column with time (time should be in seconds, like 0.001 - means 1ms)
  • {string} delimiter - delimiter into the column, e.g. nginx can to write in one column a few values from upstream the request was in

Snitch allows handle a few sources per instance

Docker build

Docker build uses multistage build

docker build -t xjewer/snitch:{tag} .

Run

docker run --rm -v "/var/log/nginx/balancer/:/var/log/:ro" xjewer/snitch:0.1 -file /var/log/for_script.access.log -statsd graphite.local:8125 -buffer 10

Benchmarks

Benchmark line parsers is:

goos: darwin
goarch: amd64
pkg: github.com/xjewer/snitch
Benchmark_parser-4   	 2000000	       946 ns/op	     416 B/op	       8 allocs/op
PASS
ok  	github.com/xjewer/snitch	2.890s

Documentation

Index

• Variables
• type Line
  • func NewLine(t string, err error) *Line
  • func (l *Line) GetEntries() []string
  • func (l *Line) GetEntry(i int) (string, error)
  • func (l *Line) Split(sep string)
• type LogReader
  • func NewFileReader(s config.Source, l simplelog.Logger) (LogReader, error)
  • func NewNoopReader(lines chan *Line) LogReader
• type Parser
  • func NewParser(r LogReader, s statsd.Statsd, cfg config.Source) (Parser, error)
• type Processor
  • func NewProcessor(p Parser, r LogReader, l simplelog.Logger) *Processor
  • func (p *Processor) Close() error
  • func (p *Processor) Run()

Variables

var (
	ErrEmptyString   = errors.New("empty string")
	ErrOutboundIndex = errors.New("outbound index")
)

var (
	ErrProcessorIsFinished = errors.New("processor is finished")
	ErrEmptyVarName        = errors.New("empty var name")
)

var (
	ErrReaderIsFinished = errors.New("reader is finished")
)

Types

type Line

type Line struct {
	// contains filtered or unexported fields
}

Line is a simplelog line structure

func NewLine

func NewLine(t string, err error) *Line

NewLine makes new text structure

func (*Line) GetEntries

func (l *Line) GetEntries() []string

func (*Line) GetEntry

func (l *Line) GetEntry(i int) (string, error)

func (*Line) Split

func (l *Line) Split(sep string)

type LogReader

type LogReader interface {
	Close() error
	GetLines(chan<- *Line)
	GetName() string
}

func NewFileReader

func NewFileReader(s config.Source, l simplelog.Logger) (LogReader, error)

NewFileReader returns log reader from files

func NewNoopReader

func NewNoopReader(lines chan *Line) LogReader

NewNoopReader returns no-op LogReader struct

type Parser

type Parser interface {
	HandleLine(*Line) error
}

Parser parses log text from reader and sends statistics

func NewParser

func NewParser(r LogReader, s statsd.Statsd, cfg config.Source) (Parser, error)

NewParser makes new Parser

type Processor

type Processor struct {
	tomb.Tomb
	// contains filtered or unexported fields
}

Processor is a structure which gets the Reader, reads line by line from that and sends them to the Parser

func NewProcessor

func NewProcessor(p Parser, r LogReader, l simplelog.Logger) *Processor

NewProcessor returns the New Processor

func (*Processor) Close

func (p *Processor) Close() error

Close processor and reader

func (*Processor) Run

func (p *Processor) Run()

Run runs handler getting readers's simplelog lines and parse them