loghisto

package
v2.3.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 18, 2016 License: Apache-2.0 Imports: 14 Imported by: 0

README

loghisto

Build Status

A metric system for high performance counters and histograms. Unlike popular metric systems today, this does not destroy the accuracy of histograms by sampling. Instead, a logarithmic bucketing function compresses values, generally within 1% of their true value (although between 0 and 1 the precision loss may not be within this boundary). This allows for extreme compression, which allows us to calculate arbitrarily high percentiles with no loss of accuracy - just a small amount of precision. This is particularly useful for highly-clustered events that are tolerant of a small precision loss, but for which you REALLY care about what the tail looks like, such as measuring latency across a distributed system.

Copied out of my work for the CockroachDB metrics system. Based on an algorithm created by Keith Frost.

running a print benchmark for quick analysis
package main

import (
  "runtime"
  "github.com/spacejam/loghisto"
)

func benchmark() {
  // do some stuff
}

func main() {
  numCPU := runtime.NumCPU()
  runtime.GOMAXPROCS(numCPU)

  desiredConcurrency := uint(100)
  loghisto.PrintBenchmark("benchmark1234", desiredConcurrency, benchmark)
}

results in something like this printed to stdout each second:

2014-12-11 21:41:45 -0500 EST
benchmark1234_count:     2.0171025e+07
benchmark1234_max:       2.4642914167480484e+07
benchmark1234_99.99:     4913.768840299134
benchmark1234_99.9:      1001.2472422902518
benchmark1234_99:        71.24044000732538
benchmark1234_95:        67.03348428941965
benchmark1234_90:        65.68633104092515
benchmark1234_75:        63.07152259993664
benchmark1234_50:        58.739891704145194
benchmark1234_min:       -657.5233632152207           // Corollary: time.Since(time.Now()) is often < 0
benchmark1234_sum:       1.648051169322668e+09
benchmark1234_avg:       81.70388809307748
benchmark1234_agg_avg:   89
benchmark1234_agg_count: 6.0962226e+07
benchmark1234_agg_sum:   5.454779078e+09
sys.Alloc:               1.132672e+06
sys.NumGC:               5741
sys.PauseTotalNs:        1.569390954e+09
sys.NumGoroutine:        113
adding an embedded metric system to your code
import (
  "time"
  "fmt"
  "github.com/spacejam/loghisto"
)
func ExampleMetricSystem() {
  // Create metric system that reports once a minute, and includes stats
  // about goroutines, memory usage and GC.
  includeGoProcessStats := true
  ms := loghisto.NewMetricSystem(time.Minute, includeGoProcessStats)
  ms.Start()

  // create a channel that subscribes to metrics as they are produced once
  // per minute.
  // NOTE: if you allow this channel to fill up, the metric system will NOT
  // block, and  will FORGET about your channel if you fail to unblock the
  // channel after 3 configured intervals (in this case 3 minutes) rather
  // than causing a memory leak.
  myMetricStream := make(chan *loghisto.ProcessedMetricSet, 2)
  ms.SubscribeToProcessedMetrics(myMetricStream)

  // create some metrics
  timeToken := ms.StartTimer("time for creating a counter and histo")
  ms.Counter("some event", 1)
  ms.Histogram("some measured thing", 123)
  timeToken.Stop()

  for m := range myMetricStream {
    fmt.Printf("number of goroutines: %f\n", m.Metrics["sys.NumGoroutine"])
  }

  // if you want to manually unsubscribe from the metric stream
  ms.UnsubscribeFromProcessedMetrics(myMetricStream)

  // to stop and clean up your metric system
  ms.Stop()
}
automatically sending your metrics to OpenTSDB, KairosDB or Graphite
func ExampleExternalSubmitter() {
  includeGoProcessStats := true
  ms := NewMetricSystem(time.Minute, includeGoProcessStats)
  ms.Start()
  // graphite
  s := NewSubmitter(ms, GraphiteProtocol, "tcp", "localhost:7777")
  s.Start()

  // opentsdb / kairosdb
  s := NewSubmitter(ms, OpenTSDBProtocol, "tcp", "localhost:7777")
  s.Start()

  // to tear down:
  s.Shutdown()
}

See code for the Graphite/OpenTSDB protocols for adding your own output plugins, it's pretty simple.

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source 
var Metrics = NewMetricSystem(60*time.Second, true)

Metrics is the default metric system, which collects and broadcasts metrics to subscribers once every 60 seconds. Also includes default system stats.

Functions

func GraphiteProtocol 

func GraphiteProtocol(ms *ProcessedMetricSet) []byte

GraphiteProtocol generates a wire representation of a ProcessedMetricSet for submission to a Graphite Carbon instance using the plaintext protocol.

func OpenTSDBProtocol 

func OpenTSDBProtocol(ms *ProcessedMetricSet) []byte

OpenTSDBProtocol generates a wire representation of a ProcessedMetricSet for submission to an OpenTSDB instance.

func PrintBenchmark 

func PrintBenchmark(name string, concurrency uint, op func())

PrintBenchmark will run the provided function at the specified concurrency, time the operation, and once per second write the following information to standard out:

2014-08-09 17:44:57 -0400 EDT raft_AppendLogEntries_count: 16488 raft_AppendLogEntries_max: 3.982478339757623e+07 raft_AppendLogEntries_99.99: 3.864778314316012e+07 raft_AppendLogEntries_99.9: 3.4366224772310276e+06 raft_AppendLogEntries_99: 2.0228126576114902e+06 raft_AppendLogEntries_50: 469769.7083161708 raft_AppendLogEntries_min: 129313.15075081984 raft_AppendLogEntries_sum: 9.975892639594093e+09 raft_AppendLogEntries_avg: 605039.5827022133 raft_AppendLogEntries_agg_avg: 618937 raft_AppendLogEntries_agg_count: 121095 raft_AppendLogEntries_agg_sum: 7.4950269894e+10 sys.Alloc: 997328 sys.NumGC: 1115 sys.PauseTotalNs: 2.94946542e+08 sys.NumGoroutine: 26

Types

type MetricSystem 

type MetricSystem struct {
	// contains filtered or unexported fields
}

MetricSystem facilitates the collection and distribution of metrics.

Example 
ms := NewMetricSystem(time.Microsecond, true)
ms.Start()
myMetricStream := make(chan *ProcessedMetricSet, 2)
ms.SubscribeToProcessedMetrics(myMetricStream)

timeToken := ms.StartTimer("submit_metrics")
ms.Counter("range_splits", 1)
ms.Histogram("some_ipc_latency", 123)
timeToken.Stop()

processedMetricSet := <-myMetricStream
ms.UnsubscribeFromProcessedMetrics(myMetricStream)

m := processedMetricSet.Metrics

example := []struct {
	Name  string
	Value float64
}{
	{
		"total range splits during the process lifetime",
		m["range_splits"],
	}, {
		"range splits in this period",
		m["range_splits_rate"],
	}, {
		"some_ipc 99.9th percentile",
		m["some_ipc_latency_99.9"],
	}, {
		"some_ipc max",
		m["some_ipc_latency_max"],
	}, {
		"some_ipc calls this period",
		m["some_ipc_latency_count"],
	}, {
		"some_ipc calls during the process lifetime",
		m["some_ipc_latency_agg_count"],
	}, {
		"some_ipc total latency this period",
		m["some_ipc_latency_sum"],
	}, {
		"some_ipc mean this period",
		m["some_ipc_latency_avg"],
	}, {
		"some_ipc aggregate man",
		m["some_ipc_latency_agg_avg"],
	}, {
		"time spent submitting metrics this period",
		m["submit_metrics_sum"],
	}, {
		"number of goroutines",
		m["sys.NumGoroutine"],
	}, {
		"time spent in GC",
		m["sys.PauseTotalNs"],
	},
}
for _, nameValue := range example {
	var result string
	if nameValue.Value == float64(0) {
		result = "NOT present"
	} else {
		result = "present"
	}
	fmt.Println(nameValue.Name, result)
}
ms.Stop()

Output:

total range splits during the process lifetime present
range splits in this period present
some_ipc 99.9th percentile present
some_ipc max present
some_ipc calls this period present
some_ipc calls during the process lifetime present
some_ipc total latency this period present
some_ipc mean this period present
some_ipc aggregate man present
time spent submitting metrics this period present
number of goroutines present
time spent in GC present

func NewMetricSystem 

func NewMetricSystem(interval time.Duration, sysStats bool) *MetricSystem

NewMetricSystem returns a new metric system that collects and broadcasts metrics after each interval.

func (*MetricSystem) Counter 

func (ms *MetricSystem) Counter(name string, amount uint64)

Counter is used for recording a running count of the total occurrences of a particular event. A rate is also exported for the amount that a counter has increased during an interval of this MetricSystem.

func (*MetricSystem) DeregisterGaugeFunc 

func (ms *MetricSystem) DeregisterGaugeFunc(name string)

DeregisterGaugeFunc deregisters a function for the <name> metric.

func (*MetricSystem) Histogram 

func (ms *MetricSystem) Histogram(name string, value float64)

Histogram is used for generating rich metrics, such as percentiles, from periodically occurring continuous values.

func (*MetricSystem) RegisterGaugeFunc 

func (ms *MetricSystem) RegisterGaugeFunc(name string, f func() float64)

RegisterGaugeFunc registers a function to be called at each interval whose return value will be used to populate the <name> metric.

func (*MetricSystem) SpecifyPercentiles 

func (ms *MetricSystem) SpecifyPercentiles(percentiles map[string]float64)

SpecifyPercentiles allows users to override the default collected and reported percentiles.

func (*MetricSystem) Start 

func (ms *MetricSystem) Start()

Start spawns a goroutine for merging metrics into caches from metric submitters, and a reaper goroutine that harvests metrics at the default interval of every 60 seconds.

func (*MetricSystem) StartTimer 

func (ms *MetricSystem) StartTimer(name string) TimerToken

StartTimer begins a timer and returns a token which is required for halting the timer. This allows for concurrent timings under the same name.

func (*MetricSystem) Stop 

func (ms *MetricSystem) Stop()

Stop shuts down a MetricSystem

func (*MetricSystem) SubscribeToProcessedMetrics 

func (ms *MetricSystem) SubscribeToProcessedMetrics(
	metricStream chan *ProcessedMetricSet)

SubscribeToProcessedMetrics registers a channel to receive ProcessedMetricSets periodically generated by reaper at each interval.

func (*MetricSystem) SubscribeToRawMetrics 

func (ms *MetricSystem) SubscribeToRawMetrics(metricStream chan *RawMetricSet)

SubscribeToRawMetrics registers a channel to receive RawMetricSets periodically generated by reaper at each interval.

func (*MetricSystem) UnsubscribeFromProcessedMetrics 

func (ms *MetricSystem) UnsubscribeFromProcessedMetrics(
	metricStream chan *ProcessedMetricSet)

UnsubscribeFromProcessedMetrics registers a channel to receive ProcessedMetricSets periodically generated by reaper at each interval.

func (*MetricSystem) UnsubscribeFromRawMetrics 

func (ms *MetricSystem) UnsubscribeFromRawMetrics(
	metricStream chan *RawMetricSet)

UnsubscribeFromRawMetrics registers a channel to receive RawMetricSets periodically generated by reaper at each interval.

type ProcessedMetricSet 

type ProcessedMetricSet struct {
	Time    time.Time
	Metrics map[string]float64
}

ProcessedMetricSet contains human-readable metrics that may also be suitable for storage in time-series databases.

type RawMetricSet 

type RawMetricSet struct {
	Time       time.Time
	Counters   map[string]uint64
	Rates      map[string]uint64
	Histograms map[string]map[int16]*uint64
	Gauges     map[string]float64
}

RawMetricSet contains metrics in a form that supports generation of percentiles and other rich statistics.

type Submitter 

type Submitter struct {
	DestinationNetwork string
	DestinationAddress string
	// contains filtered or unexported fields
}

Submitter encapsulates the state of a metric submitter.

func NewSubmitter 

func NewSubmitter(metricSystem *MetricSystem,
	serializer func(*ProcessedMetricSet) []byte, destinationNetwork string,
	destinationAddress string) *Submitter

NewSubmitter creates a Submitter that receives metrics off of a specified metric channel, serializes them using the provided serialization function, and attempts to send them to the specified destination.

func (*Submitter) Shutdown 

func (s *Submitter) Shutdown()

Shutdown shuts down a submitter

func (*Submitter) Start 

func (s *Submitter) Start()

Start creates the goroutines that receive, serialize, and send metrics.

type TimerToken 

type TimerToken struct {
	Name         string
	Start        time.Time
	MetricSystem *MetricSystem
}

TimerToken facilitates concurrent timings of durations of the same label.

func (*TimerToken) Stop 

func (tt *TimerToken) Stop() time.Duration

Stop stops a timer given by StartTimer, submits a Histogram of its duration in nanoseconds, and returns its duration in nanoseconds.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.