metrictank

module
v0.13.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 28, 2019 License: AGPL-3.0

README

Metrictank

Circle CI Go Report Card GoDoc

Introduction

Metrictank is a multi-tenant timeseries engine for Graphite and friends. It provides long term storage, high availability, efficient storage, retrieval and processing for large scale environments.

GrafanaLabs has been running metrictank in production since December 2015. It currently requires an external datastore like Cassandra or Bigtable, and we highly recommend using Kafka to support clustering, as well as a clustering manager like Kubernetes. This makes it non-trivial to operate, though GrafanaLabs has an on-premise product that makes this process much easier.

Features

  • 100% open source
  • Inspired by the Facebook gorilla paper. Most notably, the heavily compressed chunks dramatically lower cpu, memory and storage requirements.
  • Writeback RAM cache, serving most data out of memory.
  • Graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.
  • Can also act as a Graphite server itself, though the functions processing library is only partially implemented, metrictank proxies requests to Graphite if it can't handle the required processing (for those requests it will degrade to just being the backend storage)
  • Can also be used with Prometheus (but the experience won't be as good as something built just for prometheus, which we're also working on)
  • Accurate, flexible rollups by storing min/max/sum/count (which also gives us average). So we can do consolidation (combined runtime+archived) accurately and correctly, unlike most other graphite backends like whisper
  • Flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
  • Input options: carbon, metrics2.0, kafka, Prometheus (soon: json or msgpack over http)
  • Guards against excessive data requests
  • Data backfill/import from whisper

Limitations

  • No performance/availability isolation between tenants per instance. (only data isolation)
  • Minimum computation locality: we move the data from storage to processing code, which is both metrictank and graphite.
  • Backlog replaying and queries can be made faster. A Go GC issue may occasionally inflate response times.
  • We use metrics2.0 in native input protocol and indexes, but barely do anything with it yet.
  • can't overwrite old data. We support reordering the most recent time window but that's it. (unless you restart MT)

Interesting design characteristics (feature or limitation... up to you)

  • Upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and possibly re-assigning the primary role. Otherwise data loss of current chunks will be incurred. See operations guide
  • clustering works best with an orchestrator like kubernetes. MT itself does not automate master promotions. See clustering for more.
  • Only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression), No text support.
  • Only uint32 unix timestamps in second resolution. For higher resolution, consider streaming directly to grafana
  • No data locality: doesn't seem needed yet to put related series together.

Docs

installation, configuration and operation.
features in-depth
Other

Releases and versioning

  • releases and changelog

  • we aim to keep master stable and vet code before merging to master

  • We're pre-1.0 but adopt semver for our 0.MAJOR.MINOR format. The rules are simple:

    • MAJOR version for incompatible API or functionality changes
    • MINOR version when you add functionality in a backwards-compatible manner, and

    We don't do patch level releases since minor releases are frequent enough.

License

Copyright 2016-2019 Grafana Labs

This software is distributed under the terms of the GNU Affero General Public License.

Some specific packages have a different license:

Directories

Path Synopsis
api
middleware
models
response
Package batch implements batched processing for slices of points in particular aggregations
 Package batch implements batched processing for slices of points in particular aggregations
partitioner
cmd
metrictank
mt-aggs-explain
mt-explain
mt-index-cat
mt-index-cat/out
mt-index-migrate
mt-index-prune
mt-kafka-mdm-sniff
mt-kafka-mdm-sniff-out-of-order
mt-kafka-persist-sniff
mt-keygen
mt-schemas-explain
mt-split-metrics-by-ttl
mt-store-cat
mt-store-cp-experimental
mt-update-ttl
mt-view-boundaries
mt-whisper-importer-reader
mt-whisper-importer-writer
cmd-dev
mt-simulate-memory-idx-lock-contention
mt-simulate-memory-idx-lock-contention/runner
Package conf reads config data from two of carbon's config files * storage-schemas.conf (old and new retention format) see https://graphite.readthedocs.io/en/0.9.9/config-carbon.html#storage-schemas-conf * storage-aggregation.conf see http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-aggregation-conf as well as our own file index-rules.conf it also adds defaults (the same ones as graphite), so that even if nothing is matched in the user provided schemas or aggregations, a setting is *always* found uses some modified snippets from github.com/lomik/go-carbon and github.com/lomik/go-whisper
 Package conf reads config data from two of carbon's config files * storage-schemas.conf (old and new retention format) see https://graphite.readthedocs.io/en/0.9.9/config-carbon.html#storage-schemas-conf * storage-aggregation.conf see http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-aggregation-conf as well as our own file index-rules.conf it also adds defaults (the same ones as graphite), so that even if nothing is matched in the user provided schemas or aggregations, a setting is *always* found uses some modified snippets from github.com/lomik/go-carbon and github.com/lomik/go-whisper
Package consolidation provides an abstraction for consolidators
 Package consolidation provides an abstraction for consolidators
argument types.
 argument types.
tagquery
idx
bigtable
cassandra
memory
Package in provides interfaces, concrete implementations, and utilities to ingest data into metrictank
 Package in provides interfaces, concrete implementations, and utilities to ingest data into metrictank
carbon
package carbon provides a traditional carbon input for metrictank note: it does not support the "carbon2.0" protocol that serializes metrics2.0 into a plaintext carbon-like protocol
 package carbon provides a traditional carbon input for metrictank note: it does not support the "carbon2.0" protocol that serializes metrics2.0 into a plaintext carbon-like protocol
kafkamdm
prometheus
Package logger provides a custom TextFormatter for use with the github.com/sirupsen/logrus library.
 Package logger provides a custom TextFormatter for use with the github.com/sirupsen/logrus library.
Package mdata stands for "managed data" or "metrics data" if you will it has all the stuff to keep metric data in memory, store it, and synchronize save states over the network
 Package mdata stands for "managed data" or "metrics data" if you will it has all the stuff to keep metric data in memory, store it, and synchronize save states over the network
cache
cache/accnt
chunk
package chunk encodes timeseries in chunks of data see devdocs/chunk-format.md for more information.
 package chunk encodes timeseries in chunks of data see devdocs/chunk-format.md for more information.
chunk/archive
chunk/tsz
Package tsz implements time-series compression it is a fork of https://github.com/dgryski/go-tsz which implements http://www.vldb.org/pvldb/vol8/p1816-teller.pdf see devdocs/chunk-format.md for more info Package tsz implements time-series compression it is a fork of https://github.com/dgryski/go-tsz which implements http://www.vldb.org/pvldb/vol8/p1816-teller.pdf see devdocs/chunk-format.md for more info
 Package tsz implements time-series compression it is a fork of https://github.com/dgryski/go-tsz which implements http://www.vldb.org/pvldb/vol8/p1816-teller.pdf see devdocs/chunk-format.md for more info Package tsz implements time-series compression it is a fork of https://github.com/dgryski/go-tsz which implements http://www.vldb.org/pvldb/vol8/p1816-teller.pdf see devdocs/chunk-format.md for more info
errors
importer
notifierKafka
msg
stacktest
docker
fakemetrics
fakemetrics/out
fakemetrics/out/carbon
fakemetrics/out/kafkamdm
fakemetrics/out/kafkamdm/keycache
grafana
graphite
tests/predictable-tag-query-render-workload
track
Package stats provides functionality for instrumenting metrics and reporting them The metrics can be user specified, or sourced from the runtime (reporters) To use this package correctly, you must instantiate exactly 1 output.
 Package stats provides functionality for instrumenting metrics and reporting them The metrics can be user specified, or sourced from the runtime (reporters) To use this package correctly, you must instantiate exactly 1 output.
config
bigtable
cassandra
package test contains utility functions used by tests/benchmarks in various packages
 package test contains utility functions used by tests/benchmarks in various packages
package tracing contains some helpers to make working with opentracing a tad simpler
 package tracing contains some helpers to make working with opentracing a tad simpler

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.