m3

module
v0.0.0-...-9649366 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 21, 2019 License: Apache-2.0

README

M3 GoDoc Build Status Coverage Status FOSSA Status Gitter chat

M3 Logo

Distributed TSDB and Query Engine, Prometheus Sidecar, Metrics Aggregator, and more. "More" now includes Graphite storage and query engine!

More information:

Community meetings

M3 contributors and maintainers have monthly (every four weeks) meetings at 11:00 AM (USA Pacific) on Tuesday.

An initial agenda is posted to the community meeting shared Google doc a day before each meeting, and everyone is welcome to suggest additional topics or other agendas.

You can add the following public Google Calendar if you're using Google Calendar (so that any changes are reflected in your own calendar when the invite is updated): https://calendar.google.com/calendar?cid=aWc5YWFxZWw0azFmMnRoZDA2dHBtZzZva2tAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

For those not using Google Calendar, please download and import the following iCalendar (.ics) files to calendar system for the monthly invite: https://calendar.google.com/calendar/ical/ig9aaqel4k1f2thd06tpmg6okk%40group.calendar.google.com/public/basic.ics

Join Zoom Meeting: us04web.zoom.us/j/519434268

Test it out

The easiest way to testing out M3 is to follow one of the guides from the documentation. For a fully comprehensive getting started guide, see our single node how-to.

Starting a node
# to build a local m3dbnode process
make m3dbnode (note that we currently require at least Go 1.10 or higher)

# run it with the sample configuration
./bin/m3dbnode -f ./src/dbnode/config/m3dbnode-local-etcd.yml

To cross-compile and build for Linux AMD64 build with make m3dbnode-linux-amd64.

Creating a namespace to store metrics
curl -X POST http://localhost:7201/api/v1/database/create -d '{
  "type": "local",
  "namespaceName": "default",
  "retentionTime": "2h"
}'
Test RPC

To test out some of the functionality of M3DB there are some user friendly HTTP JSON APIs that you can use. These use the DB node cluster service endpoints.

Note: performance sensitive users are expected to use the more performant endpoints via either the Go src/dbnode/client/Session API, or the GRPC endpoints exposed via src/coordinator.

Write a datapoint
curl http://localhost:9003/writetagged -s -X POST -d '{
  "namespace": "default",
  "id": "foo",
  "tags": [
    {
      "name": "__name__",
      "value": "user_login"
    },
    {
      "name": "city",
      "value": "new_york"
    },
    {
      "name": "endpoint",
      "value": "/request"
    }
  ],
  "datapoint": {
    "timestamp":'"$(date +"%s")"',
    "value": 42.123456789
  }
}'
Query for reverse indexed time series data
curl http://localhost:9003/query -s -X POST -d '{
  "namespace": "default",
  "query": {
    "regexp": {
      "field": "city",
      "regexp": ".*"
    }
  },
  "rangeStart": 0,
  "rangeEnd":'"$(date +"%s")"'
}' | jq .

Building with Docker

A Dockerfile is included for both development and production deployment purposes. It uses a multi-stage build in order to produce a lightweight production image from a single Dockerfile. Accordingly, it requires Docker 17.05 or later to build.

docker build -f docker/m3dbnode/Dockerfile -t m3dbnode:$(git rev-parse head) .
docker run --name m3dbnode m3dbnode:$(git rev-parse head)

If you wish to build an image with the source code included you can stop the build after the builder stage:

docker build -f docker/m3dbnode/Dockerfile -t m3dbnode:$(git rev-parse head) --target builder .

Configuration

The default Docker image will start a single m3dbnode process with an embedded etcd instance to mimic a production environment. If you would like to further customize the configuration, you must provide your own and mount it into the container:

docker run --name m3dbnode -v /host/config.yml:/etc/m3dbnode/myconfig.yml m3dbnode:tag -f /etc/m3dbnode/myconfig.yml

This project is released under the Apache License, Version 2.0.

Directories

Path Synopsis
examples
scripts
md5
src
aggregator/aggregation/quantile/cm
Package cm implements the Cormode-Muthukrishnan algorithm for computing biased quantiles over data streams from "Effective Computation of Biased Quantiles over Data Streams" and is largely based on statsite's implementation in C: https://github.com/statsite/statsite/blob/master/src/cm_quantile.c
Package cm implements the Cormode-Muthukrishnan algorithm for computing biased quantiles over data streams from "Effective Computation of Biased Quantiles over Data Streams" and is largely based on statsite's implementation in C: https://github.com/statsite/statsite/blob/master/src/cm_quantile.c
aggregator/aggregation/quantile/tdigest
Package tdigest implements the t-digest algorithm for accurate on-line accumulation of rank-based statistics such as quantiles from "Computing Extremely Accurate Quantiles Using t-Digests" and is based on Ted Dunning's implementation in Java: https://github.com/tdunning/t-digest/blob/master/src/main/java/com/tdunning/math/stats/
Package tdigest implements the t-digest algorithm for accurate on-line accumulation of rank-based statistics such as quantiles from "Computing Extremely Accurate Quantiles Using t-Digests" and is based on Ted Dunning's implementation in Java: https://github.com/tdunning/t-digest/blob/master/src/main/java/com/tdunning/math/stats/
aggregator/aggregator
Package aggregator is a generated GoMock package.
Package aggregator is a generated GoMock package.
aggregator/aggregator/handler
Package handler is a generated GoMock package.
Package handler is a generated GoMock package.
aggregator/aggregator/handler/writer
Package writer is a generated GoMock package.
Package writer is a generated GoMock package.
aggregator/client
Package client is a generated GoMock package.
Package client is a generated GoMock package.
aggregator/generated/proto/flush
Package flush is a generated protocol buffer package.
Package flush is a generated protocol buffer package.
aggregator/hash
Package hash is a temporary measure used as in between while m3aggregator is upgraded to use native source generated maps now accessible in m3x, these types are missing from m3x when native source generated maps were added.
Package hash is a temporary measure used as in between while m3aggregator is upgraded to use native source generated maps now accessible in m3x, these types are missing from m3x when native source generated maps were added.
aggregator/runtime
Package runtime is a generated GoMock package.
Package runtime is a generated GoMock package.
aggregator/tools/deploy
Package deploy is a generated GoMock package.
Package deploy is a generated GoMock package.
cluster/client
Package client is a generated GoMock package.
Package client is a generated GoMock package.
cluster/generated/proto/changesetpb
Package changesetpb is a generated protocol buffer package.
Package changesetpb is a generated protocol buffer package.
cluster/generated/proto/changesettest
Package changesettest is a generated protocol buffer package.
Package changesettest is a generated protocol buffer package.
cluster/generated/proto/commonpb
Package commonpb is a generated protocol buffer package.
Package commonpb is a generated protocol buffer package.
cluster/generated/proto/kvtest
Package kvtest is a generated protocol buffer package.
Package kvtest is a generated protocol buffer package.
cluster/generated/proto/metadatapb
Package metadatapb is a generated protocol buffer package.
Package metadatapb is a generated protocol buffer package.
cluster/generated/proto/placementpb
Package placementpb is a generated protocol buffer package.
Package placementpb is a generated protocol buffer package.
cluster/generated/proto/testpb
Package testpb is a generated protocol buffer package.
Package testpb is a generated protocol buffer package.
cluster/kv
Package kv is a generated GoMock package.
Package kv is a generated GoMock package.
cluster/placement
Package placement is a generated GoMock package.
Package placement is a generated GoMock package.
cluster/services
Package services is a generated GoMock package.
Package services is a generated GoMock package.
cluster/services/leader
Package leader provides functionality for etcd-backed leader elections.
Package leader provides functionality for etcd-backed leader elections.
cluster/services/leader/campaign
Package campaign encapsulates the state of a campaign.
Package campaign encapsulates the state of a campaign.
cluster/services/leader/election
Package election provides a wrapper around a subset of the Election functionality of etcd's concurrency package with error handling for common failure scenarios such as lease expiration.
Package election provides a wrapper around a subset of the Election functionality of etcd's concurrency package with error handling for common failure scenarios such as lease expiration.
cmd/services/m3coordinator/downsample
Package downsample is a generated GoMock package.
Package downsample is a generated GoMock package.
cmd/services/m3coordinator/ingest
Package ingest is a generated GoMock package.
Package ingest is a generated GoMock package.
cmd/services/m3dbnode/config
Package config is a generated GoMock package.
Package config is a generated GoMock package.
cmd/tools/carbon_load/main
carbon_load is a tool for load testing carbon ingestion.
carbon_load is a tool for load testing carbon ingestion.
cmd/tools/docs_test/main
docs_test is a tool to run sanity tests on the documentation before publishing.
docs_test is a tool to run sanity tests on the documentation before publishing.
collector/reporter
Package reporter is a generated GoMock package.
Package reporter is a generated GoMock package.
ctl/service/r2/store
Package store is a generated GoMock package.
Package store is a generated GoMock package.
dbnode/client
Package client is a generated GoMock package.
Package client is a generated GoMock package.
dbnode/digest
Package digest is a generated GoMock package.
Package digest is a generated GoMock package.
dbnode/encoding
Package encoding is a generated GoMock package.
Package encoding is a generated GoMock package.
dbnode/generated/proto/index
Package index is a generated protocol buffer package.
Package index is a generated protocol buffer package.
dbnode/generated/proto/namespace
Package namespace is a generated protocol buffer package.
Package namespace is a generated protocol buffer package.
dbnode/generated/proto/pagetoken
Package pagetoken is a generated protocol buffer package.
Package pagetoken is a generated protocol buffer package.
dbnode/generated/proto/snapshot
Package snapshot is a generated protocol buffer package.
Package snapshot is a generated protocol buffer package.
dbnode/generated/thrift/rpc
Package rpc is a generated GoMock package.
Package rpc is a generated GoMock package.
dbnode/namespace
Package namespace is a generated GoMock package.
Package namespace is a generated GoMock package.
dbnode/namespace/kvadmin
Package kvadmin is a generated GoMock package.
Package kvadmin is a generated GoMock package.
dbnode/persist
Package persist is a generated GoMock package.
Package persist is a generated GoMock package.
dbnode/persist/fs
Package fs is a generated GoMock package.
Package fs is a generated GoMock package.
dbnode/persist/fs/commitlog
Package commitlog is a generated GoMock package.
Package commitlog is a generated GoMock package.
dbnode/retention
Package retention is a generated GoMock package.
Package retention is a generated GoMock package.
dbnode/runtime
Package runtime is a generated GoMock package.
Package runtime is a generated GoMock package.
dbnode/storage
Package storage is a generated GoMock package.
Package storage is a generated GoMock package.
dbnode/storage/block
Package block is a generated GoMock package.
Package block is a generated GoMock package.
dbnode/storage/bootstrap
Package bootstrap is a generated GoMock package.
Package bootstrap is a generated GoMock package.
dbnode/storage/index
Package index is a generated GoMock package.
Package index is a generated GoMock package.
dbnode/storage/series
Package series is a generated GoMock package.
Package series is a generated GoMock package.
dbnode/topology
Package topology is a generated GoMock package.
Package topology is a generated GoMock package.
dbnode/ts
Package ts is a generated GoMock package.
Package ts is a generated GoMock package.
dbnode/x/xio
Package xio is a generated GoMock package.
Package xio is a generated GoMock package.
m3em/build
Package build is a generated GoMock package.
Package build is a generated GoMock package.
m3em/generated/proto/heartbeat
Package heartbeat is a generated protocol buffer package.
Package heartbeat is a generated protocol buffer package.
m3em/generated/proto/m3em
Package m3em is a generated GoMock package.
Package m3em is a generated GoMock package.
m3em/node
Package node is a generated GoMock package.
Package node is a generated GoMock package.
m3em/os/exec/mocks
Package exec is a generated GoMock package.
Package exec is a generated GoMock package.
m3em/os/fs/mocks
Package fs is a generated GoMock package.
Package fs is a generated GoMock package.
m3ninx/doc
Package doc is a generated GoMock package.
Package doc is a generated GoMock package.
m3ninx/generated/proto/fswriter
Package fswriter is a generated protocol buffer package.
Package fswriter is a generated protocol buffer package.
m3ninx/generated/proto/querypb
Package querypb is a generated protocol buffer package.
Package querypb is a generated protocol buffer package.
m3ninx/index
Package index is a generated GoMock package.
Package index is a generated GoMock package.
m3ninx/index/segment
Package segment is a generated GoMock package.
Package segment is a generated GoMock package.
m3ninx/index/segment/fst
Package fst is a generated GoMock package.
Package fst is a generated GoMock package.
m3ninx/index/segment/mem
Package mem is a generated GoMock package.
Package mem is a generated GoMock package.
m3ninx/persist
Package persist is a generated GoMock package.
Package persist is a generated GoMock package.
m3ninx/postings
Package postings is a generated GoMock package.
Package postings is a generated GoMock package.
m3ninx/search
Package search is a generated GoMock package.
Package search is a generated GoMock package.
m3ninx/search/proptest
Package proptest contains a collection of generative tests to validate query behaviour in a variety of situations.
Package proptest contains a collection of generative tests to validate query behaviour in a variety of situations.
m3nsch/generated/proto/m3nsch
Package m3nsch is a generated protocol buffer package.
Package m3nsch is a generated protocol buffer package.
metrics/encoding/protobuf
Package protobuf is a generated GoMock package.
Package protobuf is a generated GoMock package.
metrics/generated/proto/aggregationpb
Package aggregationpb is a generated protocol buffer package.
Package aggregationpb is a generated protocol buffer package.
metrics/generated/proto/metricpb
Package metricpb is a generated protocol buffer package.
Package metricpb is a generated protocol buffer package.
metrics/generated/proto/pipelinepb
Package pipelinepb is a generated protocol buffer package.
Package pipelinepb is a generated protocol buffer package.
metrics/generated/proto/policypb
Package policypb is a generated protocol buffer package.
Package policypb is a generated protocol buffer package.
metrics/generated/proto/rulepb
Package rulepb is a generated protocol buffer package.
Package rulepb is a generated protocol buffer package.
metrics/generated/proto/transformationpb
Package transformationpb is a generated protocol buffer package.
Package transformationpb is a generated protocol buffer package.
metrics/integration
Package integration contains integration tests.
Package integration contains integration tests.
metrics/matcher
Package matcher is a generated GoMock package.
Package matcher is a generated GoMock package.
metrics/metric/id
Package id is a generated GoMock package.
Package id is a generated GoMock package.
metrics/metric/id/m3
Package m3 describes m3 metric id information.
Package m3 describes m3 metric id information.
metrics/rules
Package rules is a generated GoMock package.
Package rules is a generated GoMock package.
msg/consumer
Package consumer is a generated GoMock package.
Package consumer is a generated GoMock package.
msg/generated/proto/msgpb
Package msgpb is a generated protocol buffer package.
Package msgpb is a generated protocol buffer package.
msg/generated/proto/topicpb
Package topicpb is a generated protocol buffer package.
Package topicpb is a generated protocol buffer package.
msg/producer
Package producer is a generated GoMock package.
Package producer is a generated GoMock package.
msg/producer/writer
Package writer is a generated GoMock package.
Package writer is a generated GoMock package.
msg/protocol/proto
Package proto is a generated GoMock package.
Package proto is a generated GoMock package.
msg/topic
Package topic is a generated GoMock package.
Package topic is a generated GoMock package.
query/block
Package block is a generated GoMock package.
Package block is a generated GoMock package.
query/cost
Package cost is a generated GoMock package.
Package cost is a generated GoMock package.
query/executor
Package executor is a generated GoMock package.
Package executor is a generated GoMock package.
query/executor/transform
Package transform is a generated GoMock package.
Package transform is a generated GoMock package.
query/functions/temporal
Package temporal is a generated GoMock package.
Package temporal is a generated GoMock package.
query/generated/proto/admin
Package admin is a generated protocol buffer package.
Package admin is a generated protocol buffer package.
query/generated/proto/prompb
Package prompb is a generated protocol buffer package.
Package prompb is a generated protocol buffer package.
query/generated/proto/rpcpb
Package rpcpb is a generated protocol buffer package.
Package rpcpb is a generated protocol buffer package.
query/parser/m3ql
Package m3ql is a generated GoMock package.
Package m3ql is a generated GoMock package.
query/storage
Package storage is a generated GoMock package.
Package storage is a generated GoMock package.
query/storage/m3
Package m3 is a generated GoMock package.
Package m3 is a generated GoMock package.
query/ts
Package ts is a generated GoMock package.
Package ts is a generated GoMock package.
x/checked
Package checked is a generated GoMock package.
Package checked is a generated GoMock package.
x/clock
Package clock implements facilities for working with wall clock time.
Package clock implements facilities for working with wall clock time.
x/close
Package close provides utilities for closing resources.
Package close provides utilities for closing resources.
x/config
Package config provides utilities for loading configuration files.
Package config provides utilities for loading configuration files.
x/config/hostid
Package hostid provides a configuration struct for resolving a host ID from YAML.
Package hostid provides a configuration struct for resolving a host ID from YAML.
x/config/listenaddress
Package listenaddress provides a configuration struct for resolving a listen address from YAML.
Package listenaddress provides a configuration struct for resolving a listen address from YAML.
x/cost
Package cost is a library providing utilities for estimating the cost of operations and enforcing limits on them.
Package cost is a library providing utilities for estimating the cost of operations and enforcing limits on them.
x/cost/test
Package test contains testing utilities for the cost package.
Package test contains testing utilities for the cost package.
x/errors
Package errors provides utilities for working with different types errors.
Package errors provides utilities for working with different types errors.
x/hash
Package hash provides implementation of hash functions.
Package hash provides implementation of hash functions.
x/hash/jump
Package jump implements the jump consistent hash algorithm as described in "A Fast, Minimal Memory, Consistent Hash Algorithm"[1].
Package jump implements the jump consistent hash algorithm as described in "A Fast, Minimal Memory, Consistent Hash Algorithm"[1].
x/ident
Package ident is a generated GoMock package.
Package ident is a generated GoMock package.
x/instrument
Package instrument implements functions to make instrumenting code, including metrics and logging, easier.
Package instrument implements functions to make instrumenting code, including metrics and logging, easier.
x/net
Package net implements functions for running network servers.
Package net implements functions for running network servers.
x/net/http/cors
Package cors handles cross-origin HTTP requests (CORS).
Package cors handles cross-origin HTTP requests (CORS).
x/panicmon
Package panicmon provides a method of spawning processes and monitoring its behavior via generic pluggable handlers.
Package panicmon provides a method of spawning processes and monitoring its behavior via generic pluggable handlers.
x/pool
Package pool is a generated GoMock package.
Package pool is a generated GoMock package.
x/pprof
Package pprof provides a function for registering a HTTP handler for pprof endpoints.
Package pprof provides a function for registering a HTTP handler for pprof endpoints.
x/process
Package process provides functions for inspecting processes.
Package process provides functions for inspecting processes.
x/resource
Package resource describes require for object lifecycle management.
Package resource describes require for object lifecycle management.
x/retry
Package retry provides utilities for retrying functions.
Package retry provides utilities for retrying functions.
x/serialize
Package serialize is a generated GoMock package.
Package serialize is a generated GoMock package.
x/server
Package server implements a network server.
Package server implements a network server.
x/sync
Package sync implements synchronization facililites such as worker pools.
Package sync implements synchronization facililites such as worker pools.
x/tcp
Package tcp implements a tcp listener.
Package tcp implements a tcp listener.
x/test
Package test contains utility methods for testing.
Package test contains utility methods for testing.
x/test/testmarshal
Package testmarshal provides some assertions around marshalling/unmarshalling (serialization/deserialization) behavior for types.
Package testmarshal provides some assertions around marshalling/unmarshalling (serialization/deserialization) behavior for types.
x/time
Package time implement facilities for working with time.
Package time implement facilities for working with time.
x/unsafe
Package unsafe contains operations that step around the type safety of Go programs.
Package unsafe contains operations that step around the type safety of Go programs.
x/watch
Package watch provides utilities for watching resources for changes.
Package watch provides utilities for watching resources for changes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL