m3

module
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 7, 2022 License: Apache-2.0

README

M3

GoDoc Build Status FOSSA Status

M3 Logo

Distributed TSDB and Query Engine, Prometheus Sidecar, Metrics Aggregator, and more such as Graphite storage and query engine.

Table of Contents

More Information

Community Meetings

You can find recordings of past meetups here: https://vimeo.com/user/120001164/folder/2290331.

Install

Dependencies

The simplest and quickest way to try M3 is to use Docker, read the M3 quickstart section for other options.

This example uses jq to format the output of API calls. It is not essential for using M3DB.

Usage

The below is a simplified version of the M3 quickstart guide, and we suggest you read that for more details.

  1. Start a Container
docker run -p 7201:7201 -p 7203:7203 --name m3db -v $(pwd)/m3db_data:/var/lib/m3db quay.io/m3db/m3dbnode:v1.0.0
  1. Create a Placement and Namespace
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/database/create -d '{
  "type": "local",
  "namespaceName": "default",
  "retentionTime": "12h"
}' | jq .
  1. Ready a Namespace
curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
  "name": "default"
}' | jq .
  1. Write Metrics
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
  "tags": 
    {
      "__name__": "third_avenue",
      "city": "new_york",
      "checkout": "1"
    },
    "timestamp": '\"$(date "+%s")\"',
    "value": 3347.26
}'
  1. Query Results

Linux

curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  -d "query=third_avenue" \
  -d "start=$(date "+%s" -d "45 seconds ago")" \
  -d "end=$( date +%s )" \
  -d "step=5s" | jq .  

macOS/BSD

curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  -d "query=third_avenue > 6000" \
  -d "start=$(date -v -45S "+%s")" \
  -d "end=$( date +%s )" \
  -d "step=5s" | jq .

Contributing

You can ask questions and give feedback in the following ways:

M3 welcomes pull requests, read contributing guide to help you get setup for building and contributing to M3.


This project is released under the Apache License, Version 2.0.

Directories

Path Synopsis
examples
scripts
md5
src
aggregator/aggregation/quantile/cm
Package cm implements the Cormode-Muthukrishnan algorithm for computing biased quantiles over data streams from "Effective Computation of Biased Quantiles over Data Streams" and is largely based on statsite's implementation in C: https://github.com/statsite/statsite/blob/master/src/cm_quantile.c
Package cm implements the Cormode-Muthukrishnan algorithm for computing biased quantiles over data streams from "Effective Computation of Biased Quantiles over Data Streams" and is largely based on statsite's implementation in C: https://github.com/statsite/statsite/blob/master/src/cm_quantile.c
aggregator/aggregator
Package aggregator is a generated GoMock package.
Package aggregator is a generated GoMock package.
aggregator/aggregator/handler
Package handler is a generated GoMock package.
Package handler is a generated GoMock package.
aggregator/aggregator/handler/writer
Package writer is a generated GoMock package.
Package writer is a generated GoMock package.
aggregator/client
Package client is a generated GoMock package.
Package client is a generated GoMock package.
aggregator/generated/proto/flush
Package flush is a generated protocol buffer package.
Package flush is a generated protocol buffer package.
aggregator/hash
Package hash is a temporary measure used as in between while m3aggregator is upgraded to use native source generated maps now accessible in m3x, these types are missing from m3x when native source generated maps were added.
Package hash is a temporary measure used as in between while m3aggregator is upgraded to use native source generated maps now accessible in m3x, these types are missing from m3x when native source generated maps were added.
aggregator/integration
Package integration contains integration tests for aggregator.
Package integration contains integration tests for aggregator.
aggregator/runtime
Package runtime is a generated GoMock package.
Package runtime is a generated GoMock package.
aggregator/tools/deploy
Package deploy is a generated GoMock package.
Package deploy is a generated GoMock package.
cluster/client
Package client is a generated GoMock package.
Package client is a generated GoMock package.
cluster/generated/proto/changesetpb
Package changesetpb is a generated protocol buffer package.
Package changesetpb is a generated protocol buffer package.
cluster/generated/proto/changesettest
Package changesettest is a generated protocol buffer package.
Package changesettest is a generated protocol buffer package.
cluster/generated/proto/commonpb
Package commonpb is a generated protocol buffer package.
Package commonpb is a generated protocol buffer package.
cluster/generated/proto/kvpb
Package kvpb is a generated protocol buffer package.
Package kvpb is a generated protocol buffer package.
cluster/generated/proto/kvtest
Package kvtest is a generated protocol buffer package.
Package kvtest is a generated protocol buffer package.
cluster/generated/proto/metadatapb
Package metadatapb is a generated protocol buffer package.
Package metadatapb is a generated protocol buffer package.
cluster/generated/proto/placementpb
Package placementpb is a generated protocol buffer package.
Package placementpb is a generated protocol buffer package.
cluster/generated/proto/testpb
Package testpb is a generated protocol buffer package.
Package testpb is a generated protocol buffer package.
cluster/kv
Package kv is a generated GoMock package.
Package kv is a generated GoMock package.
cluster/placement
Package placement is a generated GoMock package.
Package placement is a generated GoMock package.
cluster/placement/algo
Package algo implements different types of placement algorithms
Package algo implements different types of placement algorithms
cluster/placementhandler
Package placementhandler provides debug/admin placement operations via HTTP
Package placementhandler provides debug/admin placement operations via HTTP
cluster/services
Package services is a generated GoMock package.
Package services is a generated GoMock package.
cluster/services/leader
Package leader provides functionality for etcd-backed leader elections.
Package leader provides functionality for etcd-backed leader elections.
cluster/services/leader/campaign
Package campaign encapsulates the state of a campaign.
Package campaign encapsulates the state of a campaign.
cluster/services/leader/election
Package election provides a wrapper around a subset of the Election functionality of etcd's concurrency package with error handling for common failure scenarios such as lease expiration.
Package election provides a wrapper around a subset of the Election functionality of etcd's concurrency package with error handling for common failure scenarios such as lease expiration.
cmd/services/m3aggregator/config
Package config contains configuration for the aggregator.
Package config contains configuration for the aggregator.
cmd/services/m3coordinator/downsample
Package downsample is a generated GoMock package.
Package downsample is a generated GoMock package.
cmd/services/m3coordinator/ingest
Package ingest is a generated GoMock package.
Package ingest is a generated GoMock package.
cmd/services/m3coordinator/ingest/carbon
Package ingestcarbon implements a carbon ingester.
Package ingestcarbon implements a carbon ingester.
cmd/services/m3coordinator/model
Package model contains model definitions for m3 coordinator.
Package model contains model definitions for m3 coordinator.
cmd/services/m3dbnode/server
Package server is a package for starting servers for M3 components.
Package server is a package for starting servers for M3 components.
cmd/tools/carbon_load/main
carbon_load is a tool for load testing carbon ingestion.
carbon_load is a tool for load testing carbon ingestion.
cmd/tools/m3ctl/topics
Package topics implements topic endpoint interaction.
Package topics implements topic endpoint interaction.
ctl/service/r2/store
Package store is a generated GoMock package.
Package store is a generated GoMock package.
dbnode/client
Package client is a generated GoMock package.
Package client is a generated GoMock package.
dbnode/digest
Package digest is a generated GoMock package.
Package digest is a generated GoMock package.
dbnode/discovery
Package discovery provides discovery configuration.
Package discovery provides discovery configuration.
dbnode/encoding
Package encoding is a generated GoMock package.
Package encoding is a generated GoMock package.
dbnode/generated/proto/annotation
Package annotation is a generated protocol buffer package.
Package annotation is a generated protocol buffer package.
dbnode/generated/proto/index
Package index is a generated protocol buffer package.
Package index is a generated protocol buffer package.
dbnode/generated/proto/namespace
Package namespace is a generated protocol buffer package.
Package namespace is a generated protocol buffer package.
dbnode/generated/proto/pagetoken
Package pagetoken is a generated protocol buffer package.
Package pagetoken is a generated protocol buffer package.
dbnode/generated/proto/snapshot
Package snapshot is a generated protocol buffer package.
Package snapshot is a generated protocol buffer package.
dbnode/generated/thrift/rpc
Package rpc is a generated GoMock package.
Package rpc is a generated GoMock package.
dbnode/namespace
Package namespace is a generated GoMock package.
Package namespace is a generated GoMock package.
dbnode/namespace/kvadmin
Package kvadmin is a generated GoMock package.
Package kvadmin is a generated GoMock package.
dbnode/network/server/tchannelthrift
Package tchannelthrift contains code for servicing RPC requests via tchannel.
Package tchannelthrift contains code for servicing RPC requests via tchannel.
dbnode/persist
Package persist is a generated GoMock package.
Package persist is a generated GoMock package.
dbnode/persist/fs
Package fs is a generated GoMock package.
Package fs is a generated GoMock package.
dbnode/persist/fs/commitlog
Package commitlog is a generated GoMock package.
Package commitlog is a generated GoMock package.
dbnode/retention
Package retention is a generated GoMock package.
Package retention is a generated GoMock package.
dbnode/runtime
Package runtime is a generated GoMock package.
Package runtime is a generated GoMock package.
dbnode/server
Package server contains the code to run the dbnode server.
Package server contains the code to run the dbnode server.
dbnode/storage
Package storage is a generated GoMock package.
Package storage is a generated GoMock package.
dbnode/storage/block
Package block is a generated GoMock package.
Package block is a generated GoMock package.
dbnode/storage/bootstrap
Package bootstrap is a generated GoMock package.
Package bootstrap is a generated GoMock package.
dbnode/storage/bootstrap/bootstrapper/commitlog
Package commitlog implements commit log bootstrapping.
Package commitlog implements commit log bootstrapping.
dbnode/storage/bootstrap/bootstrapper/fs
Package fs implements file system bootstrapping.
Package fs implements file system bootstrapping.
dbnode/storage/bootstrap/bootstrapper/peers
Package peers implements peers bootstrapping.
Package peers implements peers bootstrapping.
dbnode/storage/bootstrap/bootstrapper/uninitialized
Package uninitialized implements uninitialized topology bootstrapping.
Package uninitialized implements uninitialized topology bootstrapping.
dbnode/storage/index
Package index is a generated GoMock package.
Package index is a generated GoMock package.
dbnode/storage/index/convert
Package convert contains various conversions.
Package convert contains various conversions.
dbnode/storage/limits
Package limits contains paths to enforce read query limits.
Package limits contains paths to enforce read query limits.
dbnode/storage/limits/permits
Package permits is a generated GoMock package.
Package permits is a generated GoMock package.
dbnode/storage/series
Package series is a generated GoMock package.
Package series is a generated GoMock package.
dbnode/test
Package test is a package for shared test helpers.
Package test is a package for shared test helpers.
dbnode/topology
Package topology is a generated GoMock package.
Package topology is a generated GoMock package.
dbnode/ts/writes
Package writes is a generated GoMock package.
Package writes is a generated GoMock package.
dbnode/x/xio
Package xio is a generated GoMock package.
Package xio is a generated GoMock package.
integration/aggregator
Package aggregator contains integration tests for aggregators.
Package aggregator contains integration tests for aggregators.
integration/prometheus
Package prometheus contains resources for starting a docker-backed Prometheus.
Package prometheus contains resources for starting a docker-backed Prometheus.
integration/repair
Package repair contains integration tests for repair and replication among dbnodes.
Package repair contains integration tests for repair and replication among dbnodes.
integration/resources
Package resources contains integration test resources for spinning up M3 components.
Package resources contains integration test resources for spinning up M3 components.
integration/resources/docker
Package docker contains resources needed to setup docker containers for M3 tests.
Package docker contains resources needed to setup docker containers for M3 tests.
integration/resources/inprocess
Package inprocess contains code for spinning up M3 resources in-process for the sake of integration testing.
Package inprocess contains code for spinning up M3 resources in-process for the sake of integration testing.
integration/resources/net
Package net contains network-related helpers for integration testing.
Package net contains network-related helpers for integration testing.
integration/simple
Package simple contains code simple integration tests that writes and reads.
Package simple contains code simple integration tests that writes and reads.
m3em/build
Package build is a generated GoMock package.
Package build is a generated GoMock package.
m3em/generated/proto/heartbeat
Package heartbeat is a generated protocol buffer package.
Package heartbeat is a generated protocol buffer package.
m3em/generated/proto/m3em
Package m3em is a generated GoMock package.
Package m3em is a generated GoMock package.
m3em/node
Package node is a generated GoMock package.
Package node is a generated GoMock package.
m3em/os/exec/mocks
Package exec is a generated GoMock package.
Package exec is a generated GoMock package.
m3em/os/fs/mocks
Package fs is a generated GoMock package.
Package fs is a generated GoMock package.
m3ninx/doc
Package doc is a generated GoMock package.
Package doc is a generated GoMock package.
m3ninx/generated/proto/fswriter
Package fswriter is a generated protocol buffer package.
Package fswriter is a generated protocol buffer package.
m3ninx/generated/proto/querypb
Package querypb is a generated protocol buffer package.
Package querypb is a generated protocol buffer package.
m3ninx/index
Package index is a generated GoMock package.
Package index is a generated GoMock package.
m3ninx/index/segment
Package segment is a generated GoMock package.
Package segment is a generated GoMock package.
m3ninx/index/segment/fst
Package fst is a generated GoMock package.
Package fst is a generated GoMock package.
m3ninx/index/segment/mem
Package mem is a generated GoMock package.
Package mem is a generated GoMock package.
m3ninx/persist
Package persist is a generated GoMock package.
Package persist is a generated GoMock package.
m3ninx/postings
Package postings is a generated GoMock package.
Package postings is a generated GoMock package.
m3ninx/search
Package search is a generated GoMock package.
Package search is a generated GoMock package.
m3ninx/search/proptest
Package proptest contains a collection of generative tests to validate query behaviour in a variety of situations.
Package proptest contains a collection of generative tests to validate query behaviour in a variety of situations.
metrics/encoding/protobuf
Package protobuf is a generated GoMock package.
Package protobuf is a generated GoMock package.
metrics/generated/proto/aggregationpb
Package aggregationpb is a generated protocol buffer package.
Package aggregationpb is a generated protocol buffer package.
metrics/generated/proto/metricpb
Package metricpb is a generated protocol buffer package.
Package metricpb is a generated protocol buffer package.
metrics/generated/proto/pipelinepb
Package pipelinepb is a generated protocol buffer package.
Package pipelinepb is a generated protocol buffer package.
metrics/generated/proto/policypb
Package policypb is a generated protocol buffer package.
Package policypb is a generated protocol buffer package.
metrics/generated/proto/rulepb
Package rulepb is a generated protocol buffer package.
Package rulepb is a generated protocol buffer package.
metrics/generated/proto/transformationpb
Package transformationpb is a generated protocol buffer package.
Package transformationpb is a generated protocol buffer package.
metrics/integration
Package integration contains integration tests.
Package integration contains integration tests.
metrics/matcher
Package matcher is a generated GoMock package.
Package matcher is a generated GoMock package.
metrics/matcher/namespace
Package namespace provides namespace functionality for the matcher.
Package namespace provides namespace functionality for the matcher.
metrics/metric/id
Package id is a generated GoMock package.
Package id is a generated GoMock package.
metrics/metric/id/m3
Package m3 describes m3 metric id information.
Package m3 describes m3 metric id information.
metrics/rules
Package rules is a generated GoMock package.
Package rules is a generated GoMock package.
msg/consumer
Package consumer is a generated GoMock package.
Package consumer is a generated GoMock package.
msg/generated/mocks
mockgen rules for generating mocks for unexported interfaces (file mode).
mockgen rules for generating mocks for unexported interfaces (file mode).
msg/generated/proto/msgpb
Package msgpb is a generated protocol buffer package.
Package msgpb is a generated protocol buffer package.
msg/generated/proto/topicpb
Package topicpb is a generated protocol buffer package.
Package topicpb is a generated protocol buffer package.
msg/producer
Package producer is a generated GoMock package.
Package producer is a generated GoMock package.
msg/producer/writer
Package writer is a generated GoMock package.
Package writer is a generated GoMock package.
msg/protocol/proto
Package proto is a generated GoMock package.
Package proto is a generated GoMock package.
msg/topic
Package topic is a generated GoMock package.
Package topic is a generated GoMock package.
query/api/v1/handler/database
Package database contains API endpoints for managing the database.
Package database contains API endpoints for managing the database.
query/api/v1/handler/prom
Package prom provides custom handlers that support the prometheus query endpoints.
Package prom provides custom handlers that support the prometheus query endpoints.
query/api/v1/handler/prometheus/native
Package native handles the prometheus api endpoints using m3.
Package native handles the prometheus api endpoints using m3.
query/api/v1/httpd
Package httpd contains http routers.
Package httpd contains http routers.
query/api/v1/middleware
Package middleware contains HTTP middleware functions.
Package middleware contains HTTP middleware functions.
query/api/v1/options
Package options configures query http handlers.
Package options configures query http handlers.
query/api/v1/route
Package route stores common HTTP route handling code
Package route stores common HTTP route handling code
query/api/v1/validators
package validators contains validation logics for the api.
package validators contains validation logics for the api.
query/block
Package block is a generated GoMock package.
Package block is a generated GoMock package.
query/executor
Package executor is a generated GoMock package.
Package executor is a generated GoMock package.
query/executor/transform
Package transform is a generated GoMock package.
Package transform is a generated GoMock package.
query/functions/temporal
Package temporal is a generated GoMock package.
Package temporal is a generated GoMock package.
query/generated/proto/admin
Package admin is a generated protocol buffer package.
Package admin is a generated protocol buffer package.
query/generated/proto/prompb
Package prompb is a generated protocol buffer package.
Package prompb is a generated protocol buffer package.
query/generated/proto/rpcpb
Package rpcpb is a generated protocol buffer package.
Package rpcpb is a generated protocol buffer package.
query/graphite/storage
Package storage is a generated GoMock package.
Package storage is a generated GoMock package.
query/parser/m3ql
Package m3ql is a generated GoMock package.
Package m3ql is a generated GoMock package.
query/promqlengine
Package promqlengine contains utilities for PromQL engine.
Package promqlengine contains utilities for PromQL engine.
query/source
Package source identifies the source of query requests.
Package source identifies the source of query requests.
query/storage
Package storage is a generated GoMock package.
Package storage is a generated GoMock package.
query/storage/m3
Package m3 is a generated GoMock package.
Package m3 is a generated GoMock package.
query/storage/promremote
Package promremote implements storage interface backed by Prometheus remote write capable endpoints.
Package promremote implements storage interface backed by Prometheus remote write capable endpoints.
query/storage/promremote/promremotetest
Package promremotetest provides test utilities.
Package promremotetest provides test utilities.
query/test/compare
Package compare provides comparison functions for testing.
Package compare provides comparison functions for testing.
query/ts
Package ts is a generated GoMock package.
Package ts is a generated GoMock package.
query/util/json
Package json contains logic for writing JSON.
Package json contains logic for writing JSON.
x/checked
Package checked is a generated GoMock package.
Package checked is a generated GoMock package.
x/clock
Package clock implements facilities for working with wall clock time.
Package clock implements facilities for working with wall clock time.
x/config
Package config provides utilities for loading configuration files.
Package config provides utilities for loading configuration files.
x/config/configflag
Package configflag provides a utility for registering config related command line options with the stdlib flag package.
Package configflag provides a utility for registering config related command line options with the stdlib flag package.
x/config/hostid
Package hostid provides a configuration struct for resolving a host ID from YAML.
Package hostid provides a configuration struct for resolving a host ID from YAML.
x/debug
Package debug contains debug helpers for M3 components.
Package debug contains debug helpers for M3 components.
x/debug/ext
Package extdebug contains helpers for more detailed debug information
Package extdebug contains helpers for more detailed debug information
x/debug/test
Package debugtest contains test helper methods
Package debugtest contains test helper methods
x/errors
Package errors provides utilities for working with different types errors.
Package errors provides utilities for working with different types errors.
x/generated
Package generated is a dummy package to make sure mockgen dependencies are pulled in correctly in vendor mode.
Package generated is a dummy package to make sure mockgen dependencies are pulled in correctly in vendor mode.
x/generated/proto/test
Package m3_test is a generated protocol buffer package.
Package m3_test is a generated protocol buffer package.
x/hash
Package hash provides implementation of hash functions.
Package hash provides implementation of hash functions.
x/hash/jump
Package jump implements the jump consistent hash algorithm as described in "A Fast, Minimal Memory, Consistent Hash Algorithm"[1].
Package jump implements the jump consistent hash algorithm as described in "A Fast, Minimal Memory, Consistent Hash Algorithm"[1].
x/http
Package http adds extension to the stdlib http package.
Package http adds extension to the stdlib http package.
x/ident
Package ident is a generated GoMock package.
Package ident is a generated GoMock package.
x/instrument
Package instrument implements functions to make instrumenting code, including metrics and logging, easier.
Package instrument implements functions to make instrumenting code, including metrics and logging, easier.
x/json
Package json allows for easy duck typing of JSON, i.e.
Package json allows for easy duck typing of JSON, i.e.
x/net
Package net implements functions for running network servers.
Package net implements functions for running network servers.
x/net/http/cors
Package cors handles cross-origin HTTP requests (CORS).
Package cors handles cross-origin HTTP requests (CORS).
x/opentelemetry
Package opentelemetry provides Open Telemetry configuration.
Package opentelemetry provides Open Telemetry configuration.
x/panicmon
Package panicmon provides a method of spawning processes and monitoring its behavior via generic pluggable handlers.
Package panicmon provides a method of spawning processes and monitoring its behavior via generic pluggable handlers.
x/pool
Package pool is a generated GoMock package.
Package pool is a generated GoMock package.
x/process
Package process provides functions for inspecting processes.
Package process provides functions for inspecting processes.
x/resource
Package resource describes require for object lifecycle management.
Package resource describes require for object lifecycle management.
x/retry
Package retry provides utilities for retrying functions.
Package retry provides utilities for retrying functions.
x/serialize
Package serialize is a generated GoMock package.
Package serialize is a generated GoMock package.
x/server
Package server implements a network server.
Package server implements a network server.
x/sync
Package sync implements synchronization facililites such as worker pools.
Package sync implements synchronization facililites such as worker pools.
x/tcp
Package tcp implements a tcp listener.
Package tcp implements a tcp listener.
x/test
Package test contains utility methods for testing.
Package test contains utility methods for testing.
x/test/testmarshal
Package testmarshal provides some assertions around marshalling/unmarshalling (serialization/deserialization) behavior for types.
Package testmarshal provides some assertions around marshalling/unmarshalling (serialization/deserialization) behavior for types.
x/time
Package time implement facilities for working with time.
Package time implement facilities for working with time.
x/unsafe
Package unsafe contains operations that step around the type safety of Go programs.
Package unsafe contains operations that step around the type safety of Go programs.
x/watch
Package watch provides utilities for watching resources for changes.
Package watch provides utilities for watching resources for changes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL