gateway

README

Gateway

The SignalFx Gateway lets you aggregate metrics and send them to SignalFx. It is a multilingual datapoint demultiplexer that can accept time series data from the carbon (Graphite), collectd or SignalFx protocols and emit those datapoints to a series of servers using the carbon, collectd or SignalFx protocols. We recommend placing the gateway either on the same server as another existing metrics aggregator or on a central server that is already receiving datapoints, such as Graphite's carbon database.

The SignalFx Gateway also acts as a simple proxy and forwarder for SignalFx's Events Ingest and Trace Ingest APIs.

Usage and Configuration

Please refer to the usage and configuration documentation located here for more information on operating the SignalFx Gateway.

Development

If you want to submit patches for the gateway, make sure your code passes travis_check.sh with exit code 0. For help setting up your development environment, it should be enough to mirror the install steps of .travis.yml. You may need to make sure your GOPATH env variable is set correctly.

Code layout

You only need to read this if you want to develop the gateway or understand the gateway's code.

The gateway is divided into two main components: forwarder and listener. The forwarder and listener are glued together by the demultiplexer.

When a listener receives a datapoint, it converts the datapoint into a basic datapoint type. This core datapoint type is then sent to the multiplexer that will send a pointer to that datapoint to each forwarder.

Sometimes there is a loss of fidelity during transmission if a listener and forwarder don't support the same options. While it's impossible to make something understand an option it does not, we don't want to forget support for this option when we translate a datapoint through the multiplexer. We work around this by sometimes encoding the raw representation of the datapoint into the Datapoint object we forward. For example, points from carbon are not only translated into our core datapoint format, but also support ToCarbonLine which allows us to directly convert the abstract datapoint into what it looked like for carbon, which allows us to forward the point to another carbon database exactly as we received it.

All message passing between forwarders, multiplexer, and listeners happen on golang's built in channel abstraction.

Config file format

See the example config file for an example of how configuration looks. Configuration is a JSON file with two important fields: ListenFrom and ForwardTo.

License

The SignalFx Gateway is released under the Apache 2.0 license. See LICENSE for more details.

Dependencies

Note: this list of dependencies applies to both the SignalFx Gateway and its Smart Gateway variant.

Name	License
github.com/apache/thrift/lib/go/thrift	Apache Software License v2.0
github.com/beorn7/perks/quantile	MIT
github.com/coreos/bbolt	MIT
github.com/coreos/etcd	Apache Software License v2.0
github.com/coreos/go-semver/semver	Apache Software License v2.0
github.com/coreos/go-systemd/journal	Apache Software License v2.0
github.com/coreos/pkg/capnslog	Apache Software License v2.0
github.com/davecgh/go-spew/spew	ISC
github.com/dgrijalva/jwt-go	MIT
github.com/ghodss/yaml	MIT
github.com/go-logfmt/logfmt	MIT
github.com/go-stack/stack	MIT
github.com/gobwas/glob	MIT
github.com/gogo/protobuf	BSD 3-Clause
github.com/golang/groupcache/lru	Apache Software License v2.0
github.com/golang/protobuf	BSD 3-Clause
github.com/golang/snappy	BSD 3-Clause
github.com/google/btree	Apache Software License v2.0
github.com/google/go-cmp	BSD 3-Clause
github.com/gopherjs/gopherjs/js	BSD 2-Clause
github.com/gorilla/context	BSD 3-Clause
github.com/gorilla/mux	BSD 3-Clause
github.com/gorilla/websocket	BSD 2-Clause
github.com/grpc-ecosystem/go-grpc-middleware	Apache Software License v2.0
github.com/grpc-ecosystem/go-grpc-prometheus	Apache Software License v2.0
github.com/grpc-ecosystem/grpc-gateway	BSD 3-Clause
github.com/inconshreveable/mousetrap	Apache Software License v2.0
github.com/jaegertracing/jaeger/thrift-gen/jaeger	Apache Software License v2.0
github.com/jonboulle/clockwork	Apache Software License v2.0
github.com/jtolds/gls	MIT
github.com/konsorten/go-windows-terminal-sequences	MIT
github.com/kr/logfmt	MIT
github.com/mailru/easyjson	MIT
github.com/matttproud/golang_protobuf_extensions/pbutil	Apache Software License v2.0
github.com/mdubbyap/timespan	MIT
github.com/opentracing/opentracing-go	Apache Software License v2.0
github.com/pkg/errors	BSD 2-Clause
github.com/pmezard/go-difflib/difflib	BSD 3-Clause
github.com/prometheus/client_golang/prometheus	Apache Software License v2.0
github.com/prometheus/client_model/go	Apache Software License v2.0
github.com/prometheus/common	Apache Software License v2.0
github.com/prometheus/procfs	Apache Software License v2.0
github.com/prometheus/prometheus/prompb	Apache Software License v2.0
github.com/signalfx/com_signalfx_metrics_protobuf	Apache Software License v2.0
github.com/signalfx/embetcd/embetcd	Apache Software License v2.0
github.com/signalfx/go-distribute	Apache Software License v2.0
github.com/signalfx/go-metrics	BSD 3-Clause
github.com/signalfx/gohistogram	MIT
github.com/signalfx/gohelpers	Apache Software License v2.0
github.com/signalfx/golib	Apache Software License v2.0
github.com/signalfx/ondiskencoding	Apache Software License v2.0
github.com/mdubbyap/tdigest	Apache Software License v2.0
github.com/signalfx/xdgbasedir	Apache Software License v2.0
github.com/sirupsen/logrus	MIT
github.com/smartystreets/assertions	MIT
github.com/smartystreets/goconvey	MIT
github.com/soheilhy/cmux	Apache Software License v2.0
github.com/spaolacci/murmur3	BSD 3-Clause
github.com/spf13/cobra	Apache Software License v2.0
github.com/spf13/pflag	BSD 3-Clause
github.com/stretchr/testify/assert	MIT
github.com/tmc/grpc-websocket-proxy/wsproxy	MIT
github.com/uber/tchannel-go	MIT
github.com/ugorji/go	MIT
github.com/xiang90/probing	MIT
go.uber.org/atomic	MIT
go.uber.org/multierr	MIT
go.uber.org/zap	MIT
golang.org/x/crypto	BSD 3-Clause
golang.org/x/lint	BSD 3-Clause
golang.org/x/net	BSD 3-Clause
golang.org/x/sys	BSD 3-Clause
golang.org/x/text	BSD 3-Clause
golang.org/x/time	BSD 3-Clause
golang.org/x/tools	BSD 3-Clause
google.golang.org/googleapis	Apache Software License v2.0
google.golang.org/genproto	Apache Software License v2.0
google.golang.org/grpc	Apache Software License v2.0
gopkg.in/logfmt.v0	MIT
gopkg.in/natefinch/lumberjack.v2	MIT
gopkg.in/stack.v1	MIT
gopkg.in/yaml.v2	Apache Software License v2.0
gotest.tools	Apache Software License v2.0
stathat.com/c/consistent	MIT