opentelemetry-collector

module
Version: v0.2.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 30, 2020 License: Apache-2.0

README

OpenTelemetry Collector

IMPORTANT: The OpenTelemetry Collector is in alpha. For now, it is recommended you use the OpenCensus Service.

Build Status GoDoc Gitter chat Coverage Status

We hold regular meetings. See details at community page.

Table of contents:

Introduction

The OpenTelemetry Collector can receive traces and metrics from processes instrumented by OpenTelemetry or other monitoring/tracing libraries (e.g. Jaeger, Prometheus, etc.), handles aggregation and smart sampling, and exports traces and metrics to one or more monitoring/tracing backends.

Some frameworks and ecosystems are now providing out-of-the-box instrumentation by using OpenTelemetry, but the user is still expected to register an exporter in order to send data. This is a problem during an incident. Even though our users can benefit from having more diagnostics data coming out of services already instrumented with OpenTelemetry, they have to modify their code to register an exporter and redeploy. Asking our users to recompile and redeploy is not an ideal during an incident. In addition, currently users need to decide which service backend they want to export to, before they distribute their binary instrumented by OpenTelemetry.

The OpenTelemetry Collector is trying to eliminate these requirements. With the OpenTelemetry Collector, users do not need to redeploy or restart their applications as long as it has the OpenTelemetry exporter. All they need to do is just configure and deploy the OpenTelemetry Collector separately. The OpenTelemetry Collector will then automatically receive traces and metrics and export to any backend of the user's choice.

Some supplementail documents to review include:

Deployment

The OpenTelemetry Collector can be deployed in a variety of different ways depending on requirements. Currently the OpenTelemetry Collector consists of a single binary and two deployment methods:

  1. Agent running with the application or on the same host as the application (e.g. binary, sidecar, or daemonset)
  2. Collector running as a standalone application (e.g. container or deployment)

While the same binary is used for either deployment method, the configuration between the two may differ depending on requirements (e.g. queue size and feature-set enabled).

deployment-models

Getting Started

Demo

Instructions for setting up an end-to-end demo environment can be found here

Kubernetes

Apply the sample YAML file:

$ kubectl apply -f example/k8s.yaml
Standalone

Create an Agent configuration file based on the options described below. By default, the Agent has the opencensus receiver enabled, but no exporters configured.

Build the Agent and start it with the example configuration:

$ ./bin/$(go env GOOS)/otelcol  --config ./examples/demo/otel-agent-config.yaml
$ 2018/10/08 21:38:00 Running OpenTelemetry receiver as a gRPC service at "localhost:55678"

Create an Collector configuration file based on the options described below. By default, the Collector has the opencensus receiver enabled, but no exporters configured.

Build the Collector and start it with the example configuration:

$ make otelcol
$ ./bin/$($GOOS)/otelcol --config ./examples/demo/otel-collector-config.yaml

Run the demo application:

$ go run "$(go env GOPATH)/src/github.com/open-telemetry/opentelemetry-collector/examples/main.go"

You should be able to see the traces in your exporter(s) of choice. If you stop the otelcol, the example application will stop exporting. If you run it again, exporting will resume.

Configuration

The OpenTelemetry Collector is configured via a YAML file. In general, at least one enabled receiver and one enabled exporter needs to be configured.

Note This documentation is still in progress. For any questions, please reach out in the OpenTelemetry Gitter or refer to the issues page.

The configuration consists of the following sections:

receivers:
  ...
processors:
  ...
exporters:
  ...
extensions:
  ...
service:
  ...
Receivers

A receiver is how data gets into the OpenTelemetry Collector. One or more receivers must be configured.

A basic example of all available receivers is provided below. For detailed receiver configuration, please see the receiver README.md.

receivers:
  opencensus:
    address: "localhost:55678"

  zipkin:
    address: "localhost:9411"

  jaeger:
    protocols:
      grpc:
      thrift_http:
      thrift_tchannel:
      thrift_compact:
      thrift_binary:

  prometheus:
    config:
      scrape_configs:
        - job_name: "caching_cluster"
          scrape_interval: 5s
          static_configs:
            - targets: ["localhost:8889"]
Processors

Processors are run on spans between being received and being exported. Processors are optional.

A basic example of all available processors is provided below. For detailed processor configuration, please see the processor README.md.

processors:
  attributes/example:
    actions:
      - key: db.statement
        action: delete
  batch:
    timeout: 5s
    send_batch_size: 1024
  probabilistic_sampler:
    disabled: true
  span:
    name:
      from_attributes: ["db.svc", "operation"]
      separator: "::"
  queued_retry: {}
  tail_sampling:
    policies:
      - name: policy1
        type: rate_limiting
        rate_limiting:
          spans_per_second: 100
Exporters

An exporter is how you send data to one or more backends/destinations. One or more exporters can be configured. By default, no exporters are configured on the OpenTelemetry Collector.

A basic example of all available exporters is provided below. For detailed exporter configuration, please see the exporter README.md.

exporters:
  opencensus:
    headers: {"X-test-header": "test-header"}
    compression: "gzip"
    cert_pem_file: "server-ca-public.pem" # optional to enable TLS
    endpoint: "localhost:55678"
    reconnection_delay: 2s

  logging:
    loglevel: debug

  jaeger_grpc:
    endpoint: "http://localhost:14250"

  jaeger_thrift_http:
    headers: {"X-test-header": "test-header"}
    timeout: 5
    url: "http://localhost:14268/api/traces"

  zipkin:
    url: "http://localhost:9411/api/v2/spans"

  prometheus:
    endpoint: "localhost:8889"
    namespace: "default"
Extensions

Extensions are provided to monitor the health of the OpenTelemetry Collector. Extensions are optional.

A basic example of all available extensions is provided below. For detailed extension configuration, please see the extension README.md.

extensions:
  health_check: {}
  pprof: {}
  zpages: {}
Service

The service section is used to configure what features are enabled in the OpenTelemetry Collector based on configuration found in the receivers, processors, exporters, and extensions sections. The service section consists of two sub-sections:

  • extensions
  • pipelines

Extensions consist of a list of all extensions to enable. For example:

    service:
      extensions: [health_check, pprof, zpages]

Pipelines can be of two types:

  • metrics: collects and processes metrics data.
  • traces: collects and processes trace data.

A pipeline consists of a set of receivers, processors, and exporters. Each receiver/processor/exporter must be specified in the configuration to be included in a pipeline and each receiver/processor/exporter can be used in more than one pipeline.

Note: For processor(s) referenced in multiple pipelines, each pipeline will get a separate instance of that processor(s). This is in contrast to receiver(s)/exporter(s) referenced in multiple pipelines, one instance of a receiver/exporter is reference by all the pipelines.

The following is an example pipeline configuration. For more information, refer to pipeline documentation

service:
  pipelines:
    traces:
      receivers: [opencensus, jaeger]
      processors: [batch, queued_retry]
      exporters: [opencensus, zipkin]

Owners

Approvers (@open-telemetry/collector-approvers):

Find more about the approver role in community repository.

Maintainers (@open-telemetry/collector-maintainers):

Find more about the maintainer role in community repository.

Directories

Path Synopsis
Package client contains generic representations of clients connecting to different receivers
Package client contains generic representations of clients connecting to different receivers
cmd
otelcol
Program otelcol is the OpenTelemetry Collector that collects stats and traces and exports to a configured backend.
Program otelcol is the OpenTelemetry Collector that collects stats and traces and exports to a configured backend.
Package receivertest define types and functions used to help test packages implementing the receiver package interfaces.
Package receivertest define types and functions used to help test packages implementing the receiver package interfaces.
Package config implements loading of configuration from Viper configuration.
Package config implements loading of configuration from Viper configuration.
configcheck
Package configcheck has checks to be applied to configuration objects implemented by factories of components used in the OpenTelemetry collector.
Package configcheck has checks to be applied to configuration objects implemented by factories of components used in the OpenTelemetry collector.
configerror
Package configerror contains the common errors caused by malformed configs.
Package configerror contains the common errors caused by malformed configs.
configgrpc
Package configgrpc defines the gRPC configuration settings.
Package configgrpc defines the gRPC configuration settings.
configmodels
Package configmodels defines the data models for entities.
Package configmodels defines the data models for entities.
Package consumer contains interfaces that receive and process consumerdata.
Package consumer contains interfaces that receive and process consumerdata.
consumerdata
Package consumerdata contains data structures that holds proto metrics/spans, node and resource.
Package consumerdata contains data structures that holds proto metrics/spans, node and resource.
consumererror
Package consumererror provides wrappers to easily classify errors.
Package consumererror provides wrappers to easily classify errors.
Package defaults composes the default set of components used by the otel service
Package defaults composes the default set of components used by the otel service
Sample contains a program that exports to the OpenCensus service.
Sample contains a program that exports to the OpenCensus service.
Package exporter contains interfaces that wraps trace/metrics exporter.
Package exporter contains interfaces that wraps trace/metrics exporter.
jaeger/jaegergrpcexporter
Package jaegergrpcexporter implements an exporter that sends trace data to a Jaeger collector gRPC endpoint.
Package jaegergrpcexporter implements an exporter that sends trace data to a Jaeger collector gRPC endpoint.
jaeger/jaegerthrifthttpexporter
Package jaegerthrifthttpexporter implements an exporter that sends trace data to a Jaeger collector Thrift over HTTP endpoint.
Package jaegerthrifthttpexporter implements an exporter that sends trace data to a Jaeger collector Thrift over HTTP endpoint.
Package extension defines service extensions that can be added to the OpenTelemetry service but that not interact if the data pipelines, but provide some functionality to the service, examples: health check endpoint, z-pages, etc.
Package extension defines service extensions that can be added to the OpenTelemetry service but that not interact if the data pipelines, but provide some functionality to the service, examples: health check endpoint, z-pages, etc.
extensiontest
Package extensiontest define types and functions used to help test packages implementing the extension package interfaces.
Package extensiontest define types and functions used to help test packages implementing the extension package interfaces.
healthcheckextension
Package healthcheckextension implements an extension that enables an HTTP endpoint that can be used to check the overall health and status of the service.
Package healthcheckextension implements an extension that enables an HTTP endpoint that can be used to check the overall health and status of the service.
pprofextension
Package pprofextension implements an extension that exposes the golang net/http/pprof (Performance Profiler) in a HTTP endpoint.
Package pprofextension implements an extension that exposes the golang net/http/pprof (Performance Profiler) in a HTTP endpoint.
zpagesextension
Package zpagesextension implements an extension that exposes zPages of properly instrumented components.
Package zpagesextension implements an extension that exposes zPages of properly instrumented components.
Package oterr provides helper functions to create and process OpenTelemetry specific errors
Package oterr provides helper functions to create and process OpenTelemetry specific errors
Package processor contains interfaces that compose trace/metrics consumers.
Package processor contains interfaces that compose trace/metrics consumers.
attributesprocessor
Package attributesprocessor contains the logic to modify attributes of a span.
Package attributesprocessor contains the logic to modify attributes of a span.
memorylimiter
Package memorylimiter provides a processor for OpenTelemetry Service pipeline that drops data on the pipeline according to the current state of memory usage.
Package memorylimiter provides a processor for OpenTelemetry Service pipeline that drops data on the pipeline according to the current state of memory usage.
resourceprocessor
Package resourceprocessor implements a processor for specifying resource labels to be added to OpenCensus trace data and metrics data.
Package resourceprocessor implements a processor for specifying resource labels to be added to OpenCensus trace data and metrics data.
samplingprocessor/tailsamplingprocessor/idbatcher
Package idbatcher defines a pipeline of fixed size in which the elements are batches of ids.
Package idbatcher defines a pipeline of fixed size in which the elements are batches of ids.
samplingprocessor/tailsamplingprocessor/sampling
Package sampling contains the interfaces and data types used to implement the various sampling policies.
Package sampling contains the interfaces and data types used to implement the various sampling policies.
spanprocessor
Package spanprocessor contains logic to modify top level settings of a span, such as its name.
Package spanprocessor contains logic to modify top level settings of a span, such as its name.
opencensusreceiver/ocmetrics
Package ocmetrics is the logic for receiving OpenCensus metrics proto from already instrumented applications and then passing them onto a metricsink instance.
Package ocmetrics is the logic for receiving OpenCensus metrics proto from already instrumented applications and then passing them onto a metricsink instance.
opencensusreceiver/octrace
Package octrace is the logic for receiving OpenCensus trace protobuf defined spans from already instrumented applications and then passing them onto a TraceReceiverSink instance.
Package octrace is the logic for receiving OpenCensus trace protobuf defined spans from already instrumented applications and then passing them onto a TraceReceiverSink instance.
prometheusreceiver
Package prometheusreceiver has the logic for scraping Prometheus metrics from already instrumented applications and then passing them onto a metricsink instance.
Package prometheusreceiver has the logic for scraping Prometheus metrics from already instrumented applications and then passing them onto a metricsink instance.
vmmetricsreceiver
Package vmmetricsreceiver has the logic for scraping VM metrics and then passing them onto a metric consumer instance.
Package vmmetricsreceiver has the logic for scraping VM metrics and then passing them onto a metric consumer instance.
Package service handles the command-line, configuration, and runs the OpenTelemetry Collector.
Package service handles the command-line, configuration, and runs the OpenTelemetry Collector.
builder
Package builder handles the options to build the OpenTelemetry collector pipeline.
Package builder handles the options to build the OpenTelemetry collector pipeline.
testbed module
translator
trace/spandata
Package spandata defines translators from Trace proto spans to OpenCensus Go spanData.
Package spandata defines translators from Trace proto spans to OpenCensus Go spanData.
collector/telemetry
Package telemetry controls the telemetry settings to be used in the collector.
Package telemetry controls the telemetry settings to be used in the collector.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL