datadogconnector

README

Datadog Connector

Status
Distributions	contrib
Issues
Code Owners	@mx-psi, @dineshg13
Emeritus	@gbbr

Supported Pipeline Types

Exporter Pipeline Type	Receiver Pipeline Type	Stability Level
traces	metrics	beta
traces	traces	beta

Description

The Datadog Connector is a connector component that computes Datadog APM Stats pre-sampling in the event that your traces pipeline is sampled using components such as the tailsamplingprocessor or probabilisticsamplerprocessor.

The connector is most applicable when using the sampling components such as the tailsamplingprocessor, or the probabilisticsamplerprocessor in one of your pipelines. The sampled pipeline should be duplicated and the datadog connector should be added to the the pipeline that is not being sampled to ensure that Datadog APM Stats are accurate in the backend.

Usage

To use the Datadog Connector, add the connector to one set of the duplicated pipelines while sampling the other. The Datadog Connector will compute APM Stats on all spans that it sees. Here is an example on how to add it to a pipeline using the [probabilisticsampler]:

Before

After

# ... processors: # ... probabilistic_sampler: sampling_percentage: 20 # add the "datadog" processor definition datadog: exporters: datadog: api: key: ${env:DD_API_KEY} service: pipelines: traces: receivers: [otlp] # prepend it to the sampler in your pipeline: processors: [batch, datadog, probabilistic_sampler] exporters: [datadog] metrics: receivers: [otlp] processors: [batch] exporters: [datadog]

# ... processors: probabilistic_sampler: sampling_percentage: 20 connectors: # add the "datadog" connector definition and further configurations datadog/connector: exporters: datadog: api: key: ${env:DD_API_KEY} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [datadog/connector] traces/2: # this pipeline uses sampling receivers: [datadog/connector] processors: [batch, probabilistic_sampler] exporters: [datadog] metrics: receivers: [datadog/connector] processors: [batch] exporters: [datadog]

Here we have two traces pipelines that ingest the same data but one is being sampled. The one that is sampled has its data sent to the datadog backend for you to see the sampled subset of the total traces sent across. The other non-sampled pipeline of traces sends its data to the metrics pipeline to be used in the APM stats. This unsampled pipeline gives the full picture of how much data the application emits in traces.

Configurations

connectors:
    datadog/connector:
      traces:
        ## @param ignore_resources - list of strings - optional
        ## A blacklist of regular expressions can be provided to disable certain traces based on their resource name
        ## all entries must be surrounded by double quotes and separated by commas.
        #
        # ignore_resources: ["(GET|POST) /healthcheck"]

        ## @param span_name_remappings - map of key/value pairs - optional
        ## A map of Datadog span operation name keys and preferred name valuues to update those names to. This can be used to
        ## automatically map Datadog Span Operation Names to an updated value, and is useful when a user wants to
        ## shorten or modify span names to something more user friendly in the case of instrumentation libraries with
        ## particularly verbose names.
        #
        # span_name_remappings:
        #   io.opentelemetry.javaagent.spring.client: spring.client
        #   instrumentation:express.server: express
        #   go.opentelemetry.io_contrib_instrumentation_net_http_otelhttp.client: http.client

        ## @param span_name_as_resource_name - use OpenTelemetry semantic convention for span naming - optional
        ## Option created to maintain similarity with the OpenTelemetry semantic conventions as discussed in the issue below.
        ## https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/trace/semantic_conventions
        ## https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1909
        #
        # span_name_as_resource_name: true

        ## @param compute_stats_by_span_kind - enables APM stats computation based on `span.kind` - optional
        ## If set to true, enables an additional stats computation check on spans to see they have an eligible `span.kind` (server, consumer, client, producer).
        ## If enabled, a span with an eligible `span.kind` will have stats computed. If disabled, only top-level and measured spans will have stats computed.
        ## NOTE: For stats computed from OTel traces, only top-level spans are considered when this option is off.
        #
        # compute_stats_by_span_kind: true

        ## @param peer_tags_aggregation - enables aggregation of peer related tags in Datadog exporter - optional
        ## If set to true, enables aggregation of peer related tags (e.g., `peer.service`, `db.instance`, etc.) in Datadog exporter.
        ## If disabled, aggregated trace stats will not include these tags as dimensions on trace metrics.
        ## For the best experience with peer tags, Datadog also recommends enabling `compute_stats_by_span_kind`.
        ## If you are using an OTel tracer, it's best to have both enabled because client/producer spans with relevant peer tags
        ## may not be marked by Datadog exporter as top-level spans.
        ## If enabling both causes Datadog exporter to consume too many resources, try disabling `compute_stats_by_span_kind` first.
        ## A high cardinality of peer tags or APM resources can also contribute to higher CPU and memory consumption.
        ## You can check for the cardinality of these fields by making trace search queries in the Datadog UI.
        ## The default list of peer tags can be found in https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/stats/concentrator.go.
        #
        # peer_tags_aggregation: false

        ## @param trace_buffer - specifies the number of outgoing trace payloads to buffer before dropping - optional
        ## If unset, the default value is 1000.
        ## If you start seeing log messages like `Payload in channel full. Dropped 1 payload.` in the datadog exporter, consider
        ## setting a higher `trace_buffer` to avoid traces being dropped.
        #
        # trace_buffer: 1000

        ## @param peer_tags - [BETA] Optional list of supplementary peer tags that go beyond the defaults. The Datadog backend validates all tags
        ## and will drop ones that are unapproved. The default set of peer tags can be found at
        ## https://github.com/DataDog/datadog-agent/blob/505170c4ac8c3cbff1a61cf5f84b28d835c91058/pkg/trace/stats/concentrator.go#L55.
        #
        # peer_tags: ["tag"]

        ## @param resource_attributes_as_container_tags - enables the use of resource attributes as container tags - Optional
        ## A list of resource attributes that should be used as container tags.
        #
        # resource_attributes_as_container_tags: ["could.availability_zone", "could.region"]

NOTE: compute_stats_by_span_kind and peer_tags_aggregation only work when the feature gate connector.datadogconnector.performance is enabled. See below for details on this feature gate.

Feature Gate for Performance

In case you are experiencing high memory usage with Datadog Connector, similar to issue, use the feature gate connector.datadogconnector.performance. With the feature gate enabled, Datadog Connector takes OTLP traces and produces OTLP metric with the name dd.internal.stats.payload. This Metric has an attribute dd.internal.stats.payload that contains the bytes for StatsPayload. With the feature gate, we can use Datadog Connector only in conjunction with Datadog Exporter. Please enable the feature only if needed for performance reasons and higher throughput. Enable the feature gate on all collectors (especially in gateway deployment) in the pipeline that sends data to Datadog. We plan to refactor this component in the future so that the signals produced are usable in any metrics pipeline.

Documentation

Index

• func NewFactory() connector.Factory
• type Config
  • func (c *Config) Validate() error
• type TracesConfig

Functions

func NewFactory

func NewFactory() connector.Factory

NewFactory creates a factory for tailtracer connector.

Types

type Config

type Config struct {
	// Traces defines the Traces specific configuration
	Traces TracesConfig `mapstructure:"traces"`
}

Config defines configuration for the Datadog connector.

func (*Config) Validate

func (c *Config) Validate() error

Validate the configuration for errors. This is required by component.Config.

type TracesConfig

type TracesConfig struct {
	// ignored resources
	// A blocklist of regular expressions can be provided to disable certain traces based on their resource name
	// all entries must be surrounded by double quotes and separated by commas.
	// ignore_resources: ["(GET|POST) /healthcheck"]
	IgnoreResources []string `mapstructure:"ignore_resources"`

	// SpanNameRemappings is the map of datadog span names and preferred name to map to. This can be used to
	// automatically map Datadog Span Operation Names to an updated value. All entries should be key/value pairs.
	// span_name_remappings:
	//   io.opentelemetry.javaagent.spring.client: spring.client
	//   instrumentation:express.server: express
	//   go.opentelemetry.io_contrib_instrumentation_net_http_otelhttp.client: http.client
	SpanNameRemappings map[string]string `mapstructure:"span_name_remappings"`

	// If set to true the OpenTelemetry span name will used in the Datadog resource name.
	// If set to false the resource name will be filled with the instrumentation library name + span kind.
	// The default value is `false`.
	SpanNameAsResourceName bool `mapstructure:"span_name_as_resource_name"`

	// If set to true, enables an additional stats computation check on spans to see they have an eligible `span.kind` (server, consumer, client, producer).
	// If enabled, a span with an eligible `span.kind` will have stats computed. If disabled, only top-level and measured spans will have stats computed.
	// NOTE: For stats computed from OTel traces, only top-level spans are considered when this option is off.
	ComputeStatsBySpanKind bool `mapstructure:"compute_stats_by_span_kind"`

	// If set to true, enables aggregation of peer related tags (e.g., `peer.service`, `db.instance`, etc.) in the datadog connector.
	// If disabled, aggregated trace stats will not include these tags as dimensions on trace metrics.
	// For the best experience with peer tags, Datadog also recommends enabling `compute_stats_by_span_kind`.
	// If you are using an OTel tracer, it's best to have both enabled because client/producer spans with relevant peer tags
	// may not be marked by the datadog connector as top-level spans.
	// If enabling both causes the datadog connector to consume too many resources, try disabling `compute_stats_by_span_kind` first.
	// A high cardinality of peer tags or APM resources can also contribute to higher CPU and memory consumption.
	// You can check for the cardinality of these fields by making trace search queries in the Datadog UI.
	// The default list of peer tags can be found in https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/stats/concentrator.go.
	PeerTagsAggregation bool `mapstructure:"peer_tags_aggregation"`

	// [BETA] Optional list of supplementary peer tags that go beyond the defaults. The Datadog backend validates all tags
	// and will drop ones that are unapproved. The default set of peer tags can be found at
	// https://github.com/DataDog/datadog-agent/blob/505170c4ac8c3cbff1a61cf5f84b28d835c91058/pkg/trace/stats/concentrator.go#L55.
	PeerTags []string `mapstructure:"peer_tags"`

	// TraceBuffer specifies the number of Datadog Agent TracerPayloads to buffer before dropping.
	// The default value is 1000.
	TraceBuffer int `mapstructure:"trace_buffer"`

	// ResourceAttributesAsContainerTags specifies the list of resource attributes to be used as container tags.
	ResourceAttributesAsContainerTags []string `mapstructure:"resource_attributes_as_container_tags"`
}

TracesConfig defines the traces specific configuration options