Package tracer contains Datadog's core tracing client. It is used to trace requests as they flow across web servers, databases and microservices, giving developers visibility into bottlenecks and troublesome requests. To start the tracer, simply call the start method along with an optional set of options. By default, the trace agent is considered to be found at "localhost:8126". In a setup where this would be different (let's say, we could do:

    defer tracer.Stop()

    The tracing client can perform trace sampling. While the trace agent already samples traces to reduce bandwidth usage, client sampling reduces performance overhead. To make use of it, the package comes with a ready-to-use rate sampler that can be passed to the tracer. To use it and keep only 30% of the requests, one would do:

    s := tracer.NewRateSampler(0.3)

    More precise control of sampling rates can be configured using sampling rules. This can be applied based on span name, service or both, and is used to determine the sampling rate to apply.

    rules := []tracer.SamplingRule{
          // sample 10% of traces with the span name "web.request"
          tracer.NameRule("web.request", 0.1),
          // sample 20% of traces for the service "test-service"
          tracer.ServiceRule("test-service", 0.2),
          // sample 30% of traces when the span name is "db.query" and the service
          // is "postgres.db"
          tracer.NameServiceRule("db.query", "postgres.db", 0.3),
          // sample 100% of traces when service and name match these regular expressions
          {Service: regexp.MustCompile("^test-"), Name: regexp.MustCompile("http\\..*"), Rate: 1.0},
    defer tracer.Stop()

    Sampling rules can also be configured at runtime using the DD_TRACE_SAMPLING_RULES environment variable. When set, it overrides rules set by tracer.WithSamplingRules. The value is a JSON array of objects. Each object must have a "sample_rate", and the "name" and "service" fields are optional.

    export DD_TRACE_SAMPLING_RULES='[{"name": "web.request", "sample_rate": 1.0}]'

    All spans created by the tracer contain a context hereby referred to as the span context. Note that this is different from Go's context. The span context is used to package essential information from a span, which is needed when creating child spans that inherit from it. Thus, a child span is created from a span's span context. The span context can originate from within the same process, but also a different process or even a different machine in the case of distributed tracing.

    To make use of distributed tracing, a span's context may be injected via a carrier into a transport (HTTP, RPC, etc.) to be extracted on the other end and used to create spans that are direct descendants of it. A couple of carrier interfaces which should cover most of the use-case scenarios are readily provided, such as HTTPCarrier and TextMapCarrier. Users are free to create their own, which will work with our propagation algorithm as long as they implement the TextMapReader and TextMapWriter interfaces. An example alternate implementation is the MDCarrier in our gRPC integration.

    As an example, injecting a span's context into an HTTP request would look like this:

    req, err := http.NewRequest("GET", "", nil)
    // ...
    err := tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(req.Header))
    // ...

    Then, on the server side, to continue the trace one would do:

    sctx, err := tracer.Extract(tracer.HTTPHeadersCarrier(req.Header))
    // ...
    span := tracer.StartSpan("child.span", tracer.ChildOf(sctx))

    In the same manner, any means can be used as a carrier to inject a context into a transport. Go's context can also be used as a means to transport spans within the same process. The methods StartSpanFromContext, ContextWithSpan and SpanFromContext exist for this reason.

    Some libraries and frameworks are supported out-of-the-box by using one of our integrations. You can see a list of supported integrations here:


      A basic example demonstrating how to start the tracer, as well as how to create a root span and a child span that is a descendant of it.





      View Source
      const (
      	// DefaultBaggageHeaderPrefix specifies the prefix that will be used in
      	// HTTP headers or text maps to prefix baggage keys.
      	DefaultBaggageHeaderPrefix = "ot-baggage-"
      	// DefaultTraceIDHeader specifies the key that will be used in HTTP headers
      	// or text maps to store the trace ID.
      	DefaultTraceIDHeader = "x-datadog-trace-id"
      	// DefaultParentIDHeader specifies the key that will be used in HTTP headers
      	// or text maps to store the parent ID.
      	DefaultParentIDHeader = "x-datadog-parent-id"
      	// DefaultPriorityHeader specifies the key that will be used in HTTP headers
      	// or text maps to store the sampling priority value.
      	DefaultPriorityHeader = "x-datadog-sampling-priority"


      View Source
      var (
      	// ErrInvalidCarrier is returned when the carrier provided to the propagator
      	// does not implemented the correct interfaces.
      	ErrInvalidCarrier = errors.New("invalid carrier")
      	// ErrInvalidSpanContext is returned when the span context found in the
      	// carrier is not of the expected type.
      	ErrInvalidSpanContext = errors.New("invalid span context")
      	// ErrSpanContextCorrupted is returned when there was a problem parsing
      	// the information found in the carrier.
      	ErrSpanContextCorrupted = errors.New("span context corrupted")
      	// ErrSpanContextNotFound represents missing information in the given carrier.
      	ErrSpanContextNotFound = errors.New("span context not found")


      func ContextWithSpan

      func ContextWithSpan(ctx context.Context, s Span) context.Context

        ContextWithSpan returns a copy of the given context which includes the span s.

        func Extract

        func Extract(carrier interface{}) (ddtrace.SpanContext, error)

          Extract extracts a SpanContext from the carrier. The carrier is expected to implement TextMapReader, otherwise an error is returned. If the tracer is not started, calling this function is a no-op.

          func Flush

          func Flush()

            Flush flushes any buffered traces. Flush is in effect only if a tracer is started. Users do not have to call Flush in order to ensure that traces reach Datadog. It is a convenience method dedicated to a specific use case described below.

            Flush is of use in Lambda environments, where starting and stopping the tracer on each invokation may create too much latency. In this scenario, a tracer may be started and stopped by the parent process whereas the invokation can make use of Flush to ensure any created spans reach the agent.

            func Inject

            func Inject(ctx ddtrace.SpanContext, carrier interface{}) error

              Inject injects the given SpanContext into the carrier. The carrier is expected to implement TextMapWriter, otherwise an error is returned. If the tracer is not started, calling this function is a no-op.

              func Start

              func Start(opts ...StartOption)

                Start starts the tracer with the given set of options. It will stop and replace any running tracer, meaning that calling it several times will result in a restart of the tracer by replacing the current instance with a new one.

                func Stop

                func Stop()

                  Stop stops the started tracer. Subsequent calls are valid but become no-op.


                  type FinishOption

                  type FinishOption = ddtrace.FinishOption

                    FinishOption is a configuration option for FinishSpan. It is aliased in order to help godoc group all the functions returning it together. It is considered more correct to refer to it as the type as the origin, ddtrace.FinishOption.

                    func FinishTime

                    func FinishTime(t time.Time) FinishOption

                      FinishTime sets the given time as the finishing time for the span. By default, the current time is used.

                      func NoDebugStack

                      func NoDebugStack() FinishOption

                        NoDebugStack prevents any error presented using the WithError finishing option from generating a stack trace. This is useful in situations where errors are frequent and performance is critical.

                        func StackFrames

                        func StackFrames(n, skip uint) FinishOption

                          StackFrames limits the number of stack frames included into erroneous spans to n, starting from skip.

                          func WithError

                          func WithError(err error) FinishOption

                            WithError marks the span as having had an error. It uses the information from err to set tags such as the error message, error type and stack trace. It has no effect if the error is nil.

                            type HTTPHeadersCarrier

                            type HTTPHeadersCarrier http.Header

                              HTTPHeadersCarrier wraps an http.Header as a TextMapWriter and TextMapReader, allowing it to be used using the provided Propagator implementation.

                              func (HTTPHeadersCarrier) ForeachKey

                              func (c HTTPHeadersCarrier) ForeachKey(handler func(key, val string) error) error

                                ForeachKey implements TextMapReader.

                                func (HTTPHeadersCarrier) Set

                                func (c HTTPHeadersCarrier) Set(key, val string)

                                  Set implements TextMapWriter.

                                  type Propagator

                                  type Propagator interface {
                                  	// Inject takes the SpanContext and injects it into the carrier.
                                  	Inject(context ddtrace.SpanContext, carrier interface{}) error
                                  	// Extract returns the SpanContext from the given carrier.
                                  	Extract(carrier interface{}) (ddtrace.SpanContext, error)

                                    Propagator implementations should be able to inject and extract SpanContexts into an implementation specific carrier.

                                    func NewPropagator

                                    func NewPropagator(cfg *PropagatorConfig) Propagator

                                      NewPropagator returns a new propagator which uses TextMap to inject and extract values. It propagates trace and span IDs and baggage. To use the defaults, nil may be provided in place of the config.

                                      type PropagatorConfig

                                      type PropagatorConfig struct {
                                      	// BaggagePrefix specifies the prefix that will be used to store baggage
                                      	// items in a map. It defaults to DefaultBaggageHeaderPrefix.
                                      	BaggagePrefix string
                                      	// TraceHeader specifies the map key that will be used to store the trace ID.
                                      	// It defaults to DefaultTraceIDHeader.
                                      	TraceHeader string
                                      	// ParentHeader specifies the map key that will be used to store the parent ID.
                                      	// It defaults to DefaultParentIDHeader.
                                      	ParentHeader string
                                      	// PriorityHeader specifies the map key that will be used to store the sampling priority.
                                      	// It deafults to DefaultPriorityHeader.
                                      	PriorityHeader string

                                        PropagatorConfig defines the configuration for initializing a propagator.

                                        type RateSampler

                                        type RateSampler interface {
                                        	// Rate returns the current sample rate.
                                        	Rate() float64
                                        	// SetRate sets a new sample rate.
                                        	SetRate(rate float64)

                                          RateSampler is a sampler implementation which randomly selects spans using a provided rate. For example, a rate of 0.75 will permit 75% of the spans. RateSampler implementations should be safe for concurrent use.

                                          func NewAllSampler

                                          func NewAllSampler() RateSampler

                                            NewAllSampler is a short-hand for NewRateSampler(1). It is all-permissive.

                                            func NewRateSampler

                                            func NewRateSampler(rate float64) RateSampler

                                              NewRateSampler returns an initialized RateSampler with a given sample rate.

                                              type Sampler

                                              type Sampler interface {
                                              	// Sample returns true if the given span should be sampled.
                                              	Sample(span Span) bool

                                                Sampler is the generic interface of any sampler. It must be safe for concurrent use.

                                                type SamplingRule

                                                type SamplingRule struct {
                                                	Service *regexp.Regexp
                                                	Name    *regexp.Regexp
                                                	Rate    float64
                                                	// contains filtered or unexported fields

                                                  SamplingRule is used for applying sampling rates to spans that match the service name, operation name or both. For basic usage, consider using the helper functions ServiceRule, NameRule, etc.

                                                  func NameRule

                                                  func NameRule(name string, rate float64) SamplingRule

                                                    NameRule returns a SamplingRule that applies the provided sampling rate to spans that match the operation name provided.

                                                    func NameServiceRule

                                                    func NameServiceRule(name string, service string, rate float64) SamplingRule

                                                      NameServiceRule returns a SamplingRule that applies the provided sampling rate to spans matching both the operation and service names provided.

                                                      func RateRule

                                                      func RateRule(rate float64) SamplingRule

                                                        RateRule returns a SamplingRule that applies the provided sampling rate to all spans.

                                                        func ServiceRule

                                                        func ServiceRule(service string, rate float64) SamplingRule

                                                          ServiceRule returns a SamplingRule that applies the provided sampling rate to spans that match the service name provided.

                                                          func (*SamplingRule) MarshalJSON

                                                          func (sr *SamplingRule) MarshalJSON() ([]byte, error)

                                                            MarshalJSON implements the json.Marshaler interface.

                                                            type Span

                                                            type Span = ddtrace.Span

                                                              Span is an alias for ddtrace.Span. It is here to allow godoc to group methods returning ddtrace.Span. It is recommended and is considered more correct to refer to this type as ddtrace.Span instead.

                                                              func SpanFromContext

                                                              func SpanFromContext(ctx context.Context) (Span, bool)

                                                                SpanFromContext returns the span contained in the given context. A second return value indicates if a span was found in the context. If no span is found, a no-op span is returned.

                                                                func StartSpan

                                                                func StartSpan(operationName string, opts ...StartSpanOption) Span

                                                                  StartSpan starts a new span with the given operation name and set of options. If the tracer is not started, calling this function is a no-op.

                                                                  func StartSpanFromContext

                                                                  func StartSpanFromContext(ctx context.Context, operationName string, opts ...StartSpanOption) (Span, context.Context)

                                                                    StartSpanFromContext returns a new span with the given operation name and options. If a span is found in the context, it will be used as the parent of the resulting span. If the ChildOf option is passed, the span from context will take precedence over it as the parent span.

                                                                    type StartOption

                                                                    type StartOption func(*config)

                                                                      StartOption represents a function that can be provided as a parameter to Start.

                                                                      func WithAgentAddr

                                                                      func WithAgentAddr(addr string) StartOption

                                                                        WithAgentAddr sets the address where the agent is located. The default is localhost:8126. It should contain both host and port.

                                                                        func WithAnalytics

                                                                        func WithAnalytics(on bool) StartOption

                                                                          WithAnalytics allows specifying whether Trace Search & Analytics should be enabled for integrations.

                                                                          func WithAnalyticsRate

                                                                          func WithAnalyticsRate(rate float64) StartOption

                                                                            WithAnalyticsRate sets the global sampling rate for sampling APM events.

                                                                            func WithDebugMode

                                                                            func WithDebugMode(enabled bool) StartOption

                                                                              WithDebugMode enables debug mode on the tracer, resulting in more verbose logging.

                                                                              func WithDebugStack

                                                                              func WithDebugStack(enabled bool) StartOption

                                                                                WithDebugStack can be used to globally enable or disable the collection of stack traces when spans finish with errors. It is enabled by default. This is a global version of the NoDebugStack FinishOption.

                                                                                func WithDogstatsdAddress

                                                                                func WithDogstatsdAddress(addr string) StartOption

                                                                                  WithDogstatsdAddress specifies the address to connect to for sending metrics to the Datadog Agent. If not set, it defaults to "localhost:8125" or to the combination of the environment variables DD_AGENT_HOST and DD_DOGSTATSD_PORT. This option is in effect when WithRuntimeMetrics is enabled.

                                                                                  func WithEnv

                                                                                  func WithEnv(env string) StartOption

                                                                                    WithEnv sets the environment to which all traces started by the tracer will be submitted. The default value is the environment variable DD_ENV, if it is set.

                                                                                    func WithGlobalTag

                                                                                    func WithGlobalTag(k string, v interface{}) StartOption

                                                                                      WithGlobalTag sets a key/value pair which will be set as a tag on all spans created by tracer. This option may be used multiple times.

                                                                                      func WithHTTPClient

                                                                                      func WithHTTPClient(client *http.Client) StartOption

                                                                                        WithHTTPClient specifies the HTTP client to use when emitting spans to the agent.

                                                                                        func WithHTTPRoundTripper

                                                                                        func WithHTTPRoundTripper(r http.RoundTripper) StartOption

                                                                                          WithHTTPRoundTripper is deprecated. Please consider using WithHTTPClient instead. The function allows customizing the underlying HTTP transport for emitting spans.

                                                                                          func WithHostname

                                                                                          func WithHostname(name string) StartOption

                                                                                            WithHostname allows specifying the hostname with which to mark outgoing traces.

                                                                                            func WithLambdaMode

                                                                                            func WithLambdaMode(enabled bool) StartOption

                                                                                              WithLambdaMode enables lambda mode on the tracer, for use with AWS Lambda.

                                                                                              func WithLogger

                                                                                              func WithLogger(logger ddtrace.Logger) StartOption

                                                                                                WithLogger sets logger as the tracer's error printer.

                                                                                                func WithPrioritySampling

                                                                                                func WithPrioritySampling() StartOption

                                                                                                  WithPrioritySampling is deprecated, and priority sampling is enabled by default. When using distributed tracing, the priority sampling value is propagated in order to get all the parts of a distributed trace sampled. To learn more about priority sampling, please visit:

                                                                                                  func WithPropagator

                                                                                                  func WithPropagator(p Propagator) StartOption

                                                                                                    WithPropagator sets an alternative propagator to be used by the tracer.

                                                                                                    func WithRuntimeMetrics

                                                                                                    func WithRuntimeMetrics() StartOption

                                                                                                      WithRuntimeMetrics enables automatic collection of runtime metrics every 10 seconds.

                                                                                                      func WithSampler

                                                                                                      func WithSampler(s Sampler) StartOption

                                                                                                        WithSampler sets the given sampler to be used with the tracer. By default an all-permissive sampler is used.

                                                                                                        func WithSamplingRules

                                                                                                        func WithSamplingRules(rules []SamplingRule) StartOption

                                                                                                          WithSamplingRules specifies the sampling rates to apply to spans based on the provided rules.

                                                                                                          func WithService

                                                                                                          func WithService(name string) StartOption

                                                                                                            WithService sets the default service name for the program.

                                                                                                            func WithServiceName

                                                                                                            func WithServiceName(name string) StartOption

                                                                                                              WithServiceName is deprecated. Please use WithService. If you are using an older version and you are upgrading from WithServiceName to WithService, please note that WithService will determine the service name of server and framework integrations.

                                                                                                              func WithServiceVersion

                                                                                                              func WithServiceVersion(version string) StartOption

                                                                                                                WithServiceVersion specifies the version of the service that is running. This will be included in spans from this service in the "version" tag.

                                                                                                                func WithUDS

                                                                                                                func WithUDS(socketPath string) StartOption

                                                                                                                  WithUDS configures the HTTP client to dial the Datadog Agent via the specified Unix Domain Socket path.

                                                                                                                  type StartSpanOption

                                                                                                                  type StartSpanOption = ddtrace.StartSpanOption

                                                                                                                    StartSpanOption is a configuration option for StartSpan. It is aliased in order to help godoc group all the functions returning it together. It is considered more correct to refer to it as the type as the origin, ddtrace.StartSpanOption.

                                                                                                                    func AnalyticsRate

                                                                                                                    func AnalyticsRate(rate float64) StartSpanOption

                                                                                                                      AnalyticsRate sets a custom analytics rate for a span. It decides the percentage of events that will be picked up by the App Analytics product. It's represents a float64 between 0 and 1 where 0.5 would represent 50% of events.

                                                                                                                      func ChildOf

                                                                                                                      func ChildOf(ctx ddtrace.SpanContext) StartSpanOption

                                                                                                                        ChildOf tells StartSpan to use the given span context as a parent for the created span.

                                                                                                                        func Measured

                                                                                                                        func Measured() StartSpanOption

                                                                                                                          Measured marks this span to be measured for metrics and stats calculations.

                                                                                                                          func ResourceName

                                                                                                                          func ResourceName(name string) StartSpanOption

                                                                                                                            ResourceName sets the given resource name on the started span. A resource could be an SQL query, a URL, an RPC method or something else.

                                                                                                                            func ServiceName

                                                                                                                            func ServiceName(name string) StartSpanOption

                                                                                                                              ServiceName sets the given service name on the started span. For example "http.server".

                                                                                                                              func SpanType

                                                                                                                              func SpanType(name string) StartSpanOption

                                                                                                                                SpanType sets the given span type on the started span. Some examples in the case of the Datadog APM product could be "web", "db" or "cache".

                                                                                                                                func StartTime

                                                                                                                                func StartTime(t time.Time) StartSpanOption

                                                                                                                                  StartTime sets a custom time as the start time for the created span. By default a span is started using the creation time.

                                                                                                                                  func Tag

                                                                                                                                  func Tag(k string, v interface{}) StartSpanOption

                                                                                                                                    Tag sets the given key/value pair as a tag on the started Span.

                                                                                                                                    func WithSpanID

                                                                                                                                    func WithSpanID(id uint64) StartSpanOption

                                                                                                                                      WithSpanID sets the SpanID on the started span, instead of using a random number. If there is no parent Span (eg from ChildOf), then the TraceID will also be set to the value given here.

                                                                                                                                      type TextMapCarrier

                                                                                                                                      type TextMapCarrier map[string]string

                                                                                                                                        TextMapCarrier allows the use of a regular map[string]string as both TextMapWriter and TextMapReader, making it compatible with the provided Propagator.

                                                                                                                                        func (TextMapCarrier) ForeachKey

                                                                                                                                        func (c TextMapCarrier) ForeachKey(handler func(key, val string) error) error

                                                                                                                                          ForeachKey conforms to the TextMapReader interface.

                                                                                                                                          func (TextMapCarrier) Set

                                                                                                                                          func (c TextMapCarrier) Set(key, val string)

                                                                                                                                            Set implements TextMapWriter.

                                                                                                                                            type TextMapReader

                                                                                                                                            type TextMapReader interface {
                                                                                                                                            	// ForeachKey iterates over all keys that exist in the underlying
                                                                                                                                            	// carrier. It takes a callback function which will be called
                                                                                                                                            	// using all key/value pairs as arguments. ForeachKey will return
                                                                                                                                            	// the first error returned by the handler.
                                                                                                                                            	ForeachKey(handler func(key, val string) error) error

                                                                                                                                              TextMapReader allows iterating over sets of key/value pairs. Carriers implementing TextMapReader are compatible to be used with Datadog's TextMapPropagator.

                                                                                                                                              type TextMapWriter

                                                                                                                                              type TextMapWriter interface {
                                                                                                                                              	// Set sets the given key/value pair.
                                                                                                                                              	Set(key, val string)

                                                                                                                                                TextMapWriter allows setting key/value pairs of strings on the underlying data structure. Carriers implementing TextMapWriter are compatible to be used with Datadog's TextMapPropagator.