otelpyroscope

README

Profiling Instrumentation

NOTE: This is an experimental package -- and will be officially supported in future versions of Pyroscope

The package provides means to integrate tracing with profiling. More specifically, a TracerProvider implementation, that annotates profiling data with span IDs: when a new trace span emerges, the tracer adds a profile_id pprof tag that points to the span. This makes it possible to filter out a profile of a particular trace span in Pyroscope.

Jaeger Example

You can find a full Jaeger example (with custom Jaeger UI) in the tracing/jaeger folder in the Pyroscope repository.

Baseline Diffs

We also added functionality where each individual span is compared to a baseline of spans with similar properties and the diff can be shown in the UI:

Grafana Example

For another example of what this package allows you to do you can see with Grafana the ability to link between logs, traces and profiles in the following video (source):

Other Notes

Note that the module does not control pprof profiler itself – it still needs to be started for profiles to be collected. This can be done either via runtime/pprof package, or using the Pyroscope client.

By default, only the root span gets annotated (the first span created locally), this is done to circumvent the fact that the profiler records only the time spent on CPU. Otherwise, all the children profiles should be merged to get the full representation of the root span profile.

There are few limitations:

• Only Go CPU profiling is fully supported at the moment.
• Due to the very idea of the sampling profilers, spans shorter than the sample interval may not be captured. For example, Go CPU profiler probes stack traces 100 times per second, meaning that spans shorter than 10ms may not be captured.

Documentation

Index

• func NewTracerProvider(tp trace.TracerProvider, options ...Option) trace.TracerProvider
• type Config
• type Option
  • func WithAddSpanName(x bool) Option
  • func WithAppName(app string) Option
  • func WithDefaultProfileURLBuilder(_, _ string) Option
  • func WithProfileBaselineLabels(x map[string]string) Option
  • func WithProfileBaselineURL(x bool) Option
  • func WithProfileURL(x bool) Option
  • func WithProfileURLBuilder(b func(_ string) string) Option
  • func WithPyroscopeURL(addr string) Option
  • func WithRootSpanOnly(x bool) Option

Functions

func NewTracerProvider

func NewTracerProvider(tp trace.TracerProvider, options ...Option) trace.TracerProvider

NewTracerProvider creates a new tracer provider that annotates pprof profiles with span ID tag. This allows to establish a relationship between pprof profiles and reported tracing spans.

Types

type Config

type Config struct {
	AppName                   string
	PyroscopeURL              string
	IncludeProfileURL         bool
	IncludeProfileBaselineURL bool
	ProfileBaselineLabels     map[string]string

	RootOnly    bool
	AddSpanName bool
}

type Option

type Option func(*tracerProvider)

func WithAddSpanName

func WithAddSpanName(x bool) Option

WithAddSpanName specifies whether the current span name should be added to the profile labels. N.B if the name is dynamic, or too many values are supposed, this may significantly deteriorate performance. By default, span name is not added to profile labels.

func WithAppName

func WithAppName(app string) Option

WithAppName specifies the profiled application name. It should match the name specified in pyroscope configuration. Required, if profile URL or profile baseline URL is enabled.

func WithDefaultProfileURLBuilder

func WithDefaultProfileURLBuilder(_, _ string) Option

WithDefaultProfileURLBuilder specifies the default profile URL builder. DEPRECATED: use WithProfileURL

func WithProfileBaselineLabels

func WithProfileBaselineLabels(x map[string]string) Option

WithProfileBaselineLabels provides a map of extra labels to be added to the baseline query alongside with pprof labels set in runtime. Typically, it should match the labels specified in the Pyroscope profiler config. Note that the map must not be modified.

func WithProfileBaselineURL

func WithProfileBaselineURL(x bool) Option

WithProfileBaselineURL specifies whether to add the pyroscope.profile.baseline.url attribute with the URL to the baseline profile. See WithProfileBaselineLabels.

func WithProfileURL

func WithProfileURL(x bool) Option

WithProfileURL specifies whether to add the pyroscope.profile.url attribute with the URL to the span profile.

func WithProfileURLBuilder

func WithProfileURLBuilder(b func(_ string) string) Option

WithProfileURLBuilder specifies how profile URL is to be built. DEPRECATED: use WithProfileURL

func WithPyroscopeURL

func WithPyroscopeURL(addr string) Option

WithPyroscopeURL provides a base URL for the profile and baseline URLs. Required, if profile URL or profile baseline URL is enabled.

func WithRootSpanOnly

func WithRootSpanOnly(x bool) Option

WithRootSpanOnly indicates that only the root span is to be profiled. The profile includes samples captured during child span execution but the spans won't have their own profiles and won't be annotated with pyroscope.profile attributes. The option is enabled by default.