Documentation

Overview

Package instrument implements functions to make instrumenting code, including metrics and logging, easier.

Index

Constants

const (
	// InvariantViolatedMetricName is the name of the metric emitted upon
	// invocation of `EmitInvariantViolation`.
	InvariantViolatedMetricName = "invariant_violated"

	// InvariantViolatedLogFieldName is the name of the log field to be
	// used when generating errors/log statements pertaining to the violation
	// of an invariant.
	InvariantViolatedLogFieldName = "violation"

	// InvariantViolatedLogFieldValue is the value of the log field to be
	// used when generating errors/log statements pertaining to the violation
	// of an invariant.
	InvariantViolatedLogFieldValue = InvariantViolatedMetricName

	// ShouldPanicEnvironmentVariableName is the name of the environment variable
	// that must be set to "true" in order for the invariant violated functions
	// to panic after logging / emitting metrics. Should only be set in test
	// environments.
	ShouldPanicEnvironmentVariableName = "PANIC_ON_INVARIANT_VIOLATED"
)

Variables

var (
	// Revision is the VCS revision associated with this build. Overridden using ldflags
	// at compile time. Example:
	// $ go build -ldflags "-X github.com/m3db/m3/src/x/instrument.Revision=abcdef" ...
	// Adapted from: https://www.atatus.com/blog/golang-auto-build-versioning/
	Revision = "unknown"

	// Branch is the VCS branch associated with this build.
	Branch = "unknown"

	// Version is the version associated with this build.
	Version = "unknown"

	// BuildDate is the date this build was created.
	BuildDate = "unknown"

	// BuildTimeUnix is the seconds since epoch representing the date this build was created.
	BuildTimeUnix = "0"

	// LogBuildInfoAtStartup controls whether we log build information at startup. If its
	// set to a non-empty string, we log the build information at process startup.
	LogBuildInfoAtStartup string
)

Functions

func DefaultHistogramTimerHistogramBuckets

func DefaultHistogramTimerHistogramBuckets() tally.Buckets

DefaultHistogramTimerHistogramBuckets returns a set of default histogram timer histogram buckets, from 2ms up to 1hr.

func DefaultSummaryQuantileObjectives

func DefaultSummaryQuantileObjectives() map[float64]float64

DefaultSummaryQuantileObjectives is a set of default summary quantile objectives and allowed error thresholds.

func EmitAndLogInvariantViolation

func EmitAndLogInvariantViolation(opts Options, f func(l *zap.Logger))

EmitAndLogInvariantViolation calls EmitInvariantViolation and then calls the provided function with a supplied logger that is pre-configured with an invariant violated field. Optionally panics if the ShouldPanicEnvironmentVariableName is set to "true".

func EmitInvariantViolation

func EmitInvariantViolation(opts Options)

EmitInvariantViolation emits a metric to indicate a system invariant has been violated. Users of this method are expected to monitor/alert off this metric to ensure they're notified when such an event occurs. Further, they should log further information to aid diagnostics of the system invariant violated at the callsite of the violation. Optionally panics if the ShouldPanicEnvironmentVariableName is set to "true".

func InvariantErrorf

func InvariantErrorf(format string, a ...interface{}) error

InvariantErrorf constructs a new error, prefixed with a string indicating that an invariant violation occurred. Optionally panics if the ShouldPanicEnvironmentVariableName is set to "true".

func LogBuildInfo

func LogBuildInfo()

LogBuildInfo logs the build information to the provided logger.

func MustCreateSampledTimer

func MustCreateSampledTimer(base tally.Timer, rate float64) tally.Timer

MustCreateSampledTimer creates a new sampled timer, and panics if an error is encountered.

func NewPrometheusProcessCollector

func NewPrometheusProcessCollector(opts ProcessCollectorOpts) prometheus.Collector

NewPrometheusProcessCollector returns a collector which exports the current state of process metrics including CPU, memory and file descriptor usage as well as the process start time. The detailed behavior is defined by the provided ProcessCollectorOpts. The zero value of ProcessCollectorOpts creates a collector for the current process with an empty namespace string and no error reporting.

Currently, the collector depends on a Linux-style proc filesystem and therefore only exports metrics for Linux.

NB(r): This version of the Prometheus process collector allows skipping emitting open FDs due to excessive load reporting open FDs with processes with a large number of open FDs.

func NewSampledTimer

func NewSampledTimer(base tally.Timer, rate float64) (tally.Timer, error)

NewSampledTimer creates a new sampled timer.

func NewTestDebugLogger

func NewTestDebugLogger(t *testing.T) *zap.Logger

NewTestDebugLogger returns a new debug logger for tests.

func NewTimer

func NewTimer(scope tally.Scope, name string, opts TimerOptions) tally.Timer

NewTimer returns a new timer that is backed by a timer or a histogram based on the timer options.

Types

type BatchMethodMetrics

type BatchMethodMetrics struct {
	RetryableErrors    tally.Counter
	NonRetryableErrors tally.Counter
	Errors             tally.Counter
	Success            tally.Counter
	Latency            tally.Timer
}

BatchMethodMetrics is a bundle of common metrics for methods with batch semantics.

func NewBatchMethodMetrics

func NewBatchMethodMetrics(
	scope tally.Scope,
	methodName string,
	opts TimerOptions,
) BatchMethodMetrics

NewBatchMethodMetrics creates new batch method metrics.

func (*BatchMethodMetrics) ReportLatency

func (m *BatchMethodMetrics) ReportLatency(d time.Duration)

ReportLatency reports latency.

func (*BatchMethodMetrics) ReportNonRetryableErrors

func (m *BatchMethodMetrics) ReportNonRetryableErrors(n int)

ReportNonRetryableErrors reports non-retryable errors.

func (*BatchMethodMetrics) ReportRetryableErrors

func (m *BatchMethodMetrics) ReportRetryableErrors(n int)

ReportRetryableErrors reports retryable errors.

func (*BatchMethodMetrics) ReportSuccess

func (m *BatchMethodMetrics) ReportSuccess(n int)

ReportSuccess reports successess.

type ExtendedMetricsType

type ExtendedMetricsType int

ExtendedMetricsType is a type of extended metrics to report.

const (
	// NoExtendedMetrics describes no extended metrics.
	NoExtendedMetrics ExtendedMetricsType = iota

	// SimpleExtendedMetrics describes just a simple level of extended metrics:
	// - number of active goroutines
	// - number of configured gomaxprocs
	SimpleExtendedMetrics

	// ModerateExtendedMetrics describes a moderately verbose level of extended metrics:
	// - number of active goroutines
	// - number of configured gomaxprocs
	// - number of file descriptors
	ModerateExtendedMetrics

	// DetailedExtendedMetrics describes a detailed level of extended metrics:
	// - number of active goroutines
	// - number of configured gomaxprocs
	// - number of file descriptors
	// - memory allocated running count
	// - memory used by heap
	// - memory used by heap that is idle
	// - memory used by heap that is in use
	// - memory used by stack
	// - number of garbage collections
	// - GC pause times
	DetailedExtendedMetrics

	// DefaultExtendedMetricsType is the default extended metrics level.
	DefaultExtendedMetricsType = SimpleExtendedMetrics
)

func (ExtendedMetricsType) String

func (t ExtendedMetricsType) String() string

func (*ExtendedMetricsType) UnmarshalYAML

func (t *ExtendedMetricsType) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals an ExtendedMetricsType into a valid type from string.

type HistogramObjective

type HistogramObjective struct {
	Upper float64 `yaml:"upper"`
}

HistogramObjective is a Prometheus histogram bucket. See: https://godoc.org/github.com/prometheus/client_golang/prometheus#HistogramOpts

type HistogramTimerOptions

type HistogramTimerOptions struct {
	HistogramBuckets tally.Buckets
}

HistogramTimerOptions is a set of histogram timer options.

type MethodMetrics

type MethodMetrics struct {
	Errors         tally.Counter
	Success        tally.Counter
	ErrorsLatency  tally.Timer
	SuccessLatency tally.Timer
}

MethodMetrics is a bundle of common metrics with a uniform naming scheme.

func NewMethodMetrics

func NewMethodMetrics(scope tally.Scope, methodName string, opts TimerOptions) MethodMetrics

NewMethodMetrics returns a new Method metrics for the given method name.

func (*MethodMetrics) ReportError

func (m *MethodMetrics) ReportError(d time.Duration)

ReportError reports an error.

func (*MethodMetrics) ReportSuccess

func (m *MethodMetrics) ReportSuccess(d time.Duration)

ReportSuccess reports a success.

func (*MethodMetrics) ReportSuccessOrError

func (m *MethodMetrics) ReportSuccessOrError(e error, d time.Duration)

ReportSuccessOrError increments Error/Success counter dependending on the error.

type MetricSanitizationType

type MetricSanitizationType int

MetricSanitizationType is a type of sanitizer to use for metrics.

const (
	// NoMetricSanitization performs no metric sanitization.
	NoMetricSanitization MetricSanitizationType = iota

	// M3MetricSanitization performs M3 metric sanitization.
	M3MetricSanitization

	// PrometheusMetricSanitization performs Prometheus metric sanitization.
	PrometheusMetricSanitization
)

func (*MetricSanitizationType) NewOptions

func (t *MetricSanitizationType) NewOptions() *tally.SanitizeOptions

NewOptions returns a new set of sanitization options for the sanitization type.

func (MetricSanitizationType) String

func (t MetricSanitizationType) String() string

func (*MetricSanitizationType) UnmarshalYAML

func (t *MetricSanitizationType) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals a MetricSanitizationType into a valid type from string.

type MetricsConfiguration

type MetricsConfiguration struct {
	// Root scope configuration.
	RootScope *ScopeConfiguration `yaml:"scope"`

	// M3 reporter configuration.
	M3Reporter *m3.Configuration `yaml:"m3"`

	// Prometheus reporter configuration.
	PrometheusReporter *PrometheusConfiguration `yaml:"prometheus"`

	// Metrics sampling rate.
	SamplingRate float64 `yaml:"samplingRate" validate:"nonzero,min=0.0,max=1.0"`

	// Extended metrics type.
	ExtendedMetrics *ExtendedMetricsType `yaml:"extended"`

	// Metric sanitization type.
	Sanitization *MetricSanitizationType `yaml:"sanitization"`
}

MetricsConfiguration configures options for emitting metrics.

func (*MetricsConfiguration) NewRootScope

func (mc *MetricsConfiguration) NewRootScope() (tally.Scope, io.Closer, error)

NewRootScope creates a new tally.Scope based on a tally.CachedStatsReporter based on the the the config.

func (*MetricsConfiguration) NewRootScopeAndReporters

func (mc *MetricsConfiguration) NewRootScopeAndReporters(
	opts NewRootScopeAndReportersOptions,
) (
	tally.Scope,
	io.Closer,
	MetricsConfigurationReporters,
	error,
)

NewRootScopeAndReporters creates a new tally.Scope based on a tally.CachedStatsReporter based on the the the config along with the reporters used.

func (*MetricsConfiguration) NewRootScopeReporter

func (mc *MetricsConfiguration) NewRootScopeReporter(
	r tally.CachedStatsReporter,
) (tally.Scope, io.Closer)

NewRootScopeReporter creates a new tally.Scope based on a given tally.CachedStatsReporter and given root scope config. In most cases NewRootScope should be used, but for cases such as hooking into the reporter to manually flush it.

func (*MetricsConfiguration) ReportInterval

func (mc *MetricsConfiguration) ReportInterval() time.Duration

ReportInterval returns the metrics reporting interval.

func (*MetricsConfiguration) SampleRate

func (mc *MetricsConfiguration) SampleRate() float64

SampleRate returns the metrics sampling rate.

type MetricsConfigurationM3Reporter

type MetricsConfigurationM3Reporter struct {
	Reporter m3.Reporter
}

MetricsConfigurationM3Reporter is the M3 reporter if constructed.

type MetricsConfigurationPrometheusReporter

type MetricsConfigurationPrometheusReporter struct {
	Reporter prometheus.Reporter
	Registry *prom.Registry
}

MetricsConfigurationPrometheusReporter is the Prometheus reporter if constructed.

type MetricsConfigurationReporters

type MetricsConfigurationReporters struct {
	AllReporters       []tally.CachedStatsReporter
	M3Reporter         *MetricsConfigurationM3Reporter
	PrometheusReporter *MetricsConfigurationPrometheusReporter
}

MetricsConfigurationReporters is the reporters constructed.

type NewRootScopeAndReportersOptions

type NewRootScopeAndReportersOptions struct {
	PrometheusHandlerListener    net.Listener
	PrometheusExternalRegistries []PrometheusExternalRegistry
	PrometheusOnError            func(e error)
}

NewRootScopeAndReportersOptions is a set of options.

type Options

type Options interface {
	// SetLogger sets the zap logger
	SetLogger(value *zap.Logger) Options

	// ZapLogger returns the zap logger
	Logger() *zap.Logger

	// SetMetricsScope sets the metrics scope.
	SetMetricsScope(value tally.Scope) Options

	// MetricsScope returns the metrics scope.
	MetricsScope() tally.Scope

	// Tracer returns the tracer.
	Tracer() opentracing.Tracer

	// SetTracer sets the tracer.
	SetTracer(tracer opentracing.Tracer) Options

	// SetTimerOptions sets the metrics timer options to used
	// when building timers from timer options.
	SetTimerOptions(value TimerOptions) Options

	// TimerOptions returns the metrics timer options to used
	// when building timers from timer options.
	TimerOptions() TimerOptions

	// SetReportInterval sets the time between reporting metrics within the system.
	SetReportInterval(time.Duration) Options

	// ReportInterval returns the time between reporting metrics within the system.
	ReportInterval() time.Duration

	// SetCustomBuildTags sets custom tags to be added to build report metrics in
	// addition to the defaults.
	SetCustomBuildTags(tags map[string]string) Options

	// CustomBuildTags returns the custom build tags.
	CustomBuildTags() map[string]string
}

Options represents the options for instrumentation.

func NewOptions

func NewOptions() Options

NewOptions creates new instrument options.

func NewTestOptions

func NewTestOptions(t *testing.T) Options

NewTestOptions returns a set of instrument options useful for tests. This includes: - Logger built using development config with debug level set.

type ProcessCollectorOpts

type ProcessCollectorOpts struct {
	// PidFn returns the PID of the process the collector collects metrics
	// for. It is called upon each collection. By default, the PID of the
	// current process is used, as determined on construction time by
	// calling os.Getpid().
	PidFn func() (int, error)
	// If non-empty, each of the collected metrics is prefixed by the
	// provided string and an underscore ("_").
	Namespace string
	// DisableOpenFDs allows disabling the reporting of open FDs due to
	// the cost that is required to report the number of file descriptors.
	DisableOpenFDs bool
	// If true, any error encountered during collection is reported as an
	// invalid metric (see NewInvalidMetric). Otherwise, errors are ignored
	// and the collected metrics will be incomplete. (Possibly, no metrics
	// will be collected at all.) While that's usually not desired, it is
	// appropriate for the common "mix-in" of process metrics, where process
	// metrics are nice to have, but failing to collect them should not
	// disrupt the collection of the remaining metrics.
	ReportErrors bool
}

ProcessCollectorOpts defines the behavior of a process metrics collector created with NewProcessCollector.

type PrometheusConfiguration

type PrometheusConfiguration struct {
	// HandlerPath if specified will be used instead of using the default
	// HTTP handler path "/metrics".
	HandlerPath string `yaml:"handlerPath"`

	// ListenAddress if specified will be used instead of just registering the
	// handler on the default HTTP serve mux without listening.
	ListenAddress string `yaml:"listenAddress"`

	// TimerType is the default Prometheus type to use for Tally timers.
	TimerType string `yaml:"timerType"`

	// DefaultHistogramBuckets if specified will set the default histogram
	// buckets to be used by the reporter.
	DefaultHistogramBuckets []prometheus.HistogramObjective `yaml:"defaultHistogramBuckets"`

	// DefaultSummaryObjectives if specified will set the default summary
	// objectives to be used by the reporter.
	DefaultSummaryObjectives []prometheus.SummaryObjective `yaml:"defaultSummaryObjectives"`

	// OnError specifies what to do when an error either with listening
	// on the specified listen address or registering a metric with the
	// Prometheus. By default the registerer will panic.
	OnError string `yaml:"onError"`
}

PrometheusConfiguration is a configuration for a Prometheus reporter.

func (PrometheusConfiguration) NewReporter

func (c PrometheusConfiguration) NewReporter(
	configOpts PrometheusConfigurationOptions,
) (prometheus.Reporter, error)

NewReporter creates a new M3 Prometheus reporter from this configuration.

type PrometheusConfigurationOptions

type PrometheusConfigurationOptions struct {
	// Registry if not nil will specify the specific registry to use
	// for registering metrics.
	Registry *prom.Registry
	// ExternalRegistries if set (with combination of a specified Registry)
	// will also
	ExternalRegistries []PrometheusExternalRegistry
	// HandlerListener is the listener to register the server handler on.
	HandlerListener net.Listener
	// HandlerOpts is the reporter HTTP handler options, not specifying will
	// use defaults.
	HandlerOpts promhttp.HandlerOpts
	// OnError allows for customization of what to do when a metric
	// registration error fails, the default is to panic.
	OnError func(e error)
}

PrometheusConfigurationOptions allows some programatic options, such as using a specific registry and what error callback to register.

type PrometheusExternalRegistry

type PrometheusExternalRegistry struct {
	// Registry is the external prometheus registry to list.
	Registry *extprom.Registry
	// SubScope will add a prefix to all metric names exported by
	// this registry.
	SubScope string
}

PrometheusExternalRegistry is an external Prometheus registry to also expose as part of the handler.

type Reporter

type Reporter interface {
	// Start starts the reporter.
	Start() error
	// Stop stops the reporter.
	Stop() error
}

Reporter reports metrics about a component.

func NewBuildReporter

func NewBuildReporter(
	opts Options,
) Reporter

NewBuildReporter returns a new build version reporter.

func NewExtendedMetricsReporter

func NewExtendedMetricsReporter(
	scope tally.Scope,
	reportInterval time.Duration,
	metricsType ExtendedMetricsType,
) Reporter

NewExtendedMetricsReporter creates a new extended metrics reporter that reports runtime and process metrics.

func NewProcessReporter

func NewProcessReporter(
	scope tally.Scope,
	reportInterval time.Duration,
) Reporter

NewProcessReporter returns a new reporter that reports process metrics, currently just the process file descriptor count.

func StartReportingExtendedMetrics

func StartReportingExtendedMetrics(
	scope tally.Scope,
	reportInterval time.Duration,
	metricsType ExtendedMetricsType,
) (Reporter, error)

StartReportingExtendedMetrics creates a extend metrics reporter and starts the reporter returning it so it may be stopped if successfully started.

type ScopeConfiguration

type ScopeConfiguration struct {
	// Prefix of metrics in this scope.
	Prefix string `yaml:"prefix"`

	// Metrics reporting interval.
	ReportingInterval time.Duration `yaml:"reportingInterval"`

	// Common tags shared by metrics reported.
	CommonTags map[string]string `yaml:"tags"`
}

ScopeConfiguration configures a metric scope.

type StringListEmitter

type StringListEmitter struct {
	sync.Mutex
	// contains filtered or unexported fields
}

StringListEmitter emits a gauges indicating the order of a list of strings.

func NewStringListEmitter

func NewStringListEmitter(scope tally.Scope, name string) *StringListEmitter

NewStringListEmitter returns a StringListEmitter.

func (*StringListEmitter) Close

func (sle *StringListEmitter) Close() error

Close stops emitting the gauge.

func (*StringListEmitter) Start

func (sle *StringListEmitter) Start(sl []string) error

Start starts a goroutine that continuously emits the value of the gauges

func (*StringListEmitter) UpdateStringList

func (sle *StringListEmitter) UpdateStringList(sl []string) error

UpdateStringList updates the gauges according to the passed list of strings. It will first set the old gauges to 0, then emit new metrics with different values for the "type" label.

type SummaryObjective

type SummaryObjective struct {
	Percentile   float64 `yaml:"percentile"`
	AllowedError float64 `yaml:"allowedError"`
}

SummaryObjective is a Prometheus summary objective. See: https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts

type TimerOptions

type TimerOptions struct {
	Type               TimerType
	StandardSampleRate float64
	HistogramBuckets   tally.Buckets
}

TimerOptions is a set of timer options when creating a timer.

func NewHistogramTimerOptions

func NewHistogramTimerOptions(opts HistogramTimerOptions) TimerOptions

NewHistogramTimerOptions returns a set of histogram timer options and if no histogram buckets are set it will use the default histogram buckets defined.

func NewStandardTimerOptions

func NewStandardTimerOptions() TimerOptions

NewStandardTimerOptions returns a set of standard timer options for standard timer types.

func (TimerOptions) NewTimer

func (o TimerOptions) NewTimer(scope tally.Scope, name string) tally.Timer

NewTimer creates a new timer based on the timer options.

type TimerType

type TimerType uint

TimerType is a type of timer, standard or histogram timer.

const (
	// StandardTimerType is a standard timer type back by a regular timer.
	StandardTimerType TimerType = iota
	// HistogramTimerType is a histogram timer backed by a histogram.
	HistogramTimerType
)