trace2receiver

package module
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 28, 2024 License: MIT Imports: 33 Imported by: 0

README

Trace2 Receiver

The trace2receiver project is a trace receiver component library for an OpenTelemetry Custom Collector daemon. It receives Git Trace2 telemetry from local Git commands, translates it into an OpenTelemetry format, and forwards it to other OpenTelemetry components.

This component is useful it you want to collect performance data for Git commands, aggregate data from multiple users to create performance dashboards, build distributed traces of nested Git commands, or understand how the size and shape of your Git repositories affect command performance.

Background

This project is a GOLANG static library component that must be linked into an OpenTelemetry Custom Collector along with other pipeline and exporter components to process and forward the telemetry data to a data store, such as Azure Monitor or another OTLP aware cloud provider.

Setup and configuration details are provided in the Docs.

The sample-trace2-otel-collector peer repository contains a pre-built open source sample collector to help you get started. See the README for more details.

Contributions

This project is under active development, and loves contributions from the community. Check out the CONTRIBUTING guide for details on getting started.

Requirements

This project is written in GOLANG and uses OpenTelemetry libraries and tools. See the OpenTelemetry documentation for more information.

This project runs on Linux, macOS, and Windows.

License

This project is licensed under the terms of the MIT open source license. Please refer to LICENSE for the full terms.

Maintainers

See CODEOWNERS for a list of current project maintainers.

Support

See SUPPORT for instructions on how to file bugs, make feature requests, or seek help.

Documentation

Index

Constants

View Source 
const (
	DetailLevelDropName    string = "dl:drop"
	DetailLevelSummaryName string = "dl:summary"
	DetailLevelProcessName string = "dl:process"
	DetailLevelVerboseName string = "dl:verbose"

	DetailLevelDefaultName string = DetailLevelSummaryName
)

All detail level names have leading "dl:" to help avoid cycles when resolving a custom ruleset name.

View Source 
const (
	// Value of the `service.namespace` key that we inject into
	// all resourceAttributes.
	Trace2ServiceNamespace = "trace2"

	// Value of the `instrumentation.name` or `instrumentationlibrary.name`
	// key that we inject into the resourceAttributes.  (The actual spelling
	// of this key varies it seems.)
	Trace2InstrumentationName = "trace2receiver"
)
View Source 
const (
	// The Trace2 SID of the process.  This is the complete SID with
	// zero or more slashes describing the SIDs of any Trace2-aware
	// parent processes.
	Trace2CmdSid = attribute.Key("trace2.cmd.sid")

	// The complete command line args of the process.
	Trace2CmdArgv = attribute.Key("trace2.cmd.argv")

	// The version string of the process executable as reported in the
	// Trace2 "version" event.
	Trace2CmdVersion = attribute.Key("trace2.cmd.version")

	// The command's exit code.  Zero if it completed without error.
	// If this process was signalled, this should be 128+signo.
	Trace2CmdExitCode = attribute.Key("trace2.cmd.exit_code")

	// The base filename of the process executable (with the pathname and
	// `.exe` suffix stripped off), for example `git` or `git-remote-https`.
	Trace2CmdName = attribute.Key("trace2.cmd.name")

	// The executable name and verb isolated from the command line
	// with normalized formatting.  For example `git checkout` should
	// be reported as `git:checkout`.  For commands that do not have
	// a verb, this should just be the name of the executable.
	Trace2CmdNameVerb = attribute.Key("trace2.cmd.name_verb")

	// The executable name, verb and command mode combined with
	// normalized formatting.  For example, `git checkout -- <pathname>`
	// is different from `git checkout <branchname>`.  These should be
	// reported as `git:checkout#path` or `git:checkout#branch`.  For
	// commands that do not have a mode, this should just be the verb.
	Trace2CmdNameVerbMode = attribute.Key("trace2.cmd.name_verb_mode")

	// The verb hierarchy for the command as reported by Git itself.
	// For example when `git index-pack` is launched by `git fetch`,
	// the child process will report a verb of `index-pack` and a
	// hierarchy of `fetch/index-pack`.
	Trace2CmdHierarchy = attribute.Key("trace2.cmd.hierarchy")

	// The format string of one error message from the command.
	Trace2CmdErrFmt = attribute.Key("trace2.cmd.error.format")
	Trace2CmdErrMsg = attribute.Key("trace2.cmd.error.message")

	Trace2CmdAliasKey   = attribute.Key("trace2.cmd.alias.key")
	Trace2CmdAliasValue = attribute.Key("trace2.cmd.alias.value")

	// Optional process hierarchy that invoked this Git command.
	// Usually contains things like "bash" and "sshd".  This data
	// is read from "/proc" on Linux, for example.  It may be
	// truncated, but contain enough entries to give some crude
	// context.
	//
	// Type: array of string
	Trace2CmdAncestry = attribute.Key("trace2.cmd.ancestry")

	// Trace2 classification of the span.  For example: "process",
	// "thread", "child", or "region".
	//
	// Type: string
	Trace2SpanType = attribute.Key("trace2.span.type")

	Trace2ChildPid        = attribute.Key("trace2.child.pid")
	Trace2ChildExitCode   = attribute.Key("trace2.child.exitcode")
	Trace2ChildArgv       = attribute.Key("trace2.child.argv")
	Trace2ChildClass      = attribute.Key("trace2.child.class")
	Trace2ChildHookName   = attribute.Key("trace2.child.hook")
	Trace2ChildReadyState = attribute.Key("trace2.child.ready")

	Trace2RegionMessage = attribute.Key("trace2.region.message")
	Trace2RegionNesting = attribute.Key("trace2.region.nesting")
	Trace2RegionRepoId  = attribute.Key("trace2.region.repoid")
	Trace2RegionData    = attribute.Key("trace2.region.data")

	Trace2ExecExe      = attribute.Key("trace2.exec.exe")
	Trace2ExecArgv     = attribute.Key("trace2.exec.argv")
	Trace2ExecExitCode = attribute.Key("trace2.exec.exitcode")

	Trace2RepoSet  = attribute.Key("trace2.repo.set")
	Trace2ParamSet = attribute.Key("trace2.param.set")

	Trace2ProcessData     = attribute.Key("trace2.process.data")
	Trace2ProcessTimers   = attribute.Key("trace2.process.timers")
	Trace2ProcessCounters = attribute.Key("trace2.process.counters")

	Trace2ThreadTimers   = attribute.Key("trace2.thread.timers")
	Trace2ThreadCounters = attribute.Key("trace2.thread.counters")

	Trace2GoArch = attribute.Key("trace2.machine.arch")
	Trace2GoOS   = attribute.Key("trace2.machine.os")

	Trace2PiiHostname = attribute.Key("trace2.pii.hostname")
	Trace2PiiUsername = attribute.Key("trace2.pii.username")
)

Variables

View Source 
var CommandControlVerbPrefix []byte = []byte("cc: ")
View Source 
var Trace2ReceiverVersion string = "v0.0.0-unset"

Functions

func IsFSMonitorDaemon 

func IsFSMonitorDaemon(verb string) error

Is this Git command a `git fsmonitor--daemon` command?

Check in `apply__cmd_name()` since we know FSMonitor sends a valid `cmd_name` event. We really only need to reject `run` commands, but unfortunately, it does not send `cmd_mode` events, so we cannot distinguish between `run`, `start` and `stop`.

func NewFactory 

func NewFactory() receiver.Factory

NewFactory creates a factory for trace2 receiver.

func NewSocketInodeChangedError added in v0.5.0 

func NewSocketInodeChangedError(ie uint64, io uint64) error

func NewSocketPathnameStolenError added in v0.5.0 

func NewSocketPathnameStolenError(pathname string, err error) error

func NewTrace2Dataset 

func NewTrace2Dataset(rcvr_base *Rcvr_Base) *trace2Dataset

func WantChildSpans 

func WantChildSpans(dl FilterDetailLevel) bool

func WantMainThreadTimersAndCounters 

func WantMainThreadTimersAndCounters(dl FilterDetailLevel) bool

func WantProcessAliases 

func WantProcessAliases(dl FilterDetailLevel) bool

func WantProcessAncestry 

func WantProcessAncestry(dl FilterDetailLevel) bool

func WantProcessTimersCountersAndData 

func WantProcessTimersCountersAndData(dl FilterDetailLevel) bool

func WantRegionAndThreadSpans 

func WantRegionAndThreadSpans(dl FilterDetailLevel) bool

Types

type ApplyMap 

type ApplyMap map[string]FnApply

type Config 

type Config struct {
	// On Windows, this is a named pipe.  The canonical form is
	// (the backslash spelling of) `//./pipe/<pipename>`.
	//
	// `CreateNamedPipeW()` documents that named pipes can only be
	// created on the local NPFS and must use `.` rather than a
	// general UNC hostname.  (Subsequent clients can connect to
	// a remote pipe, but a server can only CREATE a local one.
	//
	// Therefore, we allow the pipename to be abbreviated in the
	// `config.yaml` as just `<pipename>` and assume the prefix.
	//
	// This config file field is ignored on non-Windows platforms.
	NamedPipePath string `mapstructure:"pipe"`

	// On Unix, this is a Unix domain socket.  This is an absolute
	// or relative pathname on the local file system.  To avoid
	// confusion with the existing Git Trace2 setup, we allow this
	// to be of the form `af_unix:[<mode>:]<pathname>` and strip
	// off the prefix.
	//
	// This config file field is ignored on Windows platforms.
	UnixSocketPath string `mapstructure:"socket"`

	// Allow command and control verbs to be embedded in the Trace2
	// data stream.
	AllowCommandControlVerbs bool `mapstructure:"enable_commands"`

	// Pathname to YML file containing PII settings.
	PiiSettingsPath string `mapstructure:"pii"`

	// Pathname to YML file containing our filter settings.
	FilterSettingsPath string `mapstructure:"filter"`
	// contains filtered or unexported fields
}

`Config` represents the complete configuration settings for an individual receiver declaration from the `config.yaml`.

These fields must be public (start with capital letter) so that the generic code in the collector can find them.

We have different types of OS-specific paths where we listen for Trace2 telemetry. We allow both types in a single config file, so that we can share it between clients; only the correct one for the platform will actually be used.

func (*Config) Validate 

func (cfg *Config) Validate() error

`Validate()` checks if the receiver configuration is valid.

This function is called once for each `trace2receiver[/<qualifier>]:` declaration (in the top-level `receivers:` section).

The file format and the customer collector framework allows more than one instance of a `trace2receiver` to be defined (presumably with different source types, pathnames, or verbosity) and run concurrently within this process. See: https://opentelemetry.io/docs/collector/configuration/

A receiver declaration does not imply that it will actually be instantiated (realized) in the factory. The receiver declaration causes a `cfg *Config` to be instantiated and that's it. (The instantiation in the factory is controlled by the `service.pipelines.traces.receivers:` array.)

type ExtractKeysMap 

type ExtractKeysMap map[string]FnExtractKeys

type FilterDefaults 

type FilterDefaults struct {

	// Ruleset defines the default ruleset or detail level to be
	// used when we receive data from a repo/worktree that does
	// not explicitly name one or does not have a nickname mapping.
	//
	// If not set, we default to the absolute default.
	RulesetName string `mapstructure:"ruleset"`
}

FilterDefaults defines default filtering values.

type FilterDetailLevel 

type FilterDetailLevel int

FilterDetailLevel describes the amount of detail in the output OTLP that we will generate for a Git command. 

const (
	DetailLevelUnset FilterDetailLevel = iota
	DetailLevelDrop
	DetailLevelSummary
	DetailLevelProcess
	DetailLevelVerbose
)

type FilterKeynames 

type FilterKeynames struct {

	// NicknameKey defines the Git config setting that can be used
	// to send an optional user-friendly id or nickname for a repo
	// or worktree.
	//
	// We can use the nickname to decide how to filter data
	// for the repo and to identify the repo in the cloud (and
	// possibly without exposing any PII or the actualy identity
	// of the repo/worktree).
	//
	// This can eliminate the need to rely on `remote.origin.url`
	// or the worktree root directory to identify (or guess at
	// the identity of) the repo.
	NicknameKey string `mapstructure:"nickname_key"`

	// RuleSetKey defines the Git config setting that can be used
	// to optionally send the name of the desired filter ruleset.
	// This value overrides any implied ruleset associated with
	// the RepoIdKey.
	RulesetKey string `mapstructure:"ruleset_key"`
}

FilterKeynames defines the names of the Git config settings that will be used in `def_param` events to send repository/worktree data to us. This lets a site have their own namespace for these keys. Some of these keys will also be sent to the cloud.

type FilterNicknames 

type FilterNicknames map[string]string

FilterNicknames is used to map a repo nickname to the name of the ruleset or detail-level that should be used.

This table is optional.

type FilterRulesets 

type FilterRulesets map[string]string

FilterRulesets is used to map a custom ruleset name to the pathname of the associated YML file. This form is used when parsing the filter settings YML file. We use this to create the real ruleset table (possibly with lazy loading).

type FilterSettings 

type FilterSettings struct {
	Keynames  FilterKeynames  `mapstructure:"keynames"`
	Nicknames FilterNicknames `mapstructure:"nicknames"`
	Rulesets  FilterRulesets  `mapstructure:"rulesets"`
	Defaults  FilterDefaults  `mapstructure:"defaults"`
	// contains filtered or unexported fields
}

FilterSettings describes how we should filter the OTLP output that we generate. It also describes the special keys that we look for in the Trace2 event stream to help us decide how to filter data for a particular command.

type FnApply 

type FnApply func(tr2 *trace2Dataset, evt *TrEvent) (err error)

type FnExtractKeys 

type FnExtractKeys func(evt *TrEvent, jm *jmap) (err error)

type MyYmlFileTypes 

type MyYmlFileTypes interface {
	RulesetDefinition | FilterSettings | PiiSettings
}

type MyYmlParseBufferFn 

type MyYmlParseBufferFn[T MyYmlFileTypes] func(data []byte, path string) (*T, error)

type PiiInclude 

type PiiInclude struct {
	// Lookup system hostname and add to process span.
	Hostname bool `mapstructure:"hostname"`

	// Lookup the client username and add to process span.
	Username bool `mapstructure:"username"`
}

type PiiSettings 

type PiiSettings struct {
	Include PiiInclude `mapstructure:"include"`
}

Settings to enable/disable possibly GDPR-sensitive fields in the telemetry output.

type QualifiedNames 

type QualifiedNames struct {
	// contains filtered or unexported fields
}

type Rcvr_Base 

type Rcvr_Base struct {
	// These fields should be set in ctor() in platform_*.go:createTraces()
	// when it is called from factory.go:NewFactory().
	Settings        receiver.CreateSettings
	Logger          *zap.Logger
	TracesConsumer  consumer.Traces
	MetricsConsumer consumer.Metrics
	LogsConsumer    consumer.Logs
	RcvrConfig      *Config
	// contains filtered or unexported fields
}

func (*Rcvr_Base) Start 

func (rcvr_base *Rcvr_Base) Start(unused_ctx context.Context, host component.Host) error

`Start()` handles base-class portions of receiver initialization.

type Rcvr_UnixSocket 

type Rcvr_UnixSocket struct {
	// These fields should be set in ctor()
	Base       *Rcvr_Base
	SocketPath string
	// contains filtered or unexported fields
}

`Rcvr_UnixSocket` implements the `component.TracesReceiver` (aka `component.Receiver` (aka `component.Component`)) interface.

func (*Rcvr_UnixSocket) Shutdown 

func (rcvr *Rcvr_UnixSocket) Shutdown(context.Context) error

Stop accepting new connections from Trace2 clients.

This is part of the `component.Component` interface.

func (*Rcvr_UnixSocket) Start 

func (rcvr *Rcvr_UnixSocket) Start(unused_ctx context.Context, host component.Host) error

Start receiving connections from Trace2 clients.

This is part of the `component.Component` interface.

type RejectClientError 

type RejectClientError struct {
	Err       error
	FSMonitor bool
}

func (*RejectClientError) Error 

func (rce *RejectClientError) Error() string

type RulesetCommands 

type RulesetCommands map[string]string

RulesetCommands is used to map a Git command to a detail level. This allows us to have a different verbosity for different commands. For example, verbose for `git status` and drop for `git config`.

We DO NOT support mapping to another ruleset because we want to avoid circular dependencies.

A command key should be in the format described in `trace2Dataset.setQualifiedExeVerbModeName()`.

The value must be one of [`DetailLevelDropName`, ... ].

type RulesetDefaults 

type RulesetDefaults struct {

	// The default detail level to use when exec+verb+mode
	// lookup fails.
	DetailLevelName string `mapstructure:"detail"`
}

RulesetDefaults defines default values for this custom ruleset.

type RulesetDefinition 

type RulesetDefinition struct {
	Commands RulesetCommands `mapstructure:"commands"`
	Defaults RulesetDefaults `mapstructure:"defaults"`
}

RulesetDefinition captures the content of a custom ruleset YML file.

type SocketInodeChangedError added in v0.5.0 

type SocketInodeChangedError struct {
	InodeExpected uint64
	InodeObserved uint64
}

func (*SocketInodeChangedError) Error added in v0.5.0 

func (e *SocketInodeChangedError) Error() string

type SocketPathnameStolenError added in v0.5.0 

type SocketPathnameStolenError struct {
	Pathname string
	SubErr   error
}

func (*SocketPathnameStolenError) Error added in v0.5.0 

func (e *SocketPathnameStolenError) Error() string

type TrChild 

type TrChild struct {
	// contains filtered or unexported fields
}

The `TrChild` structure captures the lifetime of a child process spawned by the current Git process. This is the "outer" time from the exec() to wait3() as observed by the invoking Git process.

This is independent of any telemetry that the child process itself may emit.

type TrEvent 

type TrEvent struct {
	// contains filtered or unexported fields
}

type TrEventAlias 

type TrEventAlias struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"alias" event

type TrEventAtExit 

type TrEventAtExit struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"exit" or "event":"atexit" event

type TrEventChildExit 

type TrEventChildExit struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"child_exit" event

type TrEventChildReady 

type TrEventChildReady struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"child_ready" event. Note that there is no exit-code, only a ready state hint.

type TrEventChildStart 

type TrEventChildStart struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"child_start" event

type TrEventCmdAncestry 

type TrEventCmdAncestry struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"ancestry" event

type TrEventCmdMode 

type TrEventCmdMode struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"cmd_mode" event

type TrEventCmdName 

type TrEventCmdName struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"cmd_name" event

type TrEventCmdPath 

type TrEventCmdPath struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"cmd_path" event

type TrEventCounter 

type TrEventCounter struct {
	// contains filtered or unexported fields
}

Event fields only present in "event":"counter" and "event":"th_counter" events

type TrEventDefParam 

type TrEventDefParam struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"def_param" event

type TrEventDefRepo 

type TrEventDefRepo struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"def_repo" event

type TrEventError 

type TrEventError struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"error" event

type TrEventExec 

type TrEventExec struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"exec" event

type TrEventExecResult 

type TrEventExecResult struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"exec_result" event

type TrEventGenericData 

type TrEventGenericData struct {
	// contains filtered or unexported fields
}

Event fields only present in a "data" or "data_json" event. The value type can be string or int64 for the former.

type TrEventRegionEnter 

type TrEventRegionEnter struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"region_enter" event

type TrEventRegionLeave 

type TrEventRegionLeave struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"region_leave" event

type TrEventSignal 

type TrEventSignal struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"signal" event

type TrEventStart 

type TrEventStart struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"start" event

type TrEventThreadExit 

type TrEventThreadExit struct {
}

Event fields only present in an "event":"thread_exit" event

type TrEventThreadStart 

type TrEventThreadStart struct {
}

Event fields only present in an "event":"thread_start" event

type TrEventTimer 

type TrEventTimer struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"timer" or "event":"th_timer" events

type TrEventVersion 

type TrEventVersion struct {
	// contains filtered or unexported fields
}

Event fields only present in an "event":"version" event

type TrExec 

type TrExec struct {
	// contains filtered or unexported fields
}

type TrProcess 

type TrProcess struct {
	// contains filtered or unexported fields
}

Data associated with the entire process.

type TrRegion 

type TrRegion struct {
	// contains filtered or unexported fields
}

type TrSpanEssentials 

type TrSpanEssentials struct {
	// contains filtered or unexported fields
}

`TrSpanEssentials` is a generic term to describe a chunk of time doing something. It may refer to the lifetime of the whole process, the lifetime of a thread, a Trace2 region, the lifetime of a child process, and etc.

type TrStopwatchTimer 

type TrStopwatchTimer struct {
	Intervals int64   `json:"intervals"`
	Total_sec float64 `json:"total_sec"`
	Min_sec   float64 `json:"min_sec"`
	Max_sec   float64 `json:"max_sec"`
}

type TrThread 

type TrThread struct {
	// contains filtered or unexported fields
}

The `TrThread` structure captures the lifetime of a thread.

Each thread (including the "main" thread) contains a `TrSpanEssentials` to document the life of the thread or process) and a "region stack" to capture in-progress Trace2 Regions as they are being reported by the client.

Yes, each thread needs its own region stack because regions are per-thread.

When we start a region-start event, we push a new frame on to the region stack. When we see the (hopefully, corresponding) region-leave event, we "complete" the region, pop it off of the region stack, and move it to the "completed regions" array for later reporting.

Source Files

View all Source files

Directories

Path Synopsis
internal
go-winio
go-winio/internal/fs
This package contains Win32 filesystem functionality.
 This package contains Win32 filesystem functionality.
go-winio/internal/stringbuffer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.