events

package
v0.0.0-...-6a70757 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 3, 2021 License: Apache-2.0, Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package events contains a number of different data types and formats that you can use to populate ftdc metrics series.

Custom and CustomPoint

The "custom" types allow you to construct arbirary key-value pairs without using maps and have them be well represented in FTDC output. Populate and interact with the data sequence as a slice of key (string) value (numbers) pairs, which are marshaled in the database as an object as a mapping of strings to numbers. The type provides some additional helpers for manipulating these data.

Histogram

The histogram representation is broadly similar to the Performance structure but stores data in a histogram format, which offers a high fidelity representation of a very large number of raw events without the storage overhead. In general, use histograms to collect data for operation with throughput in the thousands or more operations per second.

Performance Points

The Performance type represents a unified event to track an operation in a performance test. These events record three types of metrics: counters, timers, and gauges. Counters record the number of operations in different ways, including test iterations, logical operation counts, operation size (bytes), and error rate. Timers include both the latency of the core operation, for use in calculating latencies as well as the total taken which may be useful in calculating throughput. Finally gauges, capture changes in state or other information about the environment including the number of threads used in the test, or a failed Boolean when a test is aware of its own failure.

Recorder

The Recorder interface provides an interface for workloads and operations to to collect data about their internal state, without requiring workloads to be concerned with data retenation, storage, or compression. The implementations of Recorder have different strategies for data collection and persistence so that tests can easily change data collection strategies without modifying the test.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Collector 

type Collector interface {
	AddEvent(*Performance) error
	ftdc.Collector
}

Collector wraps the ftdc.Collector interface and adds specific awareness of the Performance type from this package. These collectors should be responsible for cumulative summing of values, when appropriate.

In general, implementations should obstruct calls to underlying collectors Add() method to avoid confusion, either by panicing or by no-oping.

func NewBasicCollector 

func NewBasicCollector(fc ftdc.Collector) Collector

NewBasicCollector produces a collector implementation that adds Performance points to the underlying FTDC collector. Counter values in the point are added to the previous point so that the values are cumulative.

This event Collector implementation captures the maximal amount of fidelity and should be used except when dictated by retention strategy.

func NewIntervalCollector 

func NewIntervalCollector(fc ftdc.Collector, interval time.Duration) Collector

NewIntervalCollector constructs a Collector collapses events as in the other collector implementations, but will record at most a single event per interval.

func NewPassthroughCollector 

func NewPassthroughCollector(fc ftdc.Collector) Collector

NewPassthroughCollector constructs a collector that does not sum Performance events and just passes them directly to the underlying collector.

func NewRandomSamplingCollector 

func NewRandomSamplingCollector(fc ftdc.Collector, sumAll bool, percent int) Collector

NewRandomSamplingCollector constructs a Collector that uses a psudorandom number generator (go's standard library math/rand) to select how often to record an event. All events are summed. Specify a percentage between 1 and 99 as the percent to reflect how many events to capture.

func NewSamplingCollector 

func NewSamplingCollector(fc ftdc.Collector, n int) Collector

NewSamplingCollector constructs a collector that has the same semantics as the basic, adding all sampled documents together, but only persisting every n-th sample to the underlying collector.

func NewSynchronizedCollector 

func NewSynchronizedCollector(coll Collector) Collector

NewSynchronizedCollector wraps another collector and wraps all required calls with the correct lock.

type Custom 

type Custom []CustomPoint

Custom is a collection of data points designed to store computed statistics at an interval. In general you will add a set of rolled-up data values to the custom object on an interval and then pass that sequence to an ftdc.Collector. Custom implements sort.Interface, and the CustomPoint type implements custom bson marshalling so that Points are marshaled as an object to facilitate their use with the ftdc format.

func MakeCustom 

func MakeCustom(size int) Custom

MakeCustom creates a Custom slice with the specified size hint.

func (*Custom) Add 

func (ps *Custom) Add(key string, value interface{}) error

Add appends a key to the Custom metric. Only accepts go native number types and timestamps.

func (Custom) Len 

func (ps Custom) Len() int

Len is a component of the sort.Interface.

func (Custom) Less 

func (ps Custom) Less(i, j int) bool

Less is a component of the sort.Interface.

func (Custom) MarshalBSON 

func (ps Custom) MarshalBSON() ([]byte, error)

func (Custom) MarshalDocument 

func (ps Custom) MarshalDocument() (*birch.Document, error)

func (Custom) Sort 

func (ps Custom) Sort()

Sort is a convenience function around a stable sort for the custom array.

func (Custom) Swap 

func (ps Custom) Swap(i, j int)

Swap is a component of the sort.Interface.

func (*Custom) UnmarshalBSON 

func (ps *Custom) UnmarshalBSON(in []byte) error

type CustomPoint 

type CustomPoint struct {
	Name  string
	Value interface{}
}

CustomPoint represents a computed statistic as a key value pair. Use with the Custom type to ensure that the Value types refer to number values and ensure consistent round trip semantics through BSON and FTDC.

type Performance 

type Performance struct {
	Timestamp time.Time           `bson:"ts" json:"ts" yaml:"ts"`
	ID        int64               `bson:"id" json:"id" yaml:"id"`
	Counters  PerformanceCounters `bson:"counters" json:"counters" yaml:"counters"`
	Timers    PerformanceTimers   `bson:"timers" json:"timers" yaml:"timers"`
	Gauges    PerformanceGauges   `bson:"gauges" json:"gauges" yaml:"gauges"`
}

Performance represents a single raw event in a metrics collection system for performance metric collection system.

Each point must report the timestamp of its collection.

func (*Performance) Add 

func (p *Performance) Add(in *Performance)

Add combines the values of the input Performance struct into this struct, logically, overriding the Gauges values as well as the timestamp and ID ID values, while summing the Counters and Timers values.

func (*Performance) MarshalBSON 

func (p *Performance) MarshalBSON() ([]byte, error)

MarshalBSON implements the bson marshaler interface to support converting this type into BSON without relying on a reflection-based BSON library.

func (*Performance) MarshalDocument 

func (p *Performance) MarshalDocument() (*birch.Document, error)

MarshalDocument exports the Performance type as a birch.Document to support more efficient operations.

func (*Performance) UnmarshalDocument 

func (p *Performance) UnmarshalDocument(doc *birch.Document) error

type PerformanceCounters 

type PerformanceCounters struct {
	Number     int64 `bson:"n" json:"n" yaml:"n"`
	Operations int64 `bson:"ops" json:"ops" yaml:"ops"`
	Size       int64 `bson:"size" json:"size" yaml:"size"`
	Errors     int64 `bson:"errors" json:"errors" yaml:"errors"`
}

PerformanceCounters refer to the number of operations/events or total of things since the last collection point. These values are used in computing various kinds of throughput measurements.

func (*PerformanceCounters) UnmarshalDocument 

func (p *PerformanceCounters) UnmarshalDocument(doc *birch.Document) error

type PerformanceCountersHDR 

type PerformanceCountersHDR struct {
	Number     *hdrhist.Histogram `bson:"n" json:"n" yaml:"n"`
	Operations *hdrhist.Histogram `bson:"ops" json:"ops" yaml:"ops"`
	Size       *hdrhist.Histogram `bson:"size" json:"size" yaml:"size"`
	Errors     *hdrhist.Histogram `bson:"errors" json:"errors" yaml:"errors"`
}

type PerformanceGauges 

type PerformanceGauges struct {
	State   int64 `bson:"state" json:"state" yaml:"state"`
	Workers int64 `bson:"workers" json:"workers" yaml:"workers"`
	Failed  bool  `bson:"failed" json:"failed" yaml:"failed"`
}

PerformanceGauges holds simple counters that aren't expected to change between points, but are useful as annotations of the experiment or or descriptions of events in the system configuration.

func (*PerformanceGauges) UnmarshalDocument 

func (p *PerformanceGauges) UnmarshalDocument(doc *birch.Document) error

type PerformanceHDR 

type PerformanceHDR struct {
	Timestamp time.Time              `bson:"ts" json:"ts" yaml:"ts"`
	ID        int64                  `bson:"id" json:"id" yaml:"id"`
	Counters  PerformanceCountersHDR `bson:"counters" json:"counters" yaml:"counters"`
	Timers    PerformanceTimersHDR   `bson:"timers" json:"timers" yaml:"timers"`
	Gauges    PerformanceGauges      `bson:"guages" json:"guages" yaml:"guages"`
}

PerformanceHDR the same as the Performance structure, but with all time duration and counter values stored as histograms.

func NewHistogramMillisecond 

func NewHistogramMillisecond(g PerformanceGauges) *PerformanceHDR

func NewHistogramSecond 

func NewHistogramSecond(g PerformanceGauges) *PerformanceHDR

func (*PerformanceHDR) MarshalDocument 

func (p *PerformanceHDR) MarshalDocument() (*birch.Document, error)

type PerformanceTimers 

type PerformanceTimers struct {
	Duration time.Duration `bson:"dur" json:"dur" yaml:"dur"`
	Total    time.Duration `bson:"total" json:"total" yaml:"total"`
}

PerformanceTimers refers to all of the timing data for this event. In general Total should equal the time since the last data point.

func (*PerformanceTimers) UnmarshalDocument 

func (p *PerformanceTimers) UnmarshalDocument(doc *birch.Document) error

type PerformanceTimersHDR 

type PerformanceTimersHDR struct {
	Duration *hdrhist.Histogram `bson:"dur" json:"dur" yaml:"dur"`
	Total    *hdrhist.Histogram `bson:"total" json:"total" yaml:"total"`
}

type Recorder 

type Recorder interface {
	// The Inc<> operations add values to the specified counters tracked by
	// the collector. There is an additional "iteration" counter that the
	// recorder tracks based on the number of times that
	// BeginIteration/EndIteration are called, but is also accessible via
	// the IncIteration counter.
	//
	// In general, IncOperations should refer to the number of logical
	// operations collected. This differs from the iteration count, in the
	// case of workloads that comprise of multiple logical operations.
	//
	// Use IncSize to record, typically, the number of bytes processed or
	// generated by the operation. Use this in combination with logical
	// operations to be able to explore the impact of data size on overall
	// performance. Finally use IncError to track the number of errors
	// encountered during the event.
	IncIterations(int64)
	IncOperations(int64)
	IncError(int64)
	IncSize(int64)

	// The Set<> operations replace existing values for the state, workers,
	// and failed gauges. Workers should typically report the number of
	// active threads. The meaning of state depends on the test
	// but can describe phases of an experiment or operation. Use SetFailed
	// to flag a test as failed during the operation.
	SetWorkers(int64)
	SetState(int64)
	SetFailed(bool)

	// The BeginIteration and EndIteration methods mark the beginning and
	// end of a test's iteration. Typically calling EndIteration records
	// the duration specified as its argument and increments the counter
	// for number of iterations. Additionally there is a "total duration"
	// value captured which represents the total time taken in the
	// iteration in addition to the operation latency.
	//
	// The EndTest method writes any unpersisted material if the
	// collector's EndTest method has not. In all cases EndTest reports all
	// errors since the last EndTest call, and resets the internal error
	// the internal error tracking and unsets the tracked starting time.
	// Generally you should call EndTest once at the end of every test run,
	// and fail if there are errors reported. Reset does the same as
	// EndTest, except for persisting data and returing errors.
	BeginIteration()
	EndIteration(time.Duration)
	EndTest() error
	Reset()

	// SetID sets the unique id for the event, to allow users to identify
	// events per thread.
	SetID(int64)

	// SetTime defines the timestamp of the current point. SetTime is
	// usually not needed: BeginIteration will set the time to the current
	// time; however, if you're using a recorder as part of
	// post-processing, you will want to use SetTime directly.
	SetTime(time.Time)

	// SetTotalDuration allows you to set the total time covered by the
	// event in question. The total time is usually derived by the
	// difference between the time set in BeginIteration and the time when
	// EndTest is called. Typically the duration passed to EndTest refers
	// to a subset of this time (i.e. the amount of time that the
	// operations in question took), and the total time, includes some
	// period of overhead.
	//
	// In simplest terms, this should typically be the time since the last
	// event was recorded.
	SetTotalDuration(time.Duration)

	// SetDuration allows you to define the duration of a the operation,
	// this is likely a subset of the total duration, with the difference
	// between the duration and the total duration, representing some kind
	// of operational overhead.
	SetDuration(time.Duration)
}

Recorder describes an interface that tests can use to track metrics and events during performance testing or normal operation. Implementations of recorder wrap an FTDC collector and will write data out to the collector for reporting purposes. The types produced by the collector use the Performance or PerformanceHDR types in this package.

Choose the implementation of Recorder that will capture all required data used by your test with sufficient resolution for use later. Additionally, consider the data volume produced by the recorder.

func NewGroupedRecorder 

func NewGroupedRecorder(collector ftdc.Collector, interval time.Duration) Recorder

NewGroupedRecorder blends the single and the interval recorders, but it persists during the EndIteration call only if the specified interval has elapsed. EndTest will persist any left over data.

The Group recorder is not safe for concurrent access without a synchronized wrapper.

func NewHistogramGroupedRecorder 

func NewHistogramGroupedRecorder(collector ftdc.Collector, interval time.Duration) Recorder

NewHistogramGroupedRecorder captures data and stores them with a histogram format. Like the Grouped recorder, it persists an event if the specified interval has elapsed since the last time an event was captured. The reset method also resets the last-collected time.

The timer histgrams have a minimum value of 1 microsecond, and a maximum value of 20 minutes, with 5 significant digits. The counter histograms store between 0 and 1 million, with 5 significant digits. The gauges are stored as integers.

The histogram Grouped reporter is not safe for concurrent use without a synchronixed wrapper.

func NewHistogramRecorder 

func NewHistogramRecorder(collector ftdc.Collector) Recorder

NewHistogramRecorder collects data and stores them with a histogram format. Like the Raw recorder, the system saves each data point after each call to EndIteration.

The timer histgrams have a minimum value of 1 microsecond, and a maximum value of 20 minutes, with 5 significant digits. The counter histograms store between 0 and 1 million, with 5 significant digits. The gauges are stored as integers.

The histogram reporter is not safe for concurrent use without a synchronized wrapper.

func NewIntervalHistogramRecorder 

func NewIntervalHistogramRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder

NewIntervalHistogramRecorder has similar semantics to histogram Grouped recorder, but has a background process that persists data on the specified on the specified interval rather than as a side effect of the EndTest call.

The background thread is started if it doesn't exist in the BeginIteration operation and is terminated by the EndTest operation.

The interval histogram recorder is safe for concurrent use.

func NewIntervalRecorder 

func NewIntervalRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder

NewIntervalRecorder has similar semantics to histogram Grouped recorder, but has a background process that persists data on the specified on the specified interval rather than as a side effect of the EndTest call.

The background thread is started in the BeginIteration operation if it does not already exist and is terminated by the EndTest operation.

The interval recorder is safe for concurrent use.

func NewRawRecorder 

func NewRawRecorder(collector ftdc.Collector) Recorder

NewRawRecorder records a new event every time that the EndIteration method is called.

The Raw recorder is not safe for concurrent access without a synchronized wrapper.

func NewShimRecorder 

func NewShimRecorder(r Recorder, tm TimerManager) Recorder

NewShimRecorder takes a recorder and acts as a thin recorder, using the TimeManager interface for relevant Begin & End values.

Go's standard library testing package has a *B type for benchmarking that can pass as a TimerManager.

func NewSingleHistogramRecorder 

func NewSingleHistogramRecorder(collector ftdc.Collector) Recorder

NewSingleHistogramRecorder collects data and stores them with a histogram format. Like the Single recorder, the implementation persists the histogram every time you call EndTest.

The timer histograms have a minimum value of 1 microsecond, and a maximum value of 1 minute, with 5 significant digits. The counter histograms store store between 0 and 10 thousand, with 5 significant digits. The gauges are stored as integers.

The histogram Single reporter is not safe for concurrent use without a synchronized wrapper.

func NewSingleRecorder 

func NewSingleRecorder(collector ftdc.Collector) Recorder

NewSingleRecorder records a single event every time the EndTest method is called, and otherwise just adds all counters and timing information to the underlying point.

The Single recorder is not safe for concurrent access without a synchronized wrapper.

func NewSynchronizedRecorder 

func NewSynchronizedRecorder(r Recorder) Recorder

NewSynchronizedRecorder wraps a recorder implementation that is not concurrent safe in a recorder implementation that provides safe concurrent access without modifying the semantics of the recorder.

Most Recorder implementations are not safe for concurrent use, although some have this property as a result of persisting data on an interval.

type TimerManager 

type TimerManager interface {
	ResetTimer()
	StartTimer()
	StopTimer()
}

TimerManager is a subset of the testing.B tool, used to manage setup code.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.