Documentation ¶
Overview ¶
Package testbed allows to easily set up a test that requires running the agent and a load generator, measure and define resource consumption expectations for the agent, fail tests automatically when expectations are exceeded.
Each test case requires a agent configuration file and (optionally) load generator spec file. Test cases are defined as regular Go tests.
Agent and load generator must be pre-built and their paths must be specified in test bed config file. The config file location must be provided in TESTBED_CONFIG env variable.
Index ¶
- Constants
- Variables
- func DoTestMain(m *testing.M)
- func GenerateSpanID(id uint64) []byte
- func GenerateTraceID(id uint64) []byte
- func GetAvailablePort(t *testing.T) int
- func LoadConfig() error
- func SaveResults()
- func Start() error
- type DataReceiver
- type DataReceiverBase
- type DataSender
- type DataSenderOverTraceExporter
- type GlobalConfig
- type JaegerDataReceiver
- type JaegerThriftDataSender
- type LoadGenerator
- type LoadOptions
- type MetricDataSender
- type MockBackend
- func (mb *MockBackend) ClearReceivedItems()
- func (mb *MockBackend) ConsumeMetric(md consumerdata.MetricsData)
- func (mb *MockBackend) ConsumeTrace(td consumerdata.TraceData)
- func (mb *MockBackend) Context() context.Context
- func (mb *MockBackend) DataItemsReceived() uint64
- func (mb *MockBackend) EnableRecording()
- func (mb *MockBackend) GetStats() string
- func (mb *MockBackend) ReportFatalError(err error)
- func (mb *MockBackend) Start() error
- func (mb *MockBackend) Stop()
- type MockMetricConsumer
- type MockTraceConsumer
- type OCDataReceiver
- type OCMetricsDataSender
- func (ome *OCMetricsDataSender) Flush()
- func (ome *OCMetricsDataSender) GenConfigYAMLStr() string
- func (ome *OCMetricsDataSender) GetCollectorPort() int
- func (ome *OCMetricsDataSender) ProtocolName() string
- func (ome *OCMetricsDataSender) SendMetrics(metrics consumerdata.MetricsData) error
- func (ome *OCMetricsDataSender) Start() error
- type OCTraceDataSender
- type OTLPDataReceiver
- type OTLPMetricsDataSender
- func (ome *OTLPMetricsDataSender) Flush()
- func (ome *OTLPMetricsDataSender) GenConfigYAMLStr() string
- func (ome *OTLPMetricsDataSender) GetCollectorPort() int
- func (ome *OTLPMetricsDataSender) ProtocolName() string
- func (ome *OTLPMetricsDataSender) SendMetrics(metrics consumerdata.MetricsData) error
- func (ome *OTLPMetricsDataSender) Start() error
- type OTLPTraceDataSender
- type ResourceConsumption
- type ResourceSpec
- type Results
- type TestCase
- func (tc *TestCase) AgentMemoryInfo() (uint32, uint32, error)
- func (tc *TestCase) EnableRecording()
- func (tc *TestCase) SetResourceLimits(resourceSpec ResourceSpec)
- func (tc *TestCase) Sleep(d time.Duration)
- func (tc *TestCase) StartAgent(args ...string)
- func (tc *TestCase) StartBackend()
- func (tc *TestCase) StartLoad(options LoadOptions)
- func (tc *TestCase) Stop()
- func (tc *TestCase) StopAgent()
- func (tc *TestCase) StopBackend()
- func (tc *TestCase) StopLoad()
- func (tc *TestCase) ValidateData()
- func (tc *TestCase) WaitFor(cond func() bool, errMsg ...interface{}) bool
- func (tc *TestCase) WaitForN(cond func() bool, duration time.Duration, errMsg ...interface{}) bool
- type TestCaseOption
- type TestResult
- type TraceDataSender
Constants ¶
const DefaultJaegerPort = 14268
const DefaultOCPort = 56565
const DefaultOTLPPort = 55680
const TESTCASE_DURATION_VAR = "TESTCASE_DURATION"
Variables ¶
var ErrSkipTests = errors.New("skip tests")
ErrSkipTests indicates that the tests must be skipped.
Functions ¶
func DoTestMain ¶
DoTestMain is intended to be run from TestMain somewhere in the test suit. This enables the testbed.
func GenerateSpanID ¶
func GenerateTraceID ¶
func GetAvailablePort ¶
func SaveResults ¶
func SaveResults()
Types ¶
type DataReceiver ¶
type DataReceiver interface { Start(tc *MockTraceConsumer, mc *MockMetricConsumer) error Stop() // Generate a config string to place in exporter part of collector config // so that it can send data to this receiver. GenConfigYAMLStr() string // Return protocol name to use in collector config pipeline. ProtocolName() string }
DataReceiver allows to receive traces or metrics. This is an interface that must be implemented by all protocols that want to be used in MockBackend. Note the terminology: testbed.DataReceiver is something that can listen and receive data from Collector and the corresponding entity in the Collector that sends this data is an exporter.
type DataReceiverBase ¶
type DataReceiverBase struct { // Port on which to listen. Port int }
DataReceiverBase implement basic functions needed by all receivers.
func (*DataReceiverBase) Context ¶
func (mb *DataReceiverBase) Context() context.Context
func (*DataReceiverBase) ReportFatalError ¶
func (mb *DataReceiverBase) ReportFatalError(err error)
type DataSender ¶
type DataSender interface { // Start sender and connect to the configured endpoint. Must be called before // sending data. Start() error // Send any accumulated data. Flush() // Return the port to which this sender will send data. GetCollectorPort() int // Generate a config string to place in receiver part of collector config // so that it can receive data from this sender. GenConfigYAMLStr() string // Return protocol name to use in collector config pipeline. ProtocolName() string }
DataSender defines the interface that allows sending data. This is an interface that must be implemented by all protocols that want to be used in LoadGenerator. Note the terminology: testbed.DataSender is something that sends data to Collector and the corresponding entity that receives the data in the Collector is a receiver.
type DataSenderOverTraceExporter ¶
type DataSenderOverTraceExporter struct { Port int // contains filtered or unexported fields }
DataSenderOverTraceExporter partially implements TraceDataSender via a TraceExporter.
func NewDataSenderOverExporter ¶
func NewDataSenderOverExporter(port int) *DataSenderOverTraceExporter
NewDataSenderOverExporter creates a new sender that will send to the specified port after Start is called.
func (*DataSenderOverTraceExporter) Flush ¶
func (ds *DataSenderOverTraceExporter) Flush()
func (*DataSenderOverTraceExporter) GetCollectorPort ¶
func (ds *DataSenderOverTraceExporter) GetCollectorPort() int
func (*DataSenderOverTraceExporter) SendSpans ¶
func (ds *DataSenderOverTraceExporter) SendSpans(traces consumerdata.TraceData) error
type GlobalConfig ¶
GlobalConfig defines test bed configuration.
type JaegerDataReceiver ¶
type JaegerDataReceiver struct { DataReceiverBase // contains filtered or unexported fields }
JaegerDataReceiver implements Jaeger format receiver.
func NewJaegerDataReceiver ¶
func NewJaegerDataReceiver(port int) *JaegerDataReceiver
func (*JaegerDataReceiver) GenConfigYAMLStr ¶
func (jr *JaegerDataReceiver) GenConfigYAMLStr() string
func (*JaegerDataReceiver) ProtocolName ¶
func (jr *JaegerDataReceiver) ProtocolName() string
func (*JaegerDataReceiver) Start ¶
func (jr *JaegerDataReceiver) Start(tc *MockTraceConsumer, mc *MockMetricConsumer) error
func (*JaegerDataReceiver) Stop ¶
func (jr *JaegerDataReceiver) Stop()
type JaegerThriftDataSender ¶
type JaegerThriftDataSender struct {
DataSenderOverTraceExporter
}
JaegerThriftDataSender implements TraceDataSender for Jaeger thrift_http protocol.
func NewJaegerThriftDataSender ¶
func NewJaegerThriftDataSender(port int) *JaegerThriftDataSender
NewJaegerThriftDataSender creates a new Jaeger protocol sender that will send to the specified port after Start is called.
func (*JaegerThriftDataSender) GenConfigYAMLStr ¶
func (je *JaegerThriftDataSender) GenConfigYAMLStr() string
func (*JaegerThriftDataSender) ProtocolName ¶
func (je *JaegerThriftDataSender) ProtocolName() string
func (*JaegerThriftDataSender) Start ¶
func (je *JaegerThriftDataSender) Start() error
type LoadGenerator ¶
type LoadGenerator struct {
// contains filtered or unexported fields
}
LoadGenerator is a simple load generator.
func NewLoadGenerator ¶
func NewLoadGenerator(sender DataSender) (*LoadGenerator, error)
NewLoadGenerator creates a load generator that sends data using specified sender.
func (*LoadGenerator) DataItemsSent ¶
func (lg *LoadGenerator) DataItemsSent() uint64
func (*LoadGenerator) GetStats ¶
func (lg *LoadGenerator) GetStats() string
GetStats returns the stats as a printable string.
func (*LoadGenerator) IncDataItemsSent ¶
func (lg *LoadGenerator) IncDataItemsSent()
IncDataItemsSent is used when a test bypasses the LoadGenerator and sends data directly via TestCases's Sender. This is necessary so that the total number of sent items in the end is correct, because the reports are printed from LoadGenerator's fields. This is not the best way, a better approach would be to refactor the reports to use their own counter and load generator and other sending sources to contribute to this counter. This could be done as a future improvement.
type LoadOptions ¶
type LoadOptions struct { // DataItemsPerSecond specifies how many spans or metric data points to generate each second. DataItemsPerSecond uint // ItemsPerBatch specifies how many spans or metric data points per batch to generate. // Should be greater than zero. The number of batches generated per second will be // DataItemsPerSecond/ItemsPerBatch. ItemsPerBatch uint // Attributes to add to each generated data item. Can be empty. Attributes map[string]string }
LoadOptions defines the options to use for generating the load.
type MetricDataSender ¶
type MetricDataSender interface { DataSender SendMetrics(metrics consumerdata.MetricsData) error }
MetricDataSender defines the interface that allows sending metric data. It adds ability to send a batch of Metrics to the DataSender interface.
type MockBackend ¶
type MockBackend struct { ReceivedTraces []consumerdata.TraceData ReceivedMetrics []consumerdata.MetricsData // contains filtered or unexported fields }
MockBackend is a backend that allows receiving the data locally.
func NewMockBackend ¶
func NewMockBackend(logFilePath string, receiver DataReceiver) *MockBackend
NewMockBackend creates a new mock backend that receives data using specified receiver.
func (*MockBackend) ClearReceivedItems ¶
func (mb *MockBackend) ClearReceivedItems()
ClearReceivedItems clears the list of received traces and metrics. Note: counters return by DataItemsReceived() are not cleared, they are cumulative.
func (*MockBackend) ConsumeMetric ¶
func (mb *MockBackend) ConsumeMetric(md consumerdata.MetricsData)
func (*MockBackend) ConsumeTrace ¶
func (mb *MockBackend) ConsumeTrace(td consumerdata.TraceData)
func (*MockBackend) Context ¶
func (mb *MockBackend) Context() context.Context
func (*MockBackend) DataItemsReceived ¶
func (mb *MockBackend) DataItemsReceived() uint64
DataItemsReceived returns total number of received spans and metrics.
func (*MockBackend) EnableRecording ¶
func (mb *MockBackend) EnableRecording()
EnableRecording enables recording of all data received by MockBackend.
func (*MockBackend) GetStats ¶
func (mb *MockBackend) GetStats() string
func (*MockBackend) ReportFatalError ¶
func (mb *MockBackend) ReportFatalError(err error)
type MockMetricConsumer ¶
type MockMetricConsumer struct {
// contains filtered or unexported fields
}
func (*MockMetricConsumer) ConsumeMetricsData ¶
func (mc *MockMetricConsumer) ConsumeMetricsData(ctx context.Context, md consumerdata.MetricsData) error
type MockTraceConsumer ¶
type MockTraceConsumer struct {
// contains filtered or unexported fields
}
func (*MockTraceConsumer) ConsumeTraceData ¶
func (tc *MockTraceConsumer) ConsumeTraceData(ctx context.Context, td consumerdata.TraceData) error
type OCDataReceiver ¶
type OCDataReceiver struct { DataReceiverBase // contains filtered or unexported fields }
OCDataReceiver implements OpenCensus format receiver.
func NewOCDataReceiver ¶
func NewOCDataReceiver(port int) *OCDataReceiver
NewOCDataReceiver creates a new OCDataReceiver that will listen on the specified port after Start is called.
func (*OCDataReceiver) GenConfigYAMLStr ¶
func (or *OCDataReceiver) GenConfigYAMLStr() string
func (*OCDataReceiver) ProtocolName ¶
func (or *OCDataReceiver) ProtocolName() string
func (*OCDataReceiver) Start ¶
func (or *OCDataReceiver) Start(tc *MockTraceConsumer, mc *MockMetricConsumer) error
func (*OCDataReceiver) Stop ¶
func (or *OCDataReceiver) Stop()
type OCMetricsDataSender ¶
type OCMetricsDataSender struct {
// contains filtered or unexported fields
}
OCMetricsDataSender implements MetricDataSender for OpenCensus metrics protocol.
func NewOCMetricDataSender ¶
func NewOCMetricDataSender(port int) *OCMetricsDataSender
NewOCMetricDataSender creates a new OpenCensus metric protocol sender that will send to the specified port after Start is called.
func (*OCMetricsDataSender) Flush ¶
func (ome *OCMetricsDataSender) Flush()
func (*OCMetricsDataSender) GenConfigYAMLStr ¶
func (ome *OCMetricsDataSender) GenConfigYAMLStr() string
func (*OCMetricsDataSender) GetCollectorPort ¶
func (ome *OCMetricsDataSender) GetCollectorPort() int
func (*OCMetricsDataSender) ProtocolName ¶
func (ome *OCMetricsDataSender) ProtocolName() string
func (*OCMetricsDataSender) SendMetrics ¶
func (ome *OCMetricsDataSender) SendMetrics(metrics consumerdata.MetricsData) error
func (*OCMetricsDataSender) Start ¶
func (ome *OCMetricsDataSender) Start() error
type OCTraceDataSender ¶
type OCTraceDataSender struct {
DataSenderOverTraceExporter
}
OCTraceDataSender implements TraceDataSender for OpenCensus trace protocol.
func NewOCTraceDataSender ¶
func NewOCTraceDataSender(port int) *OCTraceDataSender
NewOCTraceDataSender creates a new OCTraceDataSender that will send to the specified port after Start is called.
func (*OCTraceDataSender) GenConfigYAMLStr ¶
func (ote *OCTraceDataSender) GenConfigYAMLStr() string
func (*OCTraceDataSender) ProtocolName ¶
func (ote *OCTraceDataSender) ProtocolName() string
func (*OCTraceDataSender) Start ¶
func (ote *OCTraceDataSender) Start() error
type OTLPDataReceiver ¶
type OTLPDataReceiver struct { DataReceiverBase // contains filtered or unexported fields }
OTLPDataReceiver implements OTLP format receiver.
func NewOTLPDataReceiver ¶
func NewOTLPDataReceiver(port int) *OTLPDataReceiver
NewOTLPDataReceiver creates a new OTLPDataReceiver that will listen on the specified port after Start is called.
func (*OTLPDataReceiver) GenConfigYAMLStr ¶
func (or *OTLPDataReceiver) GenConfigYAMLStr() string
func (*OTLPDataReceiver) ProtocolName ¶
func (or *OTLPDataReceiver) ProtocolName() string
func (*OTLPDataReceiver) Start ¶
func (or *OTLPDataReceiver) Start(tc *MockTraceConsumer, mc *MockMetricConsumer) error
func (*OTLPDataReceiver) Stop ¶
func (or *OTLPDataReceiver) Stop()
type OTLPMetricsDataSender ¶
type OTLPMetricsDataSender struct {
// contains filtered or unexported fields
}
OTLPMetricsDataSender implements MetricDataSender for OpenCensus metrics protocol.
func NewOTLPMetricDataSender ¶
func NewOTLPMetricDataSender(port int) *OTLPMetricsDataSender
NewOTLPMetricDataSender creates a new OpenCensus metric protocol sender that will send to the specified port after Start is called.
func (*OTLPMetricsDataSender) Flush ¶
func (ome *OTLPMetricsDataSender) Flush()
func (*OTLPMetricsDataSender) GenConfigYAMLStr ¶
func (ome *OTLPMetricsDataSender) GenConfigYAMLStr() string
func (*OTLPMetricsDataSender) GetCollectorPort ¶
func (ome *OTLPMetricsDataSender) GetCollectorPort() int
func (*OTLPMetricsDataSender) ProtocolName ¶
func (ome *OTLPMetricsDataSender) ProtocolName() string
func (*OTLPMetricsDataSender) SendMetrics ¶
func (ome *OTLPMetricsDataSender) SendMetrics(metrics consumerdata.MetricsData) error
func (*OTLPMetricsDataSender) Start ¶
func (ome *OTLPMetricsDataSender) Start() error
type OTLPTraceDataSender ¶
type OTLPTraceDataSender struct {
DataSenderOverTraceExporter
}
OTLPTraceDataSender implements TraceDataSender for OpenCensus trace protocol.
func NewOTLPTraceDataSender ¶
func NewOTLPTraceDataSender(port int) *OTLPTraceDataSender
NewOTLPTraceDataSender creates a new OTLPTraceDataSender that will send to the specified port after Start is called.
func (*OTLPTraceDataSender) GenConfigYAMLStr ¶
func (ote *OTLPTraceDataSender) GenConfigYAMLStr() string
func (*OTLPTraceDataSender) ProtocolName ¶
func (ote *OTLPTraceDataSender) ProtocolName() string
func (*OTLPTraceDataSender) Start ¶
func (ote *OTLPTraceDataSender) Start() error
type ResourceConsumption ¶
type ResourceSpec ¶
type ResourceSpec struct { // Percentage of one core the process is expected to consume at most. // Test is aborted and failed if consumption during // ResourceCheckPeriod exceeds this number. If 0 the CPU // consumption is not monitored and does not affect the test result. ExpectedMaxCPU uint32 // Maximum RAM in MiB the process is expected to consume. // Test is aborted and failed if consumption exceeds this number. // If 0 memory consumption is not monitored and does not affect // the test result. ExpectedMaxRAM uint32 // Period during which CPU and RAM of the process are measured. // Bigger numbers will result in more averaging of short spikes. ResourceCheckPeriod time.Duration }
ResourceSpec is a resource consumption specification.
type Results ¶
type Results struct {
// contains filtered or unexported fields
}
func (*Results) Add ¶
func (r *Results) Add(testName string, result *TestResult)
Add results for one test.
type TestCase ¶
type TestCase struct { Sender DataSender Receiver DataReceiver LoadGenerator *LoadGenerator MockBackend *MockBackend // ErrorSignal indicates an error in the test case execution, e.g. process execution // failure or exceeding resource consumption, etc. The actual error message is already // logged, this is only an indicator on which you can wait to be informed. ErrorSignal chan struct{} // Duration is the requested duration of the tests. Configured via TESTBED_DURATION // env variable and defaults to 15 seconds if env variable is unspecified. Duration time.Duration // contains filtered or unexported fields }
TestCase defines a running test case.
func NewTestCase ¶
func NewTestCase( t *testing.T, sender DataSender, receiver DataReceiver, opts ...TestCaseOption, ) *TestCase
NewTestCase creates a new TestCase. It expects agent-config.yaml in the specified directory.
func (*TestCase) AgentMemoryInfo ¶
AgentMemoryInfo returns raw memory info struct about the agent as returned by github.com/shirou/gopsutil/process
func (*TestCase) EnableRecording ¶
func (tc *TestCase) EnableRecording()
EnableRecording enables recording of all data received by MockBackend.
func (*TestCase) SetResourceLimits ¶
func (tc *TestCase) SetResourceLimits(resourceSpec ResourceSpec)
SetResourceLimits sets expected limits for resource consmption. Error is signalled if consumption during ResourceCheckPeriod exceeds the limits. Limits are modified only for non-zero fields of resourceSpec, all zero-value fields fo resourceSpec are ignored and their previous values remain in effect.
func (*TestCase) StartAgent ¶
StartAgent starts the agent and redirects its standard output and standard error to "agent.log" file located in the test directory.
func (*TestCase) StartBackend ¶
func (tc *TestCase) StartBackend()
StartBackend starts the specified backend type.
func (*TestCase) StartLoad ¶
func (tc *TestCase) StartLoad(options LoadOptions)
StartLoad starts the load generator and redirects its standard output and standard error to "load-generator.log" file located in the test directory.
func (*TestCase) Stop ¶
func (tc *TestCase) Stop()
Stop stops the load generator, the agent and the backend.
func (*TestCase) ValidateData ¶
func (tc *TestCase) ValidateData()
ValidateData validates data by comparing the number of items sent by load generator and number of items received by mock backend.
func (*TestCase) WaitForN ¶
WaitForN the specific condition for up to a specified duration. Records a test error if time is out and condition does not become true. If error is signalled while waiting the function will return false, but will not record additional test error (we assume that signalled error is already recorded in indicateError()).
type TestCaseOption ¶
type TestCaseOption struct {
// contains filtered or unexported fields
}
TestCaseOption defines a TestCase option.
func WithConfigFile ¶
func WithConfigFile(file string) TestCaseOption
WithConfigFile allows a custom configuration file for TestCase.
func WithSkipResults ¶
func WithSkipResults() TestCaseOption
WithSkipResults option disables writing out results file for a TestCase.
func (TestCaseOption) Apply ¶
func (o TestCaseOption) Apply(t *TestCase)
Apply takes a TestCase and runs the option function on it.
type TestResult ¶
type TestResult struct {
// contains filtered or unexported fields
}
type TraceDataSender ¶
type TraceDataSender interface { DataSender SendSpans(traces consumerdata.TraceData) error }
TraceDataSender defines the interface that allows sending trace data. It adds ability to send a batch of Spans to the DataSender interface.