benchspy

package
v1.51.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 12, 2025 License: MIT Imports: 25 Imported by: 0

Documentation

Index

Constants

View Source
const DEFAULT_DIRECTORY = "performance_reports"
View Source
const (
	LogLevelEnvVar = "BENCHSPY_LOG_LEVEL"
)
View Source
const PrometheusUrlEnvVar = "PROMETHEUS_URL"

Variables

View Source
var (
	Loki_MedianQuery = `` /* 182-byte string literal not displayed */
	Loki_95thQuery   = `` /* 183-byte string literal not displayed */
	Loki_99thQuery   = `` /* 183-byte string literal not displayed */
	Loki_MaxQuery    = `` /* 177-byte string literal not displayed */
	Loki_ErrorRate   = `` /* 195-byte string literal not displayed */
)

all metrics, but error rate are calculated over a 10s interval

View Source
var (
	Prometheus_MedianCPU = `quantile_over_time(0.5, rate(container_cpu_usage_seconds_total{name=~"%s"}[5m])[%s:10s]) * 100`
	Prometheus_P95CPU    = `quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{name=~"%s"}[5m])[%s:10s]) * 100`
	Prometheus_MaxCPU    = `max(max_over_time(rate(container_cpu_usage_seconds_total{name=~"%s"}[5m])[%s:10s]) * 100)`
	Prometheus_MedianMem = `quantile_over_time(0.5, rate(container_memory_usage_bytes{name=~"%s"}[5m])[%s:10s]) * 100`
	Prometheus_P95Mem    = `quantile_over_time(0.95, rate(container_memory_usage_bytes{name=~"%s"}[5m])[%s:10s]) * 100`
	Prometheus_MaxMem    = `max(max_over_time(rate(container_memory_usage_bytes{name=~"%s"}[5m])[%s:10s]) * 100)`
)

all of them are calculated over 5 minutes intervals (rate query), that are later sampled every 10 seconds over %s duration (quantile_over_time query)

Functions

func CompareDirectWithThresholds

func CompareDirectWithThresholds(medianThreshold, p95Threshold, p99Threshold, maxThreshold, errorRateThreshold float64, currentReport, previousReport *StandardReport) (bool, error)

CompareDirectWithThresholds evaluates the current and previous reports against specified thresholds. It checks for significant differences in metrics and returns any discrepancies found, aiding in performance analysis.

func FetchNewStandardReportAndLoadLatestPrevious

func FetchNewStandardReportAndLoadLatestPrevious(ctx context.Context, newCommitOrTag string, newReportOpts ...StandardReportOption) (*StandardReport, *StandardReport, error)

FetchNewStandardReportAndLoadLatestPrevious creates a new standard report for a given commit or tag, loads the latest previous report, and checks their comparability. It returns the new report, the previous report, and any error encountered during the process.

func MustAllPrometheusResults

func MustAllPrometheusResults(sr *StandardReport) map[string]model.Value

MustAllPrometheusResults retrieves all Prometheus query results from a StandardReport. It returns a map of query names to their corresponding model.Values, ensuring type safety. This function is useful for aggregating and accessing Prometheus metrics efficiently.

func PrintStandardDirectMetrics

func PrintStandardDirectMetrics(currentReport, previousReport *StandardReport)

PrintStandardDirectMetrics outputs a comparison of direct metrics between two reports. It displays the current and previous values along with the percentage difference for each metric, helping users to quickly assess performance changes across different generator configurations.

func ResultsAs

func ResultsAs[Type any](newType Type, queryExecutor QueryExecutor, queryNames ...string) (map[string]Type, error)

ResultsAs retrieves and casts results from a query executor to a specified type. It returns a map of query names to their corresponding results, or an error if casting fails.

func StringSliceToFloat64Slice

func StringSliceToFloat64Slice(s []string) ([]float64, error)

StringSliceToFloat64Slice converts a slice of strings to a slice of float64 values. It returns an error if any string cannot be parsed as a float64, making it useful for data conversion tasks.

Types

type BasicData

type BasicData struct {
	TestName    string `json:"test_name"`
	CommitOrTag string `json:"commit_or_tag"`

	// Test metrics
	TestStart time.Time `json:"test_start_timestamp"`
	TestEnd   time.Time `json:"test_end_timestamp"`

	// all, generator settings, including segments
	GeneratorConfigs map[string]*wasp.Config `json:"generator_configs"`
}

BasicData is the basic data that is required for a report, common to all reports

func MustNewBasicData

func MustNewBasicData(commitOrTag string, generators ...*wasp.Generator) BasicData

MustNewBasicData creates a new BasicData instance from a commit or tag. It panics if the creation fails, ensuring that the caller receives a valid instance.

func NewBasicData

func NewBasicData(commitOrTag string, generators ...*wasp.Generator) (*BasicData, error)

NewBasicData creates a new BasicData instance using the provided commit or tag and a list of generators. It ensures that at least one generator is provided and that it is associated with a testing.T instance. This function is essential for initializing test data configurations in a structured manner.

func (*BasicData) FillStartEndTimes

func (b *BasicData) FillStartEndTimes() error

FillStartEndTimes calculates the earliest start time and latest end time from generator schedules. It updates the BasicData instance with these times, ensuring all segments have valid start and end times.

func (*BasicData) IsComparable

func (b *BasicData) IsComparable(otherData BasicData) error

IsComparable checks if two BasicData instances have the same configuration settings. It validates the count, presence, and equivalence of generator configurations, returning an error if any discrepancies are found. This function is useful for ensuring consistency between data reports before processing or comparison.

func (*BasicData) Validate

func (b *BasicData) Validate() error

Validate checks the integrity of the BasicData fields, ensuring that the test start and end times are set, and that at least one generator configuration is provided. It returns an error if any of these conditions are not met.

type Comparator

type Comparator interface {
	// IsComparable checks whether both reports can be compared (e.g. test config is the same, app's resources are the same, queries or metrics used are the same, etc.), and an error if any difference is found
	IsComparable(otherReport Reporter) error
}

type DataFetcher

type DataFetcher interface {
	// Fetch populates the report with the data from the test
	FetchData(ctx context.Context) error
}

type DirectQueryExecutor

type DirectQueryExecutor struct {
	KindName     string                   `json:"kind"`
	Generator    *wasp.Generator          `json:"generator_config"`
	Queries      map[string]DirectQueryFn `json:"queries"`
	QueryResults map[string]interface{}   `json:"query_results"`
}

func NewDirectQueryExecutor

func NewDirectQueryExecutor(generator *wasp.Generator, queries map[string]DirectQueryFn) (*DirectQueryExecutor, error)

NewDirectQueryExecutor creates a new DirectQueryExecutor with the specified generator and query functions. It initializes the executor with a kind name and prepares a map for query results, enabling efficient query execution.

func NewStandardDirectQueryExecutor

func NewStandardDirectQueryExecutor(generator *wasp.Generator) (*DirectQueryExecutor, error)

NewStandardDirectQueryExecutor creates a new DirectQueryExecutor configured for standard queries. It initializes the executor and generates the necessary queries, returning the executor or an error if the process fails.

func (*DirectQueryExecutor) Execute

func (dqe *DirectQueryExecutor) Execute(_ context.Context) error

Execute runs the defined queries using the data from the generator. It validates the generator's data and aggregates responses before executing each query. This function is essential for processing and retrieving results from multiple queries concurrently.

func (*DirectQueryExecutor) GeneratorName

func (dqe *DirectQueryExecutor) GeneratorName() string

GeneratorName returns the name of the generator associated with the query executor. It is useful for identifying and categorizing results based on their generator type.

func (*DirectQueryExecutor) IsComparable

func (dqe *DirectQueryExecutor) IsComparable(otherQueryExecutor QueryExecutor) error

IsComparable checks if the given QueryExecutor is of the same type and has comparable configurations. It returns an error if the types do not match or if the configurations are not comparable.

func (*DirectQueryExecutor) Kind

func (dqe *DirectQueryExecutor) Kind() string

Kind returns the type of the query executor as a string. It is useful for identifying the specific implementation of a query executor in a collection.

func (*DirectQueryExecutor) MarshalJSON

func (dqe *DirectQueryExecutor) MarshalJSON() ([]byte, error)

MarshalJSON customizes the JSON representation of the DirectQueryExecutor. It serializes only the relevant fields, including query names and results, making it suitable for efficient data transmission and storage.

func (*DirectQueryExecutor) Results

func (dqe *DirectQueryExecutor) Results() map[string]interface{}

Results returns the query results as a map of string keys to interface{} values. It allows users to access the outcomes of executed queries, facilitating further processing or type assertions.

func (*DirectQueryExecutor) TimeRange

func (dqe *DirectQueryExecutor) TimeRange(_, _ time.Time)

TimeRange ensures that the query executor operates within the specified time range. It is a no-op for executors that already have responses stored in the correct time range.

func (*DirectQueryExecutor) UnmarshalJSON

func (dqe *DirectQueryExecutor) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes JSON data into a DirectQueryExecutor instance. It populates the executor's fields, including queries and results, enabling seamless integration of JSON configurations into the executor's structure.

func (*DirectQueryExecutor) Validate

func (dqe *DirectQueryExecutor) Validate() error

Validate checks if the query executor is properly configured. It ensures that a generator is set and at least one query is provided. Returns an error if validation fails, helping to prevent execution issues.

type DirectQueryFn

type DirectQueryFn = func(responses *wasp.SliceBuffer[*wasp.Response]) (float64, error)

type DirectResultsByGenerator

type DirectResultsByGenerator map[string]map[string]float64

func MustAllDirectResults

func MustAllDirectResults(sr *StandardReport) DirectResultsByGenerator

MustAllDirectResults extracts and returns all direct results from a given StandardReport. It panics if any result extraction fails, ensuring that only valid results are processed.

type LocalStorage

type LocalStorage struct {
	Directory string `json:"directory"`
}

func (*LocalStorage) Load

func (l *LocalStorage) Load(testName, commitOrTag string, report interface{}) error

Load retrieves a report from local storage based on the specified test name and optional commit or tag. It decodes the report into the provided interface, enabling users to access stored test results.

func (*LocalStorage) Store

func (l *LocalStorage) Store(testName, commitOrTag string, report interface{}) (string, error)

Store saves a test report as a JSON file in local storage. It organizes reports by test name and commit/tag, ensuring easy retrieval and management. Returns the absolute path of the stored report or an error if the operation fails.

type LokiQueryExecutor

type LokiQueryExecutor struct {
	KindName            string `json:"kind"`
	GeneratorNameString string `json:"generator_name"`

	// Test metrics
	StartTime time.Time `json:"start_time"`
	EndTime   time.Time `json:"end_time"`

	// a map of name to query template, ex: "average cpu usage": "avg(rate(cpu_usage_seconds_total[5m]))"
	Queries map[string]string `json:"queries"`
	// can be anything, avg RPS, amount of errors, 95th percentile of CPU utilization, etc
	QueryResults map[string]interface{} `json:"query_results"`

	Config *wasp.LokiConfig `json:"-"`
}

func NewLokiQueryExecutor

func NewLokiQueryExecutor(generatorName string, queries map[string]string, lokiConfig *wasp.LokiConfig) *LokiQueryExecutor

NewLokiQueryExecutor creates a new LokiQueryExecutor instance. It initializes the executor with the specified generator name, queries, and Loki configuration. This function is useful for setting up a query executor to interact with Loki for log data retrieval.

func NewStandardMetricsLokiExecutor

func NewStandardMetricsLokiExecutor(lokiConfig *wasp.LokiConfig, testName, generatorName, branch, commit string, startTime, endTime time.Time) (*LokiQueryExecutor, error)

NewStandardMetricsLokiExecutor creates a LokiQueryExecutor configured with standard metrics queries. It generates queries based on provided test parameters and time range, returning the executor or an error if query generation fails.

func (*LokiQueryExecutor) Execute

func (l *LokiQueryExecutor) Execute(ctx context.Context) error

Execute runs the configured Loki queries concurrently and collects the results. It requires a valid configuration and handles basic authentication if provided. The function returns an error if any query execution fails or if the configuration is missing.

func (*LokiQueryExecutor) GeneratorName

func (l *LokiQueryExecutor) GeneratorName() string

GeneratorName returns the name of the generator associated with the LokiQueryExecutor. It is useful for identifying the source of results in reports or logs.

func (*LokiQueryExecutor) IsComparable

func (l *LokiQueryExecutor) IsComparable(otherQueryExecutor QueryExecutor) error

IsComparable checks if the given QueryExecutor is of the same type as the current instance. It compares the queries of both executors to ensure they are equivalent in structure and content. This function is useful for validating compatibility between different query executors.

func (*LokiQueryExecutor) Kind

func (l *LokiQueryExecutor) Kind() string

Kind returns the type of the query executor as a string. It is used to identify the specific kind of query executor in various operations.

func (*LokiQueryExecutor) Results

func (l *LokiQueryExecutor) Results() map[string]interface{}

Results returns the query results as a map of string to interface{}. It allows users to access the outcomes of executed queries, facilitating further processing or type assertions.

func (*LokiQueryExecutor) TimeRange

func (l *LokiQueryExecutor) TimeRange(start, end time.Time)

TimeRange sets the start and end time for the Loki query execution. This function is essential for defining the time window of the data to be fetched.

func (*LokiQueryExecutor) UnmarshalJSON

func (l *LokiQueryExecutor) UnmarshalJSON(data []byte) error

UnmarshalJSON parses the JSON-encoded data and populates the LokiQueryExecutor fields. It converts the query results from a generic map to a specific type map, enabling type-safe access to the results.

func (*LokiQueryExecutor) Validate

func (l *LokiQueryExecutor) Validate() error

Validate checks if the LokiQueryExecutor has valid queries and configuration. It returns an error if no queries are set or if the configuration is missing, ensuring that the executor is ready for execution.

type LokiResultsByGenerator

type LokiResultsByGenerator map[string]map[string][]string

func MustAllLokiResults

func MustAllLokiResults(sr *StandardReport) LokiResultsByGenerator

MustAllLokiResults retrieves and aggregates results from all Loki query executors in a StandardReport. It panics if any query execution fails, ensuring that only successful results are returned.

type NamedGenerator

type NamedGenerator interface {
	// GeneratorName returns the name of the generator
	GeneratorName() string
}

type PrometheusConfig

type PrometheusConfig struct {
	Url               string
	NameRegexPatterns []string
}
var WithoutPrometheus *PrometheusConfig

func NewPrometheusConfig

func NewPrometheusConfig(nameRegexPatterns ...string) *PrometheusConfig

NewPrometheusConfig creates a new PrometheusConfig instance with the specified name regex patterns. It retrieves the Prometheus URL from the environment and is used to configure query execution for Prometheus data sources.

type PrometheusQueryExecutor

type PrometheusQueryExecutor struct {
	KindName     string                 `json:"kind"`
	StartTime    time.Time              `json:"start_time"`
	EndTime      time.Time              `json:"end_time"`
	Queries      map[string]string      `json:"queries"`
	QueryResults map[string]interface{} `json:"query_results"`
	// contains filtered or unexported fields
}

func NewPrometheusQueryExecutor

func NewPrometheusQueryExecutor(queries map[string]string, config *PrometheusConfig) (*PrometheusQueryExecutor, error)

NewPrometheusQueryExecutor creates a new PrometheusResourceReporter, url should include basic auth if needed

func NewStandardPrometheusQueryExecutor

func NewStandardPrometheusQueryExecutor(startTime, endTime time.Time, config *PrometheusConfig) (*PrometheusQueryExecutor, error)

NewStandardPrometheusQueryExecutor creates a PrometheusQueryExecutor with standard queries based on the provided time range and configuration. It simplifies the process of generating queries for Prometheus, making it easier to integrate Prometheus data into reports.

func (*PrometheusQueryExecutor) Execute

func (pe *PrometheusQueryExecutor) Execute(ctx context.Context) error

Execute runs the defined Prometheus queries concurrently, collecting results and warnings. It returns an error if any query fails, allowing for efficient data retrieval in reporting tasks.

func (*PrometheusQueryExecutor) IsComparable

func (pe *PrometheusQueryExecutor) IsComparable(other QueryExecutor) error

IsComparable checks if the provided QueryExecutor is of the same type as the receiver. It returns an error if the types do not match, ensuring type safety for query comparisons.

func (*PrometheusQueryExecutor) Kind

func (pe *PrometheusQueryExecutor) Kind() string

Kind returns the type of the query executor as a string. It is used to identify the specific kind of executor in a collection of query executors.

func (*PrometheusQueryExecutor) MarshalJSON

func (pe *PrometheusQueryExecutor) MarshalJSON() ([]byte, error)

MarshalJSON customizes the JSON representation of PrometheusQueryExecutor. It includes only essential fields: Kind, Queries, and simplified QueryResults. This function is useful for serializing the executor's state in a concise format.

func (*PrometheusQueryExecutor) MustResultsAsValue

func (pe *PrometheusQueryExecutor) MustResultsAsValue() map[string]model.Value

MustResultsAsValue retrieves the query results as a map of metric names to their corresponding values. It ensures that the results are in a consistent format, making it easier to work with metrics in subsequent operations.

func (*PrometheusQueryExecutor) Results

func (pe *PrometheusQueryExecutor) Results() map[string]interface{}

Results returns the query results as a map of string to interface{}. It allows users to access the results of executed queries, facilitating data retrieval and manipulation.

func (*PrometheusQueryExecutor) TimeRange

func (pe *PrometheusQueryExecutor) TimeRange(startTime, endTime time.Time)

TimeRange sets the start and end time for the Prometheus query execution. This function is essential for defining the time window for data retrieval, ensuring accurate and relevant results.

func (*PrometheusQueryExecutor) UnmarshalJSON

func (pe *PrometheusQueryExecutor) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes JSON data into a PrometheusQueryExecutor instance. It populates the QueryResults field with appropriately typed metrics, enabling easy access to the results of Prometheus queries.

func (*PrometheusQueryExecutor) Validate

func (pe *PrometheusQueryExecutor) Validate() error

Validate checks the PrometheusQueryExecutor for a valid client and ensures that at least one query is provided. It returns an error if the client is nil or no queries are specified, helping to ensure proper configuration before execution.

func (*PrometheusQueryExecutor) Warnings

func (pe *PrometheusQueryExecutor) Warnings() map[string]v1.Warnings

Warnings returns a map of warnings encountered during query execution. This function is useful for retrieving any issues that may have arisen, allowing users to handle or log them appropriately.

type QueryExecutor

type QueryExecutor interface {
	// Kind returns the type of the QueryExecutor
	Kind() string
	// Validate checks if the QueryExecutor has all the necessary data and configuration to execute the queries
	Validate() error
	// Execute executes the queries and populates the QueryExecutor with the results
	Execute(ctx context.Context) error
	// Results returns the results of the queries, where key is the name of the query and value is the result
	Results() map[string]interface{}
	// IsComparable checks whether both QueryExecutors can be compared (e.g. they have the same type, queries are the same, etc.), and returns an error (if any difference is found)
	IsComparable(other QueryExecutor) error
	// TimeRange sets the time range for the queries
	TimeRange(startTime, endTime time.Time)
}

type Reporter

type Reporter interface {
	Storer
	DataFetcher
	Comparator
}

type StandardLoadMetric

type StandardLoadMetric string
const (
	MedianLatency       StandardLoadMetric = "median_latency"
	Percentile95Latency StandardLoadMetric = "95th_percentile_latency"
	Percentile99Latency StandardLoadMetric = "99th_percentile_latency"
	MaxLatency          StandardLoadMetric = "max_latency"
	ErrorRate           StandardLoadMetric = "error_rate"
)

type StandardQueryExecutorType

type StandardQueryExecutorType string
const (
	StandardQueryExecutor_Loki       StandardQueryExecutorType = "loki"
	StandardQueryExecutor_Direct     StandardQueryExecutorType = "direct"
	StandardQueryExecutor_Prometheus StandardQueryExecutorType = "prometheus"
)

type StandardReport

type StandardReport struct {
	BasicData
	LocalStorage
	QueryExecutors []QueryExecutor `json:"query_executors"`
}

StandardReport is a report that contains all the necessary data for a performance test

func NewStandardReport

func NewStandardReport(commitOrTag string, opts ...StandardReportOption) (*StandardReport, error)

NewStandardReport creates a new StandardReport based on the provided commit or tag and options. It initializes necessary data and query executors, ensuring all configurations are validated. This function is essential for generating reports that require specific data sources and execution strategies.

func (*StandardReport) FetchData

func (sr *StandardReport) FetchData(ctx context.Context) error

FetchData retrieves data for the report within the specified time range. It validates the time range and executes queries in parallel, returning any errors encountered during execution.

func (*StandardReport) IsComparable

func (sr *StandardReport) IsComparable(otherReport Reporter) error

IsComparable checks if the current report can be compared with another report. It validates the type of the other report and ensures that their basic data and query executors are comparable. This function is useful for verifying report consistency before performing further analysis.

func (*StandardReport) Load

func (sr *StandardReport) Load(testName, commitOrTag string) error

Load retrieves a report based on the specified test name and commit or tag. It utilizes local storage to find and decode the corresponding report file, ensuring that the report is available for further processing or analysis.

func (*StandardReport) LoadLatest

func (sr *StandardReport) LoadLatest(testName string) error

LoadLatest retrieves the most recent report for the specified test name from local storage. It returns an error if the report cannot be loaded, enabling users to access historical test data efficiently.

func (*StandardReport) Store

func (sr *StandardReport) Store() (string, error)

Store saves the report to local storage as a JSON file. It returns the absolute path of the stored file and any error encountered.

func (*StandardReport) UnmarshalJSON

func (sr *StandardReport) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes JSON data into a StandardReport struct. It populates the QueryExecutors and ResourceFetchers fields, allowing for dynamic handling of JSON structures in reports.

type StandardReportOption

type StandardReportOption func(*standardReportConfig)

func WithGenerators

func WithGenerators(generators ...*wasp.Generator) StandardReportOption

WithGenerators sets the generators for the standard report configuration. It allows users to specify custom generator instances to be included in the report.

func WithPrometheusConfig

func WithPrometheusConfig(prometheusConfig *PrometheusConfig) StandardReportOption

WithPrometheusConfig sets the Prometheus configuration for the standard report. It returns a StandardReportOption that can be used to customize report generation.

func WithQueryExecutors

func WithQueryExecutors(queryExecutors ...QueryExecutor) StandardReportOption

WithQueryExecutors sets the query executors for a standard report configuration. It allows customization of how queries are executed, enhancing report generation flexibility.

func WithReportDirectory

func WithReportDirectory(reportDirectory string) StandardReportOption

WithReportDirectory sets the directory for storing report files. This function is useful for configuring the output location of reports generated by the standard reporting system.

func WithStandardQueries

func WithStandardQueries(executorTypes ...StandardQueryExecutorType) StandardReportOption

WithStandardQueries sets the executor types for a standard report configuration. It allows users to specify which types of query executors to use, enabling customization of report generation based on their requirements.

type StandardResourceMetric

type StandardResourceMetric string
const (
	MedianCPUUsage StandardResourceMetric = "median_cpu_usage"
	MedianMemUsage StandardResourceMetric = "median_mem_usage"
	P95CPUUsage    StandardResourceMetric = "p95_cpu_usage"
	MaxCPUUsage    StandardResourceMetric = "max_cpu_usage"
	P95MemUsage    StandardResourceMetric = "p95_mem_usage"
	MaxMemUsage    StandardResourceMetric = "max_mem_usage"
)

type Storer

type Storer interface {
	// Store stores the report in a persistent storage and returns the path to it, or an error
	Store() (string, error)
	// Load loads the report from a persistent storage and returns it, or an error
	Load(testName, commitOrTag string) error
	// LoadLatest loads the latest report from a persistent storage and returns it, or an error
	LoadLatest(testName string) error
}

type TypedMetric

type TypedMetric struct {
	Value      model.Value `json:"value"`
	MetricType string      `json:"metric_type"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL