promqltest

package
v0.310.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 25, 2026 License: Apache-2.0 Imports: 29 Imported by: 2

README

The PromQL test scripting language

This package contains two things:

  • an implementation of a test scripting language for PromQL engines
  • a predefined set of tests written in that scripting language

The predefined set of tests can be run against any PromQL engine implementation by calling promqltest.RunBuiltinTests(). Any other test script can be run with promqltest.RunTest().

The rest of this document explains the test scripting language.

Each test script is written in plain text.

Comments can be given by prefixing the comment with a #, for example:

# This is a comment.

Each test file contains a series of commands. There are three kinds of commands:

  • load
  • clear
  • eval

Note: The eval command variants (eval_fail, eval_warn, eval_info, and eval_ordered) are deprecated. Use the new expect lines instead (explained in the eval command section). Additionally, expected_fail_message and expected_fail_regexp are also deprecated.

Each command is executed in the order given in the file.

load command

load adds some data to the test environment.

The syntax is as follows:

load <interval>
    <series> <points>
    ...
    <series> <points>
  • <interval> is the step between points (eg. 1m or 30s)
  • <series> is a Prometheus series name in the usual metric{label="value"} syntax
  • <points> is a specification of the points to add for that series, following the same expanding syntax as for promtool unittest documented here

For example:

load 1m
    my_metric{env="prod"} 5 2+3x2 _ stale {{schema:1 sum:3 count:22 buckets:[5 10 7]}}

… will create a single series with labels my_metric{env="prod"}, with the following points:

  • t=0: value is 5
  • t=1m: value is 2
  • t=2m: value is 5
  • t=3m: value is 8
  • t=4m: no point
  • t=5m: stale marker
  • t=6m: native histogram with schema 1, sum -3, count 22 and bucket counts 5, 10 and 7

Each load command is additive - it does not replace any data loaded in a previous load command. Use clear to remove all loaded data.

Native histograms with custom buckets (NHCB)

When loading a batch of classic histogram float series, you can optionally append the suffix _with_nhcb to convert them to native histograms with custom buckets and load both the original float series and the new histogram series.

clear command

clear removes all data previously loaded with load commands.

eval command

eval runs a query against the test environment and asserts that the result is as expected. It requires the query to succeed without any failures unless an expect fail line is provided. Previously eval expected no info or warn annotation, but now expect no_info and expect no_warn lines must be explicitly provided.

Both instant and range queries are supported.

The syntax is as follows:

# Instant query
eval instant at <time> <query>
    <expect>
    ...
    <expect>
    <series> <points>
    ...
    <series> <points>
    
# Range query
eval range from <start> to <end> step <step> <query>
    <expect>
    ...
    <expect>
    <series> <points>
    ...
    <series> <points>
  • <time> is the timestamp to evaluate the instant query at (eg. 1m)
  • <start> and <end> specify the time range of the range query, and use the same syntax as <time>
  • <step> is the step of the range query, and uses the same syntax as <time> (eg. 30s)
  • <expect>(optional) specifies expected annotations, errors, or result ordering.
  • <expect range vector> (optional) for an instant query you can specify expected range vector timestamps
  • <expect string> "<string>" (optional) for matching a string literal
  • <series> and <points> specify the expected values, and follow the same syntax as for load above
Special handling of counter reset hints in native histograms

Native histograms as part of <points> may or may not contain an explicit counter_reset_hint property. If a counter_reset_hint is provided explicitly, the counter reset hint of the histogram is tested to have the provided value (unknown, reset, not_reset, or gauge). However, if no counter_reset_hint is specified, the counter_reset_hint is not tested at all (rather than testing for the usual default value unknown).

expect string

This can be used to specify that a string literal is the expected result.

Note that this is only supported on instant queries.

For example;

eval instant at 50m ("Foo")
 expect string "Foo"

The expected string value must be within quotes. Double or back quotes are supported.

expect range vector

This can be used to specify the expected timestamps on a range vector resulting from an instant query.

expect range vector <start> to <end> step <step>

For example;

load 10s
  some_metric{env="a"} 1+1x5
  some_metric{env="b"} 2+2x5
eval instant at 1m some_metric[1m]
  expect range vector from 10s to 1m step 10s
  some_metric{env="a"} 2 3 4 5 6
  some_metric{env="b"} 4 6 8 10 12
expect Syntax
expect <type> <match_type>: <string>
Parameters
  • <type> is the expectation type:
    • fail expects the query to fail.
    • info expects the query to return at least one info annotation.
    • warn expects the query to return at least one warn annotation.
    • no_info expects the query to return no info annotation.
    • no_warn expects the query to return no warn annotation.
    • ordered expects the query to return the results in the specified order.
  • <match_type> (optional) specifies message matching type for annotations:
    • msg for exact string match.
    • regex for regular expression match.
    • Not applicable for ordered, no_info, and no_warn.
  • <string> is the expected annotation message.

For example:

eval instant at 1m sum by (env) (my_metric)
    expect warn
    expect no_info
    {env="prod"} 5
    {env="test"} 20
    
eval range from 0 to 3m step 1m sum by (env) (my_metric)
    expect warn msg: something went wrong
    expect info regex: something went (wrong|boom)
    {env="prod"} 2 5 10 20
    {env="test"} 10 20 30 45

eval instant at 1m ceil({__name__=~'testmetric1|testmetric2'})
expect fail

eval instant at 1m ceil({__name__=~'testmetric1|testmetric2'})
expect fail msg: "vector cannot contain metrics with the same labelset"

eval instant at 1m ceil({__name__=~'testmetric1|testmetric2'})
expect fail regex: "vector cannot contain metrics .*|something else went wrong"

eval instant at 1m sum by (env) (my_metric)
expect ordered
{env="prod"} 5
{env="test"} 20

There can be multiple <expect> lines for a given <type>. Each <type> validates its corresponding annotation, error, or ordering while ignoring others.

Every <expect> line must match at least one corresponding annotation or error.

If at least one <expect> line of type warn or info is present, then all corresponding annotations must have a matching expect line.

Migrating Test Files to the New Syntax
  • All .test files in the directory specified by the --dir flag will be updated in place.
  • Deprecated syntax will be replaced with the recommended expect line statements.

Usage:

go run ./promql/promqltest/cmd/migrate/main.go --mode=strict [--dir=<directory>]

The --mode flag controls how expectations are migrated:

  • strict: Strictly migrates all expectations to the new syntax. This is probably more verbose than intended because the old syntax implied many constraints that are often not needed.
  • basic: Like strict but never creates no_info and no_warn expectations. This can be a good starting point to manually add no_info and no_warn expectations and/or remove info and warn expectations as needed.
  • tolerant: Only creates expect fail and expect ordered where appropriate. All desired expectations about presence or absence of info and warn have to be added manually.

All three modes create valid passing tests from previously passing tests. basic and tolerant just test fewer expectations than the previous tests.

The --dir flag specifies the directory containing test files to migrate.

Documentation

Index

Constants

View Source 
const (
	Ordered expectCmdType = iota
	Fail
	Warn
	NoWarn
	Info
	NoInfo
)
View Source 
const (
	DefaultMaxSamplesPerQuery = 10000
)

Variables

This section is empty.

Functions

func GetBuiltInExprs added in v0.310.0 

func GetBuiltInExprs() ([]string, error)

GetBuiltInExprs returns all the eval statement expressions from the built-in test files.

func HistogramTestExpression 

func HistogramTestExpression(h *histogram.FloatHistogram) string

HistogramTestExpression returns TestExpression() for the given histogram or "" if the histogram is nil.

func LoadedStorage 

func LoadedStorage(t testing.TB, input string) *teststorage.TestStorage

LoadedStorage returns storage with generated data using the provided load statements. Non-load statements will cause test errors.

func MigrateTestData added in v0.305.0 

func MigrateTestData(mode, dir string) error

MigrateTestData migrates all PromQL test files to the new syntax format. It applies annotation rules based on the provided migration mode ("strict", "basic", or "tolerant"). The function parses each .test file, converts it to the new syntax and overwrites the file.

func NewTestEngine 

func NewTestEngine(tb testing.TB, enablePerStepStats bool, lookbackDelta time.Duration, maxSamples int) *promql.Engine

NewTestEngine creates a promql.Engine with enablePerStepStats, lookbackDelta and maxSamples, and returns it.

func NewTestEngineWithOpts added in v0.55.0 

func NewTestEngineWithOpts(tb testing.TB, opts promql.EngineOpts) *promql.Engine

NewTestEngineWithOpts creates a promql.Engine with opts and returns it.

func RunBuiltinTests 

func RunBuiltinTests(t TBRun, engine promql.QueryEngine)

RunBuiltinTests runs an acceptance test suite against the provided engine.

func RunBuiltinTestsWithStorage added in v0.301.0 

func RunBuiltinTestsWithStorage(t TBRun, engine promql.QueryEngine, newStorage func(testing.TB) storage.Storage)

RunBuiltinTestsWithStorage runs an acceptance test suite against the provided engine and storage.

func RunTest 

func RunTest(t testing.TB, input string, engine promql.QueryEngine)

RunTest parses and runs the test against the provided engine.

func RunTestWithStorage added in v0.301.0 

func RunTestWithStorage(t testing.TB, input string, engine promql.QueryEngine, newStorage func(testing.TB) storage.Storage)

RunTestWithStorage parses and runs the test against the provided engine and storage.

Types

type LazyLoader 

type LazyLoader struct {
	SubqueryInterval time.Duration
	// contains filtered or unexported fields
}

LazyLoader lazily loads samples into storage. This is specifically implemented for unit testing of rules.

func NewLazyLoader 

func NewLazyLoader(input string, opts LazyLoaderOpts) (*LazyLoader, error)

NewLazyLoader returns an initialized empty LazyLoader.

func (*LazyLoader) Close 

func (ll *LazyLoader) Close() error

Close closes resources associated with the LazyLoader.

func (*LazyLoader) Context 

func (ll *LazyLoader) Context() context.Context

Context returns the LazyLoader's context.

func (*LazyLoader) QueryEngine 

func (ll *LazyLoader) QueryEngine() *promql.Engine

QueryEngine returns the LazyLoader's query engine.

func (*LazyLoader) Queryable 

func (ll *LazyLoader) Queryable() storage.Queryable

Queryable allows querying the LazyLoader's data. Note: only the samples till the max timestamp used in `WithSamplesTill` can be queried.

func (*LazyLoader) Storage 

func (ll *LazyLoader) Storage() storage.Storage

Storage returns the LazyLoader's storage.

func (*LazyLoader) WithSamplesTill 

func (ll *LazyLoader) WithSamplesTill(ts time.Time, fn func(error))

WithSamplesTill loads the samples till given timestamp and executes the given function.

type LazyLoaderOpts 

type LazyLoaderOpts struct {
	// Both of these must be set to true for regular PromQL (as of
	// Prometheus v2.33). They can still be disabled here for legacy and
	// other uses.
	EnableAtModifier, EnableNegativeOffset bool
	// Currently defaults to false, matches the "promql-delayed-name-removal"
	// feature flag.
	EnableDelayedNameRemoval bool
	// StartTime is the start time for the test. If zero, defaults to Unix epoch.
	StartTime time.Time
}

LazyLoaderOpts are options for the lazy loader.

type MigrateMode added in v0.305.0 

type MigrateMode int
const (
	MigrateStrict MigrateMode = iota
	MigrateBasic
	MigrateTolerant
)

func ParseMigrateMode added in v0.305.0 

func ParseMigrateMode(s string) (MigrateMode, error)

type TBRun added in v0.54.0 

type TBRun interface {
	testing.TB
	Run(string, func(*testing.T)) bool
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
migrate command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.