querybench

README

Distributed query benchmark

This package benchmarks row vs vectorized distributed query on a local in-process BanyanDB distributed cluster. The harness is gated to run only inside a resource-limited Docker container, and every (mode, scenario, cardinality, variant) combo runs in its own Go process so heap and CPU profiles describe exactly that combo -- no cross-scenario leakage in the cumulative allocation snapshot.

DQB_ENGINE=measure is the default and preserves the original measure benchmark behavior. DQB_ENGINE=trace enables the distributed trace benchmark.

Measure scenarios:

• scan_all: mirrors test/cases/measure/data/input/all.yaml for service_cpm_minute.
• top_with_filter: mirrors test/cases/measure/data/input/top_with_filter.yaml (id != svc3, group by id, MEAN(value), top 2 desc).

Trace scenarios:

• trace_by_id: trace ID point/bulk lookup using trace_id IN (...), with limit pinned to the trace ID batch size.
• trace_tag_filter: ordered tag filter using service_id = svc-0, order_by timestamp DESC, and a pinned trace-count limit.

Default measure cardinalities: 1024,10000,100000,1000000,2000000. Default trace cardinalities: 1000,10000,100000,1000000,2000000.

How the harness is shaped

run-docker.sh starts one resource-limited container and invokes orchestrate.sh inside it. The inner orchestrator:

1. Builds the test binary once into /tmp/dqb.test.
2. Clears ${DQB_REPORT_DIR}/shards/ and ${DQB_REPORT_DIR}/profiles/ so the merged report reflects only this run.
3. Iterates the selected matrix and invokes the binary once per combo with DQB_ENGINE, DQB_MODE, DQB_SCENARIO, DQB_CARDINALITY, and any trace variant dimensions set. Each invocation:
   • Boots a fresh cluster (row mode plain, vec mode with the selected engine's vectorized flag enabled on every node).
   • Writes the data set at the configured cardinality.
   • Runs warmup + timed queries for one scenario.
   • Captures CPU and heap profiles bracketed around the timed phase.
   • Writes a shard JSON under ${DQB_REPORT_DIR}/shards/ and exits. Measure keeps the original <mode>_<scenario>_<cardinality>.json filenames; trace shard names include the full variant key.
4. Invokes the binary one last time with DQB_MERGE=1. The merge pass reads every shard, computes vec/row correctness, and writes distributed-querybench.json + distributed-querybench.md with the unified results and the vec/row ratio table.

Run profiles

Smoke (single small cardinality, fast iteration):

DQB_CARDINALITIES=1024 \
DQB_QUERY_ITERATIONS=10 \
DQB_PROFILE=1 \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

Standard sweep through 1M rows:

DQB_CARDINALITIES=1024,10000,100000,1000000 \
DQB_QUERY_ITERATIONS=50 \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

Stress (>1M):

DQB_CARDINALITIES=2000000 \
DQB_QUERY_ITERATIONS=20 \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

Trace smoke:

DQB_ENGINE=trace \
DQB_MATRIX=A \
DQB_CARDINALITIES=1000 \
DQB_SCENARIOS=trace_by_id,trace_tag_filter \
DQB_SPANS_PER_TRACE=20 \
DQB_QUERY_ITERATIONS=10 \
DQB_TRACE_BY_ID_ITERATIONS=20 \
DQB_PROFILE=1 \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

Trace standard sweep through 1M spans:

DQB_ENGINE=trace \
DQB_MATRIX=A \
DQB_CARDINALITIES=1000,10000,100000,1000000 \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

Trace stress (>1M spans):

DQB_ENGINE=trace \
DQB_MATRIX=A \
DQB_CARDINALITIES=2000000 \
DQB_QUERY_ITERATIONS=20 \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

Trace deep dive:

DQB_ENGINE=trace \
DQB_MATRIX=B \
test/integration/distributed/querybench/run-docker.sh --cpus 4 --memory 8g

DQB_SCENARIOS selects the scenario set. DQB_QUERY_WORKERS, DQB_WARMUP_ITERATIONS, DQB_WRITERS, and DQB_SMALL_EXACT_ROWS map to the corresponding knobs. Trace-specific knobs include DQB_SPANS_PER_TRACE, DQB_SPAN_DIST=uniform|heavytail, DQB_FILTER_SELECTIVITY, DQB_TRACE_ID_BATCH, DQB_SHARD_NUM, DQB_DATA_NODES, DQB_SPAN_BYTES, and DQB_QUERY_MEMORY_MIB.

DQB_QUERY_ITERATIONS (default 50) sets the timed-loop iteration count for most scenarios. The trace_by_id scenario is a sub-10ms point lookup, so at 50 iterations its p95/p99 are dominated by a few GC pauses and read as noise; it instead uses DQB_TRACE_BY_ID_ITERATIONS (default 1000) to report a stable tail. Lower it for a quick smoke (e.g. DQB_TRACE_BY_ID_ITERATIONS=20).

Trace DQB_MATRIX modes:

• A (default): primary volume sweep over DQB_CARDINALITIES, using the fixed realistic shape DQB_SPANS_PER_TRACE=20.
• B: fixed-volume deep dive at about 1M target spans, varying one axis at a time: trace size, heavy-tail distribution, selectivity, trace ID batch size, storage shard count, and data-node fan-out.
• both: runs Matrix A and then Matrix B sequentially inside the same Docker invocation.

Trace cardinality means target total spans. Uniform trace count is derived as round(total_spans / DQB_SPANS_PER_TRACE). Heavy-tail uses 95% traces at about 10 spans and 5% at about 500 spans, so the mean is about 34.5 spans/trace. service_id=svc-0 selectivity is assigned per trace so tag-filter result sizes are deterministic and reviewable.

Output layout

Reports default to .omx/bench-reports/distributed-query/:

• distributed-querybench.json — full per-result detail plus the environment and config view.
• distributed-querybench.md — per-mode summary table and the Vec/Row Ratios table (values < 1.00x mean vec is faster or lighter).
• shards/*.json — one per single-shot invocation, the raw evidence the merge pass aggregates.
• profiles/<scenario>/<cardinality>/.../<mode>/{cpu,heap}.pprof — when DQB_PROFILE=1. Trace profiles include the variant key in the path. Each pprof file was captured inside a fresh process so the heap snapshot is not polluted by other scenarios or modes.

Direct invocation contract

The test binary refuses to run when RUN_DISTRIBUTED_QUERY_BENCH=1 is set without DQB_IN_CONTAINER=1 -- host VM execution is not a supported benchmark path. Direct go test calls that bypass the orchestrator must set either:

• DQB_MODE=row|vec, DQB_SCENARIO=scan_all|top_with_filter, and DQB_CARDINALITY=<int> for a single-shot run, or
• DQB_MERGE=1 for a merge run.

Anything else is a hard configuration error so the orchestrator contract stays the only well-formed entry point. Host-runnable unit/config tests do not set RUN_DISTRIBUTED_QUERY_BENCH=1 and do not boot a cluster.

In-process cluster note

startBenchCluster runs the two data nodes and the liaison inside the same Go process. CPU, RSS, and allocation metrics in each shard are process-level deltas covering all three nodes plus the client. The report's environment.resource_note records this so consumers know the metrics are not per-node.

Documentation

Overview

Package querybench is the docker-gated distributed-query benchmark harness that drives row vs vec mode comparisons across scenarios and cardinalities.

Index

• type AllocationStats
• type Config
  • func LoadConfig() Config
  • func (c Config) IsSingleShot() bool
  • func (c Config) Validate() error
  • func (c Config) ValidateSoak() error
• type ConfigView
• type Environment
• type LatencyStats
• type Report
• type ResourceStats
• type Result
• type Scenario

Types

type AllocationStats

type AllocationStats struct {
	MetricSource       string  `json:"metric_source"`
	MallocsDelta       uint64  `json:"mallocs_delta"`
	TotalAllocDelta    uint64  `json:"total_alloc_delta"`
	MallocsPerQuery    float64 `json:"mallocs_per_query"`
	AllocBytesPerQuery float64 `json:"alloc_bytes_per_query"`
}

AllocationStats records allocation counters for the timed read phase.

type Config

type Config struct {
	ReportDir            string
	DockerImage          string
	CPULimit             string
	MemoryLimit          string
	Engine               string
	Matrix               string
	Mode                 string
	Scenario             Scenario
	SpanDist             string
	Cardinality          int
	SpansPerTrace        int
	TraceIDBatch         int
	ShardNum             int
	DataNodes            int
	SpanBytes            int
	QueryMemoryMiB       int
	QueryWorkers         int
	QueryIterations      int
	ByIDIterations       int
	WarmupIterations     int
	Writers              int
	SmallExactRows       int
	FilterSelectivity    float64
	RunBench             bool
	InContainer          bool
	Profile              bool
	Merge                bool
	Soak                 bool
	SoakHeapGrowthMaxPct int
}

Config drives a single test-binary invocation. The shell orchestrator (run-docker.sh -> orchestrate.sh) owns the (mode × scenario × cardinality) matrix; each invocation runs either:

• single-shot: one combo selected by Mode + Scenario + Cardinality, writes a shard JSON under ReportDir/shards/;
• merge: reads every shard, computes correctness, writes the unified report under ReportDir/.

Direct go test invocations without the right env vars surface a hard configuration error.

func LoadConfig

func LoadConfig() Config

LoadConfig reads benchmark settings from environment variables.

func (Config) IsSingleShot

func (c Config) IsSingleShot() bool

IsSingleShot reports whether the caller selected a single (mode, scenario, cardinality) combo. Returns true when any of the three singular env vars is set so Validate can flag a partially-set selection as a hard error.

func (Config) Validate

func (c Config) Validate() error

Validate enforces the single-shot OR merge contract.

func (Config) ValidateSoak

func (c Config) ValidateSoak() error

ValidateSoak enforces the DQB_SOAK contract: must run inside the container (DQB_IN_CONTAINER=1) and must target the trace engine.

type ConfigView

type ConfigView struct {
	ReportDir        string     `json:"report_dir"`
	Engine           string     `json:"engine"`
	Cardinalities    []int      `json:"cardinalities"`
	Scenarios        []Scenario `json:"scenarios"`
	QueryWorkers     int        `json:"query_workers"`
	QueryIterations  int        `json:"query_iterations"`
	WarmupIterations int        `json:"warmup_iterations"`
	Writers          int        `json:"writers"`
	Profile          bool       `json:"profile"`
}

ConfigView is a JSON-friendly copy of Config.

type Environment

type Environment struct {
	GoVersion    string `json:"go_version"`
	GOOS         string `json:"goos"`
	GOARCH       string `json:"goarch"`
	DockerImage  string `json:"docker_image,omitempty"`
	CPULimit     string `json:"cpu_limit,omitempty"`
	MemoryLimit  string `json:"memory_limit,omitempty"`
	Cgroup       string `json:"cgroup,omitempty"`
	ContainerID  string `json:"container_id,omitempty"`
	ResourceNote string `json:"resource_note,omitempty"`
	NumCPU       int    `json:"num_cpu"`
}

Environment records reproducibility metadata.

type LatencyStats

type LatencyStats struct {
	P50Ms  float64 `json:"p50_ms"`
	P90Ms  float64 `json:"p90_ms"`
	P95Ms  float64 `json:"p95_ms"`
	P99Ms  float64 `json:"p99_ms"`
	MaxMs  float64 `json:"max_ms"`
	MeanMs float64 `json:"mean_ms"`
}

LatencyStats contains latency percentiles in milliseconds.

type Report

type Report struct {
	GeneratedAt time.Time   `json:"generated_at"`
	Environment Environment `json:"environment"`
	Results     []Result    `json:"results"`
	Config      ConfigView  `json:"config"`
}

Report is the machine-readable benchmark output.

type ResourceStats

type ResourceStats struct {
	MetricSource    string  `json:"metric_source"`
	CPUSecondsDelta float64 `json:"cpu_seconds_delta,omitempty"`
	RSSBytes        uint64  `json:"rss_bytes,omitempty"`
	HeapAllocBytes  uint64  `json:"heap_alloc_bytes,omitempty"`
	HeapSysBytes    uint64  `json:"heap_sys_bytes,omitempty"`
	NumGC           uint32  `json:"num_gc,omitempty"`
}

ResourceStats records process-level resource deltas for the in-process cluster harness.

type Result

type Result struct {
	Profiles            map[string]string `json:"profiles,omitempty"`
	Scenario            Scenario          `json:"scenario"`
	SampleDataPointText string            `json:"sample_data_point_text,omitempty"`
	SampleTraceText     string            `json:"sample_trace_text,omitempty"`
	Error               string            `json:"error,omitempty"`
	Engine              string            `json:"engine"`
	SpanDist            string            `json:"span_dist,omitempty"`
	Mode                string            `json:"mode"`
	Correctness         string            `json:"correctness"`
	Allocations         AllocationStats   `json:"allocations"`
	Resources           ResourceStats     `json:"resources"`
	Latency             LatencyStats      `json:"latency"`
	PointsEach          int               `json:"points_each"`
	QPS                 float64           `json:"qps"`
	FilterSelectivity   float64           `json:"filter_selectivity,omitempty"`
	MeanSpansPerTrace   float64           `json:"mean_spans_per_trace,omitempty"`
	QueryWorkers        int               `json:"query_workers"`
	QueryIterations     int               `json:"query_iterations"`
	ResponseRows        int               `json:"response_rows"`
	ResponseTraces      int               `json:"response_traces,omitempty"`
	ResponseSpans       int               `json:"response_spans,omitempty"`
	Entities            int               `json:"entities"`
	TracesTotal         int               `json:"traces_total,omitempty"`
	SpansPerTrace       int               `json:"spans_per_trace,omitempty"`
	ServiceCardinality  int               `json:"service_cardinality,omitempty"`
	EndpointCardinality int               `json:"endpoint_cardinality,omitempty"`
	TraceIDBatch        int               `json:"trace_id_batch,omitempty"`
	ShardNum            int               `json:"shard_num,omitempty"`
	DataNodes           int               `json:"data_nodes,omitempty"`
	SpanBytes           int               `json:"span_bytes,omitempty"`
	QueryMemoryMiB      int               `json:"query_memory_mib,omitempty"`
	SegmentCount        int               `json:"segment_count,omitempty"`
	VecQueryCountDelta  int64             `json:"vec_query_count_delta,omitempty"`
	ApproxResultHash    uint64            `json:"approx_result_hash,omitempty"`
	Cardinality         int               `json:"cardinality"`
}

Result records one mode/scenario/cardinality benchmark outcome.

type Scenario

type Scenario string

Scenario identifies a distributed query benchmark shape.

const (
	// ScenarioScanAll benchmarks the measure scan-all fixture.
	ScenarioScanAll Scenario = "scan_all"
	// ScenarioTopWithFilter benchmarks the measure Top-N-with-filter fixture.
	ScenarioTopWithFilter Scenario = "top_with_filter"
	// ScenarioTraceByID benchmarks distributed trace-id lookup.
	ScenarioTraceByID Scenario = "trace_by_id"
	// ScenarioTraceTagFilter benchmarks distributed ordered trace tag filters.
	ScenarioTraceTagFilter Scenario = "trace_tag_filter"
)