README
¶
ParadeDB Benchmarker
Benchmark any database with k6. Real-time dashboard, consistent metrics, reproducible results.
Scripting Guide • Dashboard • Datasets • Data Loader • Docker • Contributing
A k6 extension for benchmarking databases with a unified API, real-time dashboard, and comprehensive data loading tools. While the included datasets focus on full-text search, the framework works for any query workload. You write the SQL or API calls, it handles timing, metrics, and visualization.
Compare performance across ParadeDB, PostgreSQL FTS, Elasticsearch, OpenSearch, ClickHouse, and MongoDB Atlas Search with consistent metrics and visualization. Docker Compose profiles are included for single-node benchmarking, but you can point at any database: local installs, remote servers, or cloud services.
We built this at ParadeDB to drive our iterative performance improvement process and power our benchmarks. It's very early, but we hope it can help others get their TTFB (time to first benchmark) down. We'd ❤️ your help improving it, PRs welcome!
How It Works
The benchmarker is built on Grafana k6, an amazing load testing tool written in Go. You write a k6 JavaScript script that defines scenarios and backends. Each scenario specifies an executor (how load is generated), a duration, a number of virtual users (VUs), and which functions to run, using which backends. k6 spins up VUs as concurrent goroutines, each looping over your test function for the duration of the scenario.
This project extends k6 with the k6/x/database module, adding backend drivers, automatic metrics, a real-time dashboard, and an export format on top.
Composing tests
A single script can compose multiple scenarios across multiple backends. You might run queries against ParadeDB for 30 seconds, then against Elasticsearch for 30 seconds, with a built-in phase timer staggering them so they don't compete for system resources. Within each phase you can layer different workloads: a full-text search at 200 QPS, an aggregation query at 100 QPS, and a 1,000 row/s ingest stream, all running concurrently. The framework times every operation, tags it with the backend name, and pushes metrics to the dashboard automatically.
While the framework exposes many backends, it's up to the user to write the queries to test (in JSON or SQL). A user can expect that the way the queries are run is optimal, but must still make sure the content of the queries is sane.
Ingest & update workloads
In addition to queries, you can run concurrent ingest (insert) and update workloads. The primary purpose is to put write pressure on the database while queries are running, measuring how query latency degrades under a realistic mixed workload. This is more meaningful than comparing raw ingest or update throughput across backends, since each database handles write consistency, indexing, and flush semantics differently.
Load strategies
k6 gives you several strategies for generating load:
constant-vus: fixed number of users hammering queries in a loop. The simplest way to measure throughput and latency.ramping-vus: gradually increases concurrency over stages, letting you find the point where latencies spike or errors appear.constant-arrival-rate: fixed number of requests per second regardless of response time. Useful for SLA testing or measuring ingest at a target throughput.
You can mix these freely across scenarios in the same script.
Dashboard
Results stream to a browser in real-time: latency percentiles (P50/P90/P95/P99), queries per second, ingest rate, and Docker container CPU/memory per backend. The dashboard also captures backend configs, setup scripts, and query patterns so results can be understood and reproduced later. Export as standalone HTML to share.
Quick Start
1. Build
make
2. Start backends
Each dataset under datasets/ ships with its own docker-compose.yml that pins
the exact images and tuning used for that benchmark. Run compose from the
dataset directory so the captured Container tab in the dashboard reflects the
real configuration:
docker compose -f datasets/sample/docker-compose.yml up -d
The repo-root docker-compose.yml is a kitchen-sink template containing every
supported backend with profiles; it's intended as a starting point when
authoring a new dataset, not for running an existing one.
For backends running off-host (e.g. AWS RDS, managed Elasticsearch), pass
{ type: "paradedb", container: "" } in the backend config to skip docker
metrics for that backend.
Please note the 'sample' dataset which is included does not provide a meaningful benchmark, it's designed to show how to use the system.
See Docker Setup for all available profiles and services.
3. Load data
./bin/loader load --backend paradedb ./datasets/sample
See Data Loader for advanced options like custom connection strings, parallel workers and S3 pulls.
4. Run a benchmark
./k6 run --out dashboard datasets/sample/k6/simple.js
Open http://localhost:5665/static/ to see real-time results. See Dashboard for export and replay options.
Writing Scripts
Scripts are standard k6 JavaScript with the k6/x/database extension. Here's a minimal example that benchmarks ParadeDB with 5 concurrent users for 30 seconds:
import db from "k6/x/database";
// Connect to the paradedb container using the standard settings from docker-compose
const backends = db.backends({ backends: ["paradedb"] });
// Open a file which has a termset we can use to customise queries
const terms = db.terms(open("./search_terms.json"));
// Define the scenarios to run
const scenarios = {
paradedb: {
executor: "constant-vus",
vus: 5,
duration: "30s",
exec: "paradedbQuery",
},
};
// Add docker based metric collection
export const collectMetrics = backends.addDockerMetricsCollector(
scenarios,
"35s",
);
export const options = { scenarios };
// Create the function which k6 will call on each iteration
export function paradedbQuery() {
// Activate the paradedb backend and run the query, cycling through the items in terms
backends
.get("paradedb")
.query(
`SELECT id, title FROM documents WHERE content ||| $1 LIMIT 10`,
terms.next(),
);
}
Each VU runs the paradedbQuery function in a loop for 30 seconds. The framework times every call, records the hit count, and pushes metrics to the dashboard. Swap "constant-vus" for "ramping-vus" to ramp load up and down, or "constant-arrival-rate" to send a fixed number of requests per second.
The Scripting Guide covers the full module API, backend configuration, multi-backend comparisons with phase timing, ingest/update benchmarks, and query examples for every supported backend.
Documentation
| Guide | Description |
|---|---|
| Scripting Guide | Module API, backend config, benchmark patterns, query reference |
| Dashboard | Real-time UI, metrics reference, export & replay |
| Datasets | Directory structure, schema format, pre/post scripts |
| Data Loader | CLI usage, connection strings, S3 pulls |
| Docker Setup | Compose profiles, service ports, TLS |
| Contributing | Adding backends, development setup, PR workflow |
License
MIT License - see LICENSE for details.
Documentation
¶
Overview ¶
Package search provides a k6 extension for benchmarking search backends.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Backends ¶
type Backends struct {
Metrics *metrics.Collector `js:"metrics"`
// contains filtered or unexported fields
}
Backends holds all configured backend clients.
func (*Backends) AddDockerMetricsCollector ¶
func (b *Backends) AddDockerMetricsCollector(call sobek.FunctionCall) sobek.Value
AddDockerMetricsCollector adds a metrics_collector scenario to the given scenarios object. Pass a Timer or a duration string (e.g. "500s"). Returns a function that the script should export as collectMetrics:
export const collectMetrics = backends.addDockerMetricsCollector(scenarios, timer);
func (*Backends) Close ¶
func (b *Backends) Close()
Close closes all backend connections. Use this at the end of a test or between groups to clean up connections.
func (*Backends) Collect ¶
Collect collects metrics from all enabled containers. Includes a 500ms sleep to avoid polling too frequently.
func (*Backends) SetTimeout ¶
SetTimeout sets the query timeout for all backends in seconds. Use 0 to disable timeout (default).
type ModuleInstance ¶
type ModuleInstance struct {
// contains filtered or unexported fields
}
ModuleInstance represents an instance of the module for a single VU.
func (*ModuleInstance) Exports ¶
func (m *ModuleInstance) Exports() modules.Exports
Exports returns the exports of the module.
type RootModule ¶
type RootModule struct{}
RootModule is the global module instance that will create client instances for each VU.
func (*RootModule) NewModuleInstance ¶
func (*RootModule) NewModuleInstance(vu modules.VU) modules.Instance
NewModuleInstance creates a new instance of the module for each VU.
type Terms ¶
type Terms struct {
// contains filtered or unexported fields
}
Terms provides sequential and random access to a list of search terms.
db.terms(JSON.parse(open("./terms.json")))
db.terms(allTerms.filter(t => !t.includes(" ")))
type Timer ¶
type Timer struct {
// contains filtered or unexported fields
}
Timer manages scenario timing for sequential benchmark phases. Create with search.timer({ duration: "30s", gap: "2s" }).
func (*Timer) AdvanceAndGet ¶
AdvanceAndGet advances to the next phase and returns the startTime string.
func (*Timer) Get ¶
Get returns the current phase startTime without advancing. Use for parallel scenarios that share a phase with the preceding advanceAndGet. Auto-advances on first call if no phase has been started.
func (*Timer) TotalDuration ¶
TotalDuration returns the total covering duration as a string (e.g. "70s").
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package backends provides the driver interface and shared infrastructure for search backends.
|
Package backends provides the driver interface and shared infrastructure for search backends. |
|
clickhouse
Package clickhouse provides the ClickHouse driver implementation.
|
Package clickhouse provides the ClickHouse driver implementation. |
|
elasticsearch
Package elasticsearch registers the Elasticsearch backend.
|
Package elasticsearch registers the Elasticsearch backend. |
|
mongodb
Package mongodb provides the MongoDB driver implementation.
|
Package mongodb provides the MongoDB driver implementation. |
|
opensearch
Package opensearch registers the OpenSearch backend.
|
Package opensearch registers the OpenSearch backend. |
|
paradedb
Package paradedb registers the ParadeDB backend.
|
Package paradedb registers the ParadeDB backend. |
|
postgres
Package postgres registers the PostgreSQL backend.
|
Package postgres registers the PostgreSQL backend. |
|
shared/elasticsearch
Package elasticsearch provides the shared Elasticsearch/OpenSearch driver implementation.
|
Package elasticsearch provides the shared Elasticsearch/OpenSearch driver implementation. |
|
shared/postgres
Package postgres provides the shared PostgreSQL driver implementation.
|
Package postgres provides the shared PostgreSQL driver implementation. |
|
cmd
|
|
|
dashboard-viewer
command
Command dashboard-viewer serves saved k6-search dashboard JSON files.
|
Command dashboard-viewer serves saved k6-search dashboard JSON files. |
|
loader
command
Loader CLI for bulk loading data into search backends.
|
Loader CLI for bulk loading data into search backends. |
|
Package dashboard provides a web-based dashboard for k6 search benchmarks.
|
Package dashboard provides a web-based dashboard for k6 search benchmarks. |
|
Package loader provides data loading functionality for k6 benchmarks.
|
Package loader provides data loading functionality for k6 benchmarks. |
|
Package metrics provides container metrics collection via Docker API.
|
Package metrics provides container metrics collection via Docker API. |