llm-test-studio

README

English · 简体中文

A local-first acceptance and benchmarking studio for LLM providers, gateways, and deployments.
Run the same versioned tests across every channel, measure compatibility and performance, and keep the evidence behind every release decision.

Wails + React desktop · Go CLI · SQLite · OS credential store

[!NOTE] LLM Test Studio is publicly available under Apache-2.0. See the compatibility and data-upgrade policy before changing released contracts or upgrading stored data.

New in v0.2.0: dataset evaluation, dataset download/cache management, and historical report comparison.

Product proof

_{Real Windows desktop build · revision-aware case catalog · no concept mockup or generated UI}

The case catalog is the center of the product, not a setup screen hidden behind a run. Cases remain inspectable, revisioned assets that can be assembled into suites and plans, executed against different provider or gateway endpoints, and traced into reports.

Accept a channel before production

A successful HTTP response does not prove that an LLM endpoint is ready to carry production traffic. The same logical model can behave differently across providers, gateways, regions, or private deployments. LLM Test Studio turns that uncertainty into a repeatable acceptance decision:

• Protocol compatibility — verify request parameters, streaming events, tool calls, multimodal inputs, usage fields, and error behavior.
• Capability truth — prove that an advertised model or endpoint actually supports the behavior your application depends on.
• Performance under load — measure latency, token timing, throughput, queue delay, and scheduler pressure with single, fixed-concurrency, or open-loop traffic.
• Operational reliability — separate transport, protocol, semantic, timeout, and SLA failures instead of reducing them to one success rate.
• Auditable evidence — pin the exact case, target, load profile, SLA, and environment behind each conclusion.

Gateways route production traffic. LLM Test Studio validates those routes before launch and after a provider, model, gateway, or deployment changes.

Why case-first testing

Most LLM tests begin as a prompt, a script, or a one-off dashboard run. That is useful for exploration, but difficult to reuse or audit later. LLM Test Studio treats the test definition as a durable asset:

• Cases outlive runs — requests, expected behavior, and assertions live in portable case.json files.
• Intent stays separate from infrastructure — a logical model maps to the upstream model name used by each channel, so the same case can exercise different providers or gateways.
• Every run is reproducible — the plan, model, channel, case revisions, load profile, SLA, and environment are pinned into an immutable snapshot.
• Conclusions keep their evidence — request results distinguish transport, protocol, semantic, and SLA failures before producing one sealed report model.
• Desktop and automation agree — the Wails application and script-friendly CLI use the same Go Application Core and domain rules.

From channel acceptance to evidence

Versioned Test Case ──> Suite ──> Plan ──> Immutable Run Snapshot
                           targets + load + SLA          │
                                                          ▼
                                           Results + Evidence ──> Report

The reusable boundary is the exact case revision. Suites group Case references; plans define their ordered execution, inputs, load, timeouts and SLA thresholds; runs resolve current revisions, bind the selected channel/model, and freeze the complete input before execution. This keeps test intent independent from the provider-specific endpoint that happens to serve the model.

Quick start

Use a desktop release

Check GitHub Releases for available builds. Tagged releases are configured for Windows amd64 installers and archives, plus macOS universal DMG and ZIP packages. Linux does not currently have a packaged desktop release.

Pushing a v* version tag runs the shared CI workflow against that tagged commit. Go checks, frontend checks, and the full-history secret scan must all pass before packaging and GitHub Release publication can start.

Build the desktop application

Requirements: Go 1.26+ with the Go 1.26.10 toolchain, Node.js 24, pnpm 10.30.3, and Wails v2.15.0.

go install github.com/wailsapp/wails/v2/cmd/wails@v2.15.0
wails version

cd apps/desktop/frontend
pnpm install --frozen-lockfile
pnpm test
pnpm build

cd ..
wails build -nosyncgomod

Use wails dev -nosyncgomod instead of wails build -nosyncgomod for desktop development. On first start, structured application data is created below the operating system's user configuration directory.

Build the CLI

go build -o llm-test-studio ./cmd/llm-test-studio
./llm-test-studio doctor --format human

On Windows PowerShell, run the binary as .\llm-test-studio.exe.

Current protocol runtime

Desktop and CLI audit execution share the protocol runtime for openai-chat, openai-responses, anthropic-messages, seedance, wan-video, and minimax-video. Case files use schema 2 with an explicit stable ID and protocol-keyed definitions; each definition owns its native inputs, body, workflow and assertions. A Suite selects one protocol and uses schema 1 with ordered Case references and input bindings. Plans retain per-entry load, warmup, settings and seed. The Run binds one model, channel and credential.

CLI audit run --seed 42 --inputs '{"prompt":"Hello"}' accepts only inputs declared by selected Cases. Omit --inputs to use their declared defaults. --all-cases selects enabled automatic Cases; manual/disabled selections fail explicitly. --dry-run prepares requests without network calls or verification claims. Waiting is authored as workflow.mode, replacing --no-wait.

The built-in catalog contains 1,099 Cases, 84 Suites and 32 Plans. The K3 Responses foundation Suite contains 230 enabled automatic Cases; see the Kimi catalog for applicability and coverage gaps. Models and credentials belong to Run binding. Empty assertions produce observation-only results and do not inflate verification pass rates. See multi-protocol Cases and native text APIs for format changes and historical-data impact.

CLI examples

List the built-in OpenAI-compatible cases (including Kimi models):

go run ./cmd/llm-test-studio audit list \
  --suite openai-chat \
  --cases-root data/cases \
  --format human

Run one compatibility case in Bash or another POSIX shell:

API_AUDIT_API_KEY='replace-me' go run ./cmd/llm-test-studio audit run \
  --suite openai-chat \
  --cases-root data/cases \
  --base-url https://gateway.example/v1 \
  --model kimi-k3 \
  --case must.usage_non_stream \
  --format human

Run a fixed-concurrency load test:

LOADTEST_API_KEY='replace-me' \
LOADTEST_URL='https://gateway.example/v1/chat/completions' \
LOADTEST_MODEL='kimi-k3' \
go run ./cmd/llm-test-studio load run \
  --requests 100 \
  --concurrency 10 \
  --stream \
  --input-tokens 100 \
  --max-tokens 10 \
  --output load-result.json

Use --rate with --duration for open-loop scheduling. Use --request-file data/cases/<group>/<case>/case.json to load a case request body. Both http:// and https:// endpoints are supported; use HTTPS whenever transport confidentiality is required.

Avoid passing credentials directly as command arguments because shells may retain them in history. Windows PowerShell users can set the same variables with $env:VARIABLE_NAME = 'value' before running a command.

What you can run

• Smoke-test a URL, model, or saved channel without first building a full catalog.
• Create and maintain built-in or user-authored test cases.
• Group exact case revisions into reusable suites and plans.
• Run single-request, fixed-concurrency, or open-loop tests with request timeouts and SLA thresholds.
• Exercise OpenAI Chat, OpenAI Responses, Anthropic Messages, Kimi, GLM, Seedance, Wan, and MiniMax compatibility cases with native Go execution engines.
• Compare one logical model across two or more configured channels.
• Run local dataset evaluations with Inspect AI, manage downloads and caches, and inspect per-sample scores and reports. See the data testing guide.
• Compare 2–8 archived functional or performance reports with a selectable baseline, difference filtering, and visual exports. See report comparison.
• Inspect run history, request-level results, latency and token metrics, failure dimensions, SLA conclusions, and environment snapshots.
• Export the same report as JSON, standalone HTML, PNG, or PDF.
• Run compatibility audits and load tests from the CLI with stable JSON, JSONL, or human-readable output.

Built-in case library

The repository currently includes 1,099 portable Cases, counted by authoring directory and unique Case ID:

Case directory	Cases	Focus
openai-chat	499	Chat compatibility, Kimi and GLM contracts, streaming, tools, reasoning, multimodal inputs, and usage
openai-responses	208	K3 native Responses parameters, tools, reasoning, structured output, caching, and rejection boundaries
kimi-cache-usage	20	Separate cache token-usage and response-header conformance checks across three text protocols
kimi-protocol	4	Response structure, streaming integrity, reasoning output, and usage consistency
seedance	6	Text, image, video, and mixed-reference video requests
wan-video	213	Wan video creation, task polling, model-specific boundaries, and media fixture templates
minimax-video	149	MiniMax H3 creation, polling, parameter rejection, and media-dependent scenarios
Total	1,099	784 enabled automatic Cases, 107 enabled manual Cases, and 208 disabled Cases

Kimi Chat Cases are included in openai-chat. Some Cases in that directory also define native openai-responses or anthropic-messages behavior under the same ID; these definitions are not counted again. The 84 Suites cover all six runtime protocols. The 32 built-in Plans include 18 performance presets (six per text protocol: quick check, baseline, sustained load, capacity steps, uncached input, and cached suffix), three input-token staircase Plans, and 11 functional or mixed Plans.

Built-in Cases are starting points, not a claim that every Case applies to every model. Counts include manual scenarios and disabled templates; only enabled automatic Cases can be selected for automated execution. Protocol compatibility is validated before a Plan can run.

User cases are stored as data/cases/<group>/<case>/case.json beside the desktop executable. They can be copied, reviewed, versioned, and shared without including API keys. A user case overrides a built-in case with the same identity when the two sources are merged.

Authored data uses the following layout. Source paths are relative to the repository root; desktop paths are relative to the executable, including apps/desktop/build/bin during Wails development and local builds.

Data	Source / desktop path
Cases	data/cases/<group>/<case>/case.json
Suites	data/suites/<group>/<suite>/suite.json
Model profile fixtures (source only)	data/definitions/models/*.json
User models	data/models.json
Channels and model mappings	data/channels.json
Built-in performance Plans (source, embedded)	data/performanceplans/*.json
User Plans	data/plans/<id>.json

Built-in Cases, Suites and performance Plans are embedded from data/ into the executable; a fresh release needs no external catalog copy. Desktop edits and catalog lock files are written under the executable's data/ directory. CLI commands default --cases-root to data/cases relative to the working directory; use an explicit path when launching elsewhere.

For an existing installation, close the app and move its adjacent cases/, suites/, plans/, models.json, and channels.json into data/, including revision files and lock sidecars. Keep the executable in the same installation directory so existing keyring credentials remain accessible. The app reads only the new paths. SQLite run history, reports, and logs retain their existing user-configuration locations.

Domain model

Concept	Meaning
Model	The logical model being evaluated, independent of a provider-specific model name.
Channel	A callable service path with a Base URL, protocol, enabled state, and credential reference.
Channel Model	The upstream model name used for a logical model on a specific channel.
Test Case	A versioned request, expected outcome, and assertion set.
Suite	A reusable collection of exact test case revisions.
Plan	Test intent: cases, candidate targets, load profile, and SLA thresholds.
Run	One execution with an immutable input snapshot.
Report	The validated conclusion shared by JSON, HTML, PNG, and PDF exports.

Architecture

Wails + React desktop ──┐
                        ├── Go Application Core ──┬── execution engines
llm-test-studio CLI ────┘                         ├── SQLite repository
                                                  ├── OS credential store
                                                  ├── filesystem authored catalog
                                                  └── report renderer

The desktop and CLI share the same application services and domain rules. There is no Python or Streamlit runtime.

Repository map

• apps/desktop — Wails bindings and the React/Tailwind desktop interface.
• cmd/llm-test-studio — versioned CLI commands and JSON/JSONL output.
• internal/application — catalog, run, comparison, reporting, and workspace orchestration.
• internal/execution and engine — load and compatibility execution engines.
• internal/persistence/sqlite — operational schema and runtime-evidence repositories.
• data/cases — embedded, shareable cases grouped in authoring folders.
• data/suites — embedded suite definitions and scenario manifests.
• data/definitions — non-secret model definition fixtures.

Security and local data

• Test definitions reject credential-bearing headers, credential fields, and credential-like values.
• Channel keys are stored through the operating-system credential service. Updating a key creates a new credential revision.
• Persistent application data contains credential references, masked suffixes, and fingerprints, not plaintext secrets.
• Reports and results contain measurements and redacted evidence metadata, not authorization headers or credential bytes.
• Model, Channel, Channel Model, Test Case, Suite, and Plan definitions are file-backed. SQLite stores only operational data: immutable Run snapshots, Results, Evidence, Comparisons, performance reports, and sealed Reports.
• Plain HTTP endpoints are rejected except for explicitly enabled loopback testing.

Current scope

• LLM Test Studio calls configured endpoints to validate them; it is not a production traffic gateway, router, or provider SDK.
• The product focuses on endpoint acceptance and comparative evidence, not on maintaining a public model leaderboard or profiling inference hardware below the API boundary.
• Historical report comparison accepts 2–8 functional or performance reports. Dataset evaluations use their own report reader and are not included in that matrix.
• Dataset evaluation runs locally and requires uv plus preparation of the pinned Python environment. Its 24 entries include years and variants; they are not 24 unrelated benchmark families.
• Case reuse is protocol-aware. A case cannot be attached to an incompatible model or channel.
• Windows amd64 and macOS universal are the configured packaged desktop targets. Other platforms can build from source where Wails supports them.

Verify a source checkout

go test ./... -count=1
go vet ./...

cd apps/desktop/frontend
pnpm test
pnpm build

The legacy Bash benchmark remains as an independent load-engine acceptance fixture. Its mock LLM is implemented in Go:

bash scripts/test_llm_benchmark.sh

License

LLM Test Studio source code, built-in case definitions, and project-owned media fixtures are available under the Apache License 2.0. See NOTICE, third-party notices, and the case and media provenance statement for attribution and scope details.

Open-source status

LLM Test Studio is publicly released. Contributions must preserve supported user-facing contracts and provide application-shipped migrations for routine data upgrades; private implementation details can evolve with their callers. SQLite upgrades now cover the ten most recent distinct storage structures identified in code history, with verified backups and historical fixtures. The compatibility and data-upgrade policy lists exact source commits, read-only historical job limitations, archive export commands, and remaining release verification. This does not establish that any actual user installation has been upgraded.

Check each release's notes for available packages, supported upgrade sources, and signing/notarization status. External media fixture stability and binary signing/notarization remain separate verification items; public source availability does not establish their completion.