qualflare

README

qualflare-go

A native Go test reporter for Qualflare — captures results directly from go test: statuses Go's JSON stream can express but nobody reads, subtests as first-class cases, nested steps, attachments, and author-facing metadata (labels, links, tags, priority, custom parameters).

Without it, Go results reach Qualflare through qualflare-cli's go test -json parser, which records a status, a duration and a name — no steps, no attachments, no metadata, no flakiness, and every non-terminal outcome flattened into error.

The reporter makes no network calls. It writes a report directory, and qualflare-cli uploads it — which is what lets any number of sharded CI jobs merge into a single Launch.

Why this one is not a plugin

Every other Qualflare reporter hooks into its framework. Go has no reporter plugin API and no per-test hook, and TestMain receives only an exit code from m.Run() — so a library alone can never see every test, only the ones that opted in. Complete results have to come from go test -json.

That splits the package in two, and the split explains most of what follows:

• The binary consumes the event stream and writes the report. It is the only part that can see every test.
• The library you import in your tests adds metadata to them. It cannot report a result, and it never tries to.

Install

go install github.com/Qualflare/qualflare-go/cmd/qualflare-go@latest

Or from Homebrew, which is easier to pin in a CI image without a Go toolchain:

brew install qualflare/tap/qualflare-go

Signed binaries for linux, macOS and windows on amd64 and arm64 are attached to each release, with a cosign signature over checksums.txt.

For the metadata API, add the module to your own project:

go get github.com/Qualflare/qualflare-go

Requires Go 1.21+. The module has zero external dependencies, so importing it adds nothing to your go.mod graph — that is enforced by a test, not a promise.

Quickstart

qualflare-go ./...
npm install -g @qualflare/cli
qf login my-project "$QUALFLARE_TOKEN" --force
qf my-project collect ./qualflare-results

qualflare-go ./... is shorthand for qualflare-go -- go test ./...; it adds -json for you. Pass any flags go test accepts:

qualflare-go -- go test -race -timeout 5m ./...

Prefer the wrapper over a pipe. go test -json ./... | qualflare-go works, but a pipe cannot see two things that matter:

• The exit code. Without set -o pipefail a shell reports the pipe's success, not go test's.
• stderr. On Go 1.21 and 1.23 a build failure produces no fail event at all — the JSON stream is entirely green while go test exits 1, and the compiler error goes only to stderr. Measured, not theorised.

The wrapper sees both, so it can fail loudly where a pipe would upload a green launch for a broken build. In pipe mode the reporter says so rather than pretending.

Subtests are cases

TestUsers/rejects_a_blank_email becomes its own case, not a step inside TestUsers. Table-driven tests are the Go idiom, and folding a 200-row table into one case would destroy exactly the per-row history and flakiness signal that makes the data worth having. Steps stay reserved for what you mark explicitly.

Parallel runs and sharding

Point every shard at the same --output-dir and collect once at the end. Each invocation writes its own uniquely-named file, so shards never overwrite each other, and qf collect merges every file in the directory into one Launch.

Shards must share a run id. In CI it is derived from the build automatically, so they agree without coordinating; otherwise pass --run-id. Use --shard-index to attribute cases to a worker — Go has no equivalent of PYTEST_XDIST_WORKER.

Enriching your tests

import "github.com/Qualflare/qualflare-go"

func TestCheckout(t *testing.T) {
    qualflare.Label(t, "feature", "checkout")
    qualflare.Tag(t, "smoke")
    qualflare.Link(t, "https://example.com/issue/42", qualflare.LinkIssue, "QF-42")
    qualflare.Priority(t, qualflare.PriorityHigh)

    qualflare.Step(t, "add to cart", func() {
        qualflare.Parameter(t, "sku", "widget")
        qualflare.MaskedParameter(t, "token")
    })
}

Every call takes a testing.TB first. That is not ceremony: Go has no goroutine-local storage, so the explicit t is the only reliable handle on which test is running — and it makes metadata outside a test a compile error rather than a runtime rule, because there is no t to pass.

Nothing in the API can fail your test. No function returns an error, none panics, and calls are inert unless a reporter is actually listening — so go test -v stays clean.

MaskedParameter takes no value at all. masked is a display hint the server does not act on, so withholding the value here is the only thing that actually keeps a secret out of the report; a signature that cannot accept one cannot leak one.

Full reference in docs/METADATA-API.md.

Configuration

Precedence, highest first: flag → QUALFLARE_* environment → CI/git detection → default. Every option and variable is in docs/CONFIGURATION.md.

There is deliberately no config file and no token option. The reporter makes no network calls, so it has no credential; qf login holds it.

Test reports

This reporter is tested with itself. e2e/ is a Go suite covering this package's own behaviour — the metadata API, nested steps, subtests as cases, attachments and the step-cap regression — run by this reporter and uploaded to Qualflare on every merge to main by the published qualflare-cli. The results below are that suite's, reported through the code this README documents:

Every case there is meant to pass, so a red run is a real regression rather than a fixture failing on purpose. The deliberately awkward cases — panics, timeouts, os.Exit, a package that does not compile — live in test/integration/fixtures/, which is never uploaded.

Known limitations

• Go has no retry concept. retryCount, isFlaky and attempts are populated only under -count=N, where the same test genuinely runs more than once. Flakiness is never inferred across separate go test invocations — a guessed flake is worse than a missing one.
• Build failures are invisible to the stream before Go 1.24. They are caught by the wrapper's exit-code check instead, which is why pipe mode is the weaker mode.
• Metadata from a stray goroutine can be misattributed. After a test completes, t.Log reattributes to the closest incomplete parent rather than failing, and that is not something a library can correct.

Full details in docs/LIMITATIONS.md.

Development

go test ./...                  # unit
go test -race ./...
go vet ./...
go build -o /tmp/qualflare-go ./cmd/qualflare-go

# the dogfood, then the check that runs before any upload
cd e2e && /tmp/qualflare-go --output-dir e2e-results -- go test -count=1 ./...
QUALFLARE_OUTPUT_DIR=e2e-results go run ./verify

test/integration/fixtures/awkward is a separate module of deliberately difficult packages — a parent that fails while its subtests pass, a panicking subtest, a goroutine panic, a timeout, os.Exit(3), a package with no test files, eight parallel tests logging 100 KB each. The unit tests run against streams captured from really running it, so a diff in a capture is itself the alarm that a Go release changed the output shape.

Related reporters

Qualflare has a native reporter for each framework, all writing the same report format so one qf collect handles a polyglot repository:

jest · vitest · mocha · cypress · playwright · cucumberjs · pytest · cli

License

Apache-2.0 — see LICENSE.

Documentation

Overview

Package qualflare is the author-facing metadata API for the Qualflare Go reporter.

func TestCheckout(t *testing.T) {
    qualflare.Label(t, "feature", "checkout")
    qualflare.Tag(t, "smoke")
    qualflare.Step(t, "add to cart", func() {
        qualflare.Parameter(t, "sku", "widget")
    })
}

Every call takes a testing.TB. Go has no goroutine-local storage, so the explicit tb is the only reliable handle on which test is running -- and it dissolves a whole class of problem the sibling reporters had to engineer around: metadata emitted outside a test is a COMPILE error here, not a runtime rule, because there is no tb to pass.

Nothing in this package can fail a test. Every function returns nothing, and the one call into testing.TB is guarded. A metadata problem must never be the reason somebody's suite goes red.

The results themselves do not come from here. Go has no per-test hook and TestMain sees only an exit code, so a library alone can never observe every test; the qualflare-go binary reads `go test -json` for that. This package only enriches.

Index

• Constants
• func Attach(tb testing.TB, name string, data []byte, mimeType string)
• func AttachText(tb testing.TB, name, text, mimeType string)
• func Description(tb testing.TB, text string)
• func Enabled() bool
• func Label(tb testing.TB, name, value string)
• func Link(tb testing.TB, url, linkType, name string)
• func MaskedParameter(tb testing.TB, name string)
• func Parameter(tb testing.TB, name, value string)
• func Priority(tb testing.TB, level string)
• func Step(tb testing.TB, name string, fn func())
• func Tag(tb testing.TB, tags ...string)

Constants

const (
	EnvEnable = "QUALFLARE_GO"
	EnvSpill  = "QUALFLARE_GO_SPILL"
)

Environment variables the library reads.

const (
	PriorityLow      = "low"
	PriorityMedium   = "medium"
	PriorityHigh     = "high"
	PriorityCritical = "critical"
)

Priority levels, matching the wire contract's vocabulary.

const (
	LinkIssue  = "issue"
	LinkTMS    = "tms"
	LinkCustom = "custom"
)

Link types. An unrecognised type is passed through and rejected server-side rather than silently rewritten here.

Functions

func Attach

func Attach(tb testing.TB, name string, data []byte, mimeType string)

Attach records in-memory bytes against the current test.

The payload travels inline, base64-encoded, inside a single log line, so it is bounded: anything that will not fit is dropped with a warning rather than truncated, because half a base64 payload is not a usable file. For large artifacts, write the file and reference it from your own storage.

func AttachText

func AttachText(tb testing.TB, name, text, mimeType string)

AttachText records a string. The mime type defaults to text/plain.

func Description

func Description(tb testing.TB, text string)

Description attaches prose describing what the test covers.

func Enabled

func Enabled() bool

Enabled reports whether anything is listening.

Metadata rides t.Log, so emitting when no reporter is consuming the stream would put encoded noise in a developer's `go test -v` output for no benefit. The gate is `go test -json`, which passes -test.v=test2json -- measured on Go 1.21, 1.23, 1.25 and 1.26, so it holds at this module's floor. Plain `go test -v` sets "true" and is not enough.

QUALFLARE_GO=1 forces emission on (the wrapper sets it, which is also what makes it work on a toolchain whose -json does not imply test2json), and QUALFLARE_GO=0 forces it off.

func Label

func Label(tb testing.TB, name, value string)

Label attaches Allure-style name/value metadata (epic, feature, story, owner, ...).

func Link

func Link(tb testing.TB, url, linkType, name string)

Link attaches an external link. linkType is one of LinkIssue, LinkTMS or LinkCustom; empty defaults to custom.

func MaskedParameter

func MaskedParameter(tb testing.TB, name string)

MaskedParameter records that a parameter exists without recording its value.

It deliberately takes no value at all. The wire contract treats `masked` as a display hint and the server does NOT redact, so withholding the value here is the only thing that actually keeps a secret out of the report -- and a signature that cannot accept one cannot leak one.

func Parameter

func Parameter(tb testing.TB, name, value string)

Parameter records a case- or step-level parameter. A parameter emitted while a Step is open belongs to that step.

func Priority

func Priority(tb testing.TB, level string)

Priority marks the test's importance.

func Step

func Step(tb testing.TB, name string, fn func())

Step records a named step around fn.

Closure form only, and that is a correctness decision rather than a taste one. A begin/end pair would leak an unclosed step on t.Fatal, which calls runtime.Goexit -- and Goexit DOES run deferred functions, so a defer closes the step correctly where an explicit End() would simply never be reached.

The user's panic is re-raised unchanged after the step is recorded. Nothing here alters control flow: a reporter that swallowed a panic would turn a failing test green, which is the worst thing it could possibly do.

func Tag

func Tag(tb testing.TB, tags ...string)

Tag attaches one or more free-form tags.