llmkit

README

llmkit

Shared LLM plumbing for Go services: a vendor-neutral provider contract, strict JSON-schema generation, tolerant response parsing, prompt/schema version tags, and a resumable harness for comparing models.

It was extracted from three services that had each grown their own copy of this code and then drifted apart — one had a JSON extractor that broke on a { inside prose, another had a version hash that ignored half its own prompt template. Improving one and hand-porting to the others stopped being sustainable.

go get github.com/sam0sva1/llmkit

Requires Go 1.25.

Packages

Package	What it does
provider	The request/response contract. No vendor names appear here.
provider/openrouter	The only package that names a vendor.
parse	Pull JSON out of a model's answer — markdown fences, surrounding prose, decoy braces.
schema	Generate a strict-mode-safe JSON schema from a Go type, and check that it is one.
promptver	Short, collision-resistant version tags for prompts and schemas.
model	Which model to call (Ref), and separately what it costs (Priced).
run	The one call sequence: call, decode into T, optionally fall back to a second model.
evalharness	Run cases against several models, resumably, and report the comparison.

One call

client := openrouter.New(openrouter.Config{APIKey: os.Getenv("LLM_API_KEY")})

req := myModel.Apply(provider.Request{
    Messages: []provider.Message{
        provider.System("You are a careful language tutor."),
        provider.UserText(prompt),
    },
    ResponseSchema: schema.Response[Analysis]("analysis", ""),
})

analysis, raw, err := run.Run[Analysis](ctx, client, req)

raw carries what belongs in your database next to the result: which model actually answered, the token usage, the finish reason.

Complete, compiling versions of this and everything below are in example_test.go — including a multimodal request and a full evalharness.Suite. It is a test file on purpose: the compiler checks the documented usage, where a README snippet drifts silently.

Where to look for what:

Question	Read
What does this do, what will bite me	this file
How do I actually call it	example_test.go — it compiles
How do I wire it into an existing project	CONSUMING.md
What belongs in the library, how to test it	RULES.md
How to change and publish the library itself	RELEASING.md

Things worth knowing before you use it

Never put omitempty on a structured-output type. invopop/jsonschema omits such fields from required, and strict mode rejects a schema whose properties are not all required. Express optionality with a pointer. Then assert it:

if missing := schema.MissingRequired(schema.Generate[Analysis]()); len(missing) != 0 {
    t.Errorf("not required, strict mode will reject: %v", missing)
}

Without that test the mistake surfaces as a provider error that does not name the field.

A nil ResponseSchema is an error, not a plain call. run.Run refuses before touching the network. Sending no schema would leave the output unconstrained while every call site still looked correct — a much more expensive way to find out.

On a reasoning model, set Request.Reasoning on purpose. Left at the provider default, hidden reasoning was 72% to 83% of every response's completion tokens in one measured case, and swung between 516 and 1448 tokens across byte-identical requests. It is billed as output, and the model is not answering while it spends it, so that spread is both the bill and the latency tail.

Effort, MaxTokens and Enabled are three spellings of one setting and are mutually exclusive — providers disagree about which wins, so a request carrying two of them is rejected before it is sent. Exclude is orthogonal and may accompany any of them. Levels are the router's seven, none through max; MaxTokens is what Anthropic, Gemini and Qwen models take instead.

Reasoning: &provider.Reasoning{Effort: provider.ReasoningEffortMinimal},

minimal is the right level when the prompt already carries the reasoning in the open — a schema with explicit reasoning fields, for instance — because a hidden chain of thought then repeats work already being done where you can see it.

Response.Provider is how you know ProviderOrder was honoured. The request body only proves what was asked for. A router that accepted provider.order and then served whoever it liked would look identical from inside the process, so the serving upstream is surfaced and there is a live test comparing two of them.

Verified against the live router, including all four pins the consumers use in production: asking for one upstream and then another returned each in turn, and AllowFallbacks: false against an upstream that cannot serve the model returns a 404 rather than a silent substitution.

Compare it case-insensitively. A consumer asking for "openai" is served by "OpenAI" — the router matches names loosely and reports them canonically, so an equality check would report a mismatch that is not one.

The reasoning budget is sent as OpenRouter's reasoning object, whatever BaseURL says. Official OpenAI spells the same idea as a top-level reasoning_effort, so an endpoint that validates its request body strictly would reject this one rather than ignore the extra key. Pointing BaseURL elsewhere and setting Reasoning is the one combination to verify with a real call before trusting — the same caveat already applies to ProviderOrder.

Usage.ReasoningTokens is part of CompletionTokens, not an addition to it. Providers bill reasoning as output and report this only as a breakdown, so summing the two double counts. Zero means the provider sent no breakdown, which is indistinguishable from a model that did no reasoning. The reasoning text itself arrives in Response.Reasoning, empty when Exclude was set.

RunWithFallback costs one call in the common case. The second model is tried only when the first outright fails, and skipped entirely when the context is already dead, since the retry would be doomed too.

Build a suite's client with MaxRetries: provider.Ptr(0). SDK-level retries multiply with the suite's own Attempts — 3 × 3 is up to nine HTTP calls per job — and they distort the comparison: a model the SDK quietly retried looks more reliable than it is.

Build SuiteFingerprint with promptver.Hash, and put the schema in it as marshalled JSON. There is no helper for this on purpose — which parts matter is domain knowledge — but the shape that works is: every rendered prompt in every language, each response schema as json.Marshaled bytes, and a literal string naming the sampling parameters. Sort the parts before hashing so a map iteration cannot change the fingerprint between runs.

SuiteFingerprint must cover your sampling parameters — temperature, max_tokens and the reasoning budget. The reasoning budget is the one most worth naming: an effort level changes the answer more than a temperature nudge does, so a run that adds Reasoning without touching the fingerprint reuses paid results produced without it and calls that a comparison.

SuiteFingerprint must cover your sampling parameters, not just the prompt and schema. The library never sees the Request — your Run function builds it — and the resumability key has to exist before the call in order to decide whether to skip it. So temperature, token caps and the reasoning budget are yours to fold in, exactly like the prompt text.

There is a safety net, but it is a net and not the rule: the harness records the sampling parameters that actually went out and compares them on the next run, stopping after one call if they changed while SuiteFingerprint did not. It cannot fire on a run where every job was already cached — there is no call to observe — so it catches the mistake rather than preventing it. Fold them in.

The reported cost includes failed attempts and the judging pass. Result.Usage and Result.CostUSD total every attempt, including ones that came back unparseable — those were billed too — and Result.Attempts records how many calls a job took, so "failed twice, succeeded on the third" is distinguishable from "answered first time". The report shows a Retried column when any job needed more than one call, and the judging pass's own cost as a line under the table. Earlier versions kept only the successful attempt's usage and cleared it entirely on failure, so a run whose answers were unparseable reported as free.

Score is recomputed on every load, so it must be pure. That is what makes editing quality criteria free: a whole run's paid results survive it. The same holds for pricing — the key covers neither, on purpose.

Report a soft finding through Observe, not through Score. Passed() is defined as "Score found nothing", so putting a stylistic preference or a near-miss in Score marks shippable output as failed and flattens the summary table. Observe has the same contract — pure, cheap, recomputed on load — and lands in Result.Observations and an Observed column, without affecting Passed. It was added after a consumer had to build a parallel aggregation outside the harness to avoid exactly that.

Score:   func(c Case, r *Answer) []string { return hardViolations(c, r) },
Observe: func(c Case, r *Answer) []string { return styleNotes(c, r) },

Resuming a run

Each (model, case) job is written to its own file the moment it completes, not buffered until the end. That is a fix for a real incident: a run without it was killed by a laptop going to sleep and lost an entire in-flight batch of paid API calls with nothing recoverable.

Re-run with the same RunID and completed jobs are skipped while errored ones are retried. If the prompt, schema or RunVersion changed since that directory was created, the run stops and says so rather than silently re-running everything at full cost.

BENCHMARK_RUN_ID=candidates_20260805_010203 go test ./tests/benchmark/... -v

Variable	Default	Meaning
<PREFIX>_CONCURRENCY	2	Simultaneous in-flight calls. Higher trips per-provider rate limiting.
<PREFIX>_ATTEMPTS	3	Total tries per job, not retries after the first.
<PREFIX>_RUN_ID	—	Resume this run directory. Empty starts a fresh one.
<PREFIX>_NO_JUDGE	false	Skip the cross-model judging pass.
<PREFIX>_SKIP_ERRORS	false	Re-render the report from disk without retrying failures.

A malformed value is an error, not a silent fallback: NO_JUDGE=yes would otherwise bill you for every judge call and never say why.

API stability in v0.1

Treat as settled: provider, parse, schema, promptver, model, run. These are exercised by three consumers and by golden tests of the outgoing request body.

Three live tests, behind LLMKIT_LIVE=1, cover what only a real model can confirm — that an image reaches it, that a strict schema is honoured end to end, and that the reasoning fields are not silently ignored:

LLMKIT_LIVE=1 LLM_API_KEY=... go test ./... -run TestLive -v

They are the release gate, because coverage cannot see any of it.

Treat as experimental:

• evalharness generic ergonomics. Report[C, R] has to be generic — a non-generic type holding []Result[C, R] does not compile — and the cost is type parameters appearing in your own signatures. It is tolerable, but the shape may still change.
• provider.Response has no escape hatch to the raw SDK response yet. If you need a provider field that is not modelled here, say so rather than working around it.

Contributing

Read RULES.md first. It says what belongs in this library, what stays in your project, and how tests here are expected to be written — including why every new test should be checked by breaking the implementation, which caught four tests in this repository that asserted less than they claimed.

License

MIT.

Documentation

Overview

Package llmkit is a namespace holder; the library lives in its subpackages.

llmkit is the shared LLM plumbing extracted from three Go services that had each grown their own copy of it. The split between packages follows one rule: mechanism belongs here, judgement belongs to the project using it.

provider            vendor-neutral request/response contract
provider/openrouter the only package that names a vendor
parse               pull JSON out of a model's answer, fences and prose included
schema              generate a strict-mode-safe JSON schema from a Go type
promptver           short, collision-resistant version tags for prompts and schemas
model               which model to call, and separately what it costs
run                 the one call sequence: call, decode into T, optional fallback
evalharness         run cases against models, resumably, and report

Start with run.Run for production calls and evalharness.Suite for comparing models. See RULES.md in the repository for what belongs in this library and what does not.