providertest

Documentation

Overview

Package providertest is a contract suite every skyl adapter must pass.

Adapters translate between skyl's types and a vendor's wire format, and the rules they must obey are identical regardless of vendor (docs/rules.md §6): always populate Raw, read the model from the response, classify errors onto skyl's sentinels, never leak a credential, honour the context, and never leak a goroutine when a stream is abandoned.

Writing those assertions once means a new adapter inherits them by filling in a Suite, and a regression in one adapter cannot hide behind another adapter's tests.

Index

• type Live
  • func (l Live) Run(t *testing.T)
• type Suite
  • func (s Suite) Run(t *testing.T)

Types

type Live

type Live struct {
	// Name identifies the adapter in test output.
	Name string

	// EnvKey is the environment variable holding the credential. The test
	// skips when it is unset, so a partial key set still runs what it can.
	EnvKey string

	// New builds the adapter from the credential.
	New func(apiKey string) skyl.Provider

	// Model is the model to call. Keep it small and cheap.
	Model string
}

Live exercises an adapter against a provider's real API.

This is the check that unit tests structurally cannot perform. Every adapter test in this repository replays payloads written from provider documentation, so it proves the mapping logic is self-consistent — but a fake echoes our own assumptions back at us. If a field name is wrong, the fake is wrong in exactly the same way and the test still passes. Only a real call can catch that.

Live tests are build-tagged `integration` and never run in default CI: they cost money and need real credentials.

func (Live) Run

func (l Live) Run(t *testing.T)

Run executes the live checks, skipping if the credential is absent.

type Suite

type Suite struct {
	// Name identifies the adapter in test output.
	Name string

	// New builds the adapter pointed at a test server.
	New func(baseURL string) skyl.Provider

	// APIKey is the credential New bakes in, so the suite can assert it never
	// reaches an error message.
	APIKey string

	// SuccessBody is a minimal successful completion in the vendor's format.
	SuccessBody string

	// WantText is the assistant text SuccessBody encodes.
	WantText string

	// WantModel is the model SuccessBody reports, which must differ from the
	// model the suite requests — that is how it proves the adapter reads the
	// model from the response instead of echoing the request.
	WantModel string

	// ErrorBody is an error payload in the vendor's format. Used with a
	// variety of status codes.
	ErrorBody string

	// StreamFrames are the SSE `data:` payloads of a stream whose text
	// deltas concatenate to "ab".
	StreamFrames []string

	// StreamRawSSE is the complete SSE body, written verbatim, for an adapter
	// whose framing is not bare `data:` frames — Anthropic's stream is a
	// sequence of *named* events, so StreamFrames cannot express it.
	//
	// When set it replaces StreamFrames. Its text deltas must still
	// concatenate to "ab", so the same assertions hold for every adapter.
	StreamRawSSE string

	// SkipStream turns off the streaming assertions, for an adapter that does
	// not support streaming.
	//
	// It is not for an adapter whose *framing* is merely unusual: use
	// StreamRawSSE for that. Skipping costs three checks, and one of them —
	// the goroutine-leak check — has no equivalent anywhere else.
	SkipStream bool
}

Suite describes one adapter well enough to exercise the shared contract.

The payload fields are vendor-specific by necessity: the whole point of an adapter is that its wire format differs. Everything asserted about them is vendor-neutral.

func (Suite) Run

func (s Suite) Run(t *testing.T)

Run executes the whole contract against the adapter.