modeltest

Documentation

Overview

Package modeltest provides reusable contract tests for implementations of Core model interfaces. Provider modules use these suites and transport fixtures from their tests; production code should not import this package.

Example

package main

import (
	"fmt"
	"iter"

	"github.com/Tangerg/scope/core/modeltest"
)

func main() {
	sequence := iter.Seq2[string, error](func(yield func(string, error) bool) {
		yield("first", nil)
		yield("second", nil)
	})
	values, err := modeltest.Collect(sequence)
	fmt.Println(values, err)
}

Output:
[first second] <nil>

Index

• func AnthropicSSEServer(events []AnthropicEvent) *httptest.Server
• func BinaryServer(status int, contentType string, body []byte, ...) *httptest.Server
• func Collect[T any](seq iter.Seq2[T, error]) ([]T, error)
• func CollectN[T any](sequence iter.Seq2[T, error], count int) ([]T, error)
• func JSONServer(status int, body string, inspections ...func(request *http.Request)) *httptest.Server
• func LookupEnv(name string) (string, bool)
• func MuxServer(routes ...Route) *httptest.Server
• func OpenAISSEServer(chunks []string) *httptest.Server
• func RequireEnv(t *testing.T, name string) string
• func RequireKey(t *testing.T, provider string) string
• func RunEmbeddingContract(t *testing.T, contract EmbeddingContract)
• func RunIntegrationEmbedding(t *testing.T, probe IntegrationEmbeddingProbe)
• func RunIntegrationRerank(t *testing.T, probe IntegrationRerankProbe)
• func RunRerankContract(t *testing.T, contract RerankContract)
• func WithTimeout(t *testing.T, duration time.Duration) (context.Context, context.CancelFunc)
• type AnthropicEvent
• type CallBehaviorCase
• type ChatBehaviorSuite
  • func (c ChatBehaviorSuite) Run(t *testing.T)
• type ChatSuite
  • func (c ChatSuite) Run(t *testing.T)
• type EmbeddingContract
• type IntegrationEmbeddingProbe
• type IntegrationRerankProbe
• type Lifecycle
  • func NewBlockingServer(t *testing.T, writeInitial func(http.ResponseWriter)) (*httptest.Server, Lifecycle)
• type PollCounter
  • func (p *PollCounter) Inc() int32
  • func (p *PollCounter) N() int32
• type RerankContract
• type Route
• type StreamBehaviorCase

Examples

• Package

Functions

func AnthropicSSEServer

func AnthropicSSEServer(events []AnthropicEvent) *httptest.Server

AnthropicSSEServer returns an httptest.Server that streams `events` as Anthropic-shaped SSE:

event: message_start\ndata: {...}\n\n
event: content_block_delta\ndata: {...}\n\n
...
event: message_stop\ndata: {...}\n\n

Anthropic uses named events rather than a single sentinel; the caller is responsible for providing the right sequence.

func BinaryServer

func BinaryServer(status int, contentType string, body []byte, inspections ...func(request *http.Request)) *httptest.Server

func Collect

func Collect[T any](seq iter.Seq2[T, error]) ([]T, error)

Collect drains an iter.Seq2[T, error] iterator into a slice. The iteration stops on the first non-nil error, which is returned along with whatever was yielded so far.

This is the canonical helper for streaming-test assertions: spin up a mock SSE server, call model.Stream(ctx, req), Collect the result, then assert on the slice + final error.

func CollectN

func CollectN[T any](sequence iter.Seq2[T, error], count int) ([]T, error)

CollectN drains at most n items from the iterator. Use this for cancellation tests — break early to verify the iterator's stop function tears down the upstream connection cleanly.

func JSONServer

func JSONServer(status int, body string, inspections ...func(request *http.Request)) *httptest.Server

func LookupEnv

func LookupEnv(name string) (string, bool)

func MuxServer

func MuxServer(routes ...Route) *httptest.Server

func OpenAISSEServer

func OpenAISSEServer(chunks []string) *httptest.Server

OpenAISSEServer returns an httptest.Server that streams `chunks` as OpenAI-shaped Server-Sent Events:

data: <chunk-1>\n\n
data: <chunk-2>\n\n
...
data: [DONE]\n\n

Each chunk should be a JSON-encoded `ChatCompletionChunk` body. The server is registered with t.Cleanup so callers don't have to defer Close().

Used by every OpenAI-compatible vendor (openai / azureopenai / deepseek / moonshot / openrouter / xai / groq / together / fireworks / perplexity / alibaba / zhipu / minimax / ...).

func RequireEnv

func RequireEnv(t *testing.T, name string) string

func RequireKey

func RequireKey(t *testing.T, provider string) string

func RunEmbeddingContract

func RunEmbeddingContract(t *testing.T, contract EmbeddingContract)

func RunIntegrationEmbedding

func RunIntegrationEmbedding(t *testing.T, probe IntegrationEmbeddingProbe)

func RunIntegrationRerank

func RunIntegrationRerank(t *testing.T, probe IntegrationRerankProbe)

func RunRerankContract

func RunRerankContract(t *testing.T, contract RerankContract)

func WithTimeout

func WithTimeout(t *testing.T, duration time.Duration) (context.Context, context.CancelFunc)

Types

type AnthropicEvent

type AnthropicEvent struct {
	Event string
	Data  string
}

AnthropicEvent is a single named SSE event for Anthropic's multi-event-type streaming protocol.

type CallBehaviorCase

type CallBehaviorCase struct {
	Model     chat.Model
	Lifecycle Lifecycle
}

CallBehaviorCase supplies an in-flight Call and its provider lifecycle.

type ChatBehaviorSuite

type ChatBehaviorSuite struct {
	Request            func(t *testing.T) *chat.Request
	CallCancellation   func(t *testing.T) CallBehaviorCase
	StreamCancellation func(t *testing.T) StreamBehaviorCase
	EarlyStop          func(t *testing.T) StreamBehaviorCase
	FirstError         func(t *testing.T) chat.Streamer
}

ChatBehaviorSuite exercises lifecycle and terminal-error behavior against a provider's real SDK transport. Each factory must return fresh state.

func (ChatBehaviorSuite) Run

func (c ChatBehaviorSuite) Run(t *testing.T)

Run executes the shared Call/Stream behavior contract.

type ChatSuite

type ChatSuite struct {
	New              func(t *testing.T) (chat.Model, chat.Streamer)
	Request          func(t *testing.T) *chat.Request
	AssertCall       func(t *testing.T, response *chat.Response)
	AssertStream     func(t *testing.T, responses []*chat.Response)
	AssertAggregated func(t *testing.T, response *chat.Response)
}

ChatSuite describes one provider's happy-path Model and Streamer contract. New and Request are called independently for each subtest so provider state and request mutation cannot leak between Call and Stream.

func (ChatSuite) Run

func (c ChatSuite) Run(t *testing.T)

Run executes the shared synchronous and streaming conformance cases.

type EmbeddingContract

type EmbeddingContract struct {
	// ModelID is the model id passed into the embedding request.
	ModelID string
	// Response is the canned JSON body — must encode 2 outputs so the
	// contract can validate batching.
	Response string
	// ExpectedPath is the URL path the SDK should hit (e.g. "/embeddings"
	// or "/embedding/text"). Empty means skip the path assertion.
	ExpectedPath string
	// Build returns the model wired against the mock server.
	Build func(t *testing.T, baseURL string) embedding.Model
}

EmbeddingContract drives the mock-test contract for any embedding vendor. The `Response` field is the canned JSON body the mock server returns — it should encode a response with 2 embeddings (matching the 2-input request the contract sends).

type IntegrationEmbeddingProbe

type IntegrationEmbeddingProbe struct {
	Provider string
	Build    func(t *testing.T, key string) embedding.Model
}

IntegrationEmbeddingProbe is the standard real-API embedding smoke probe: Call returns 2 outputs with non-empty embeddings.

type IntegrationRerankProbe

type IntegrationRerankProbe struct {
	Provider string
	Build    func(t *testing.T, key string) rerank.Model
}

type Lifecycle

type Lifecycle struct {
	Started <-chan struct{}
	Stopped <-chan struct{}
}

Lifecycle observes one in-flight provider request. Started closes after the mock has sent any initial stream event; Stopped closes when the request context is released and the handler exits.

func NewBlockingServer

func NewBlockingServer(t *testing.T, writeInitial func(http.ResponseWriter)) (*httptest.Server, Lifecycle)

type PollCounter

type PollCounter struct {
	// contains filtered or unexported fields
}

PollCounter holds a goroutine-safe attempt counter. Polling vendors typically need to return "in-progress" for the first N polls then "completed" — bind a PollCounter to the GET handler to drive that.

func (*PollCounter) Inc

func (p *PollCounter) Inc() int32

func (*PollCounter) N

func (p *PollCounter) N() int32

type RerankContract

type RerankContract struct {
	ModelID      string
	Response     string
	ExpectedPath string
	Build        func(t *testing.T, baseURL string) rerank.Model
}

RerankContract drives the mock transport contract for a reranking provider.

type Route

type Route struct {
	Method   string
	Contains string
	Handle   http.HandlerFunc
}

Route names an HTTP method + path-substring pair plus its handler. The Contains field is matched against r.URL.Path with strings.Contains, so "/transcript" matches both "/v2/transcript" (the POST) and "/v2/transcript/job-1" (the GET poll). When Contains is empty the route matches every path — useful as a fallback.

type StreamBehaviorCase

type StreamBehaviorCase struct {
	Streamer  chat.Streamer
	Lifecycle Lifecycle
}

StreamBehaviorCase supplies an in-flight Stream and its provider lifecycle.