llmtest

Documentation

Overview

Package llmtest provides utilities for testing LLM implementations.

Inspired by Go's testing/fstest package, llmtest offers a simple, backend-independent way to verify that LLM implementations conform to the expected interfaces and behaviors.

Design Philosophy

Following the principles of testing/fstest:

• Minimal API surface - one main function (TestLLM)
• Automatic capability discovery - no configuration required
• Comprehensive by default - tests all detected capabilities
• Interface testing - works with any llms.Model implementation
• Simple usage pattern - just pass the model to test

Usage

Testing an LLM implementation is straightforward:

func TestMyLLM(t *testing.T) {
    llm, err := mylllm.New()
    if err != nil {
        t.Fatal(err)
    }
    llmtest.TestLLM(t, llm)
}

Automatic Capability Discovery

The package automatically detects and tests supported capabilities:

• Basic operations (Call, GenerateContent)
• Streaming (if model implements streaming interface)
• Tool/Function calling (probed with test tool)
• Reasoning/Thinking mode (if supported)
• Token counting (if usage information provided)
• Context caching (if implemented)

Mock Implementation

A MockLLM is provided for testing without making actual API calls:

mock := &llmtest.MockLLM{
    CallFunc: func(ctx context.Context, prompt string, options ...llms.CallOption) (string, error) {
        return "mocked response", nil
    },
}
llmtest.TestLLM(t, mock)

Parallel Testing

All tests run in parallel by default for better performance:

• Core tests (Call, GenerateContent) run concurrently
• Capability tests run in parallel when detected
• Safe for concurrent execution with independent contexts

Provider Coverage

The package is used to test all LangChain Go providers: anthropic, bedrock, cloudflare, cohere, ernie, fake, googleai, huggingface, llamafile, local, maritaca, mistral, ollama, openai, watsonx, and more.

Package llmtest provides support for testing LLM implementations.

Following the design of testing/fstest, this package provides a simple TestLLM function that verifies an LLM implementation behaves correctly.

Index

• func TestLLM(t *testing.T, model llms.Model)
• func TestLLMWithOptions(t *testing.T, model llms.Model, opts TestOptions, expected ...string)
• func ValidateLLM(model llms.Model) error
• type MockLLM
  • func (m *MockLLM) Call(ctx context.Context, prompt string, options ...llms.CallOption) (string, error)
  • func (m *MockLLM) GenerateContent(ctx context.Context, messages []llms.MessageContent, ...) (*llms.ContentResponse, error)
  • func (m *MockLLM) GenerateContentStream(ctx context.Context, messages []llms.MessageContent, ...) (<-chan llms.ContentResponse, error)
• type TestOptions

Functions

func TestLLM

func TestLLM(t *testing.T, model llms.Model)

TestLLM tests an LLM implementation. It performs basic operations and checks that the model behaves correctly. It automatically discovers and tests capabilities by probing the model.

If TestLLM finds any misbehaviors, it reports them via t.Error/t.Fatal.

Typical usage inside a test:

func TestLLM(t *testing.T) {
    llm, err := mylllm.New(...)
    if err != nil {
        t.Fatal(err)
    }
    llmtest.TestLLM(t, llm)
}

func TestLLMWithOptions

func TestLLMWithOptions(t *testing.T, model llms.Model, opts TestOptions, expected ...string)

TestLLMWithOptions tests an LLM with specific test options.

func ValidateLLM

func ValidateLLM(model llms.Model) error

ValidateLLM checks if a model satisfies basic requirements without running tests. It returns an error describing what's wrong, or nil if the model is valid.

Types

type MockLLM

type MockLLM struct {
	// Response to return from Call
	CallResponse string
	CallError    error

	// Response to return from GenerateContent
	GenerateResponse *llms.ContentResponse
	GenerateError    error

	// Track calls for verification
	CallCount     int
	GenerateCount int
	LastPrompt    string
	LastMessages  []llms.MessageContent
}

MockLLM provides a simple mock implementation for testing.

func (*MockLLM) Call

func (m *MockLLM) Call(ctx context.Context, prompt string, options ...llms.CallOption) (string, error)

Call implements llms.Model

func (*MockLLM) GenerateContent

func (m *MockLLM) GenerateContent(ctx context.Context, messages []llms.MessageContent, options ...llms.CallOption) (*llms.ContentResponse, error)

GenerateContent implements llms.Model

func (*MockLLM) GenerateContentStream

func (m *MockLLM) GenerateContentStream(ctx context.Context, messages []llms.MessageContent, options ...llms.CallOption) (<-chan llms.ContentResponse, error)

GenerateContentStream implements streaming

type TestOptions

type TestOptions struct {
	// Timeout for each test operation
	Timeout time.Duration

	// Skip specific test categories
	SkipCall            bool
	SkipGenerateContent bool
	SkipStreaming       bool

	// Custom test prompts
	TestPrompt   string
	TestMessages []llms.MessageContent

	// For providers that need special options
	CallOptions []llms.CallOption
}

TestOptions configures test execution.