assay

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2026 License: MIT Imports: 11 Imported by: 0

README

assay

Go Version Go Reference

A framework-agnostic evaluation library and CLI for Go. Point assay at an MCP server or an AI agent, give it a suite of checks in YAML, and it scores whether the right tools were called, with the right arguments, fast enough, and producing the expected output — then fails your CI when something regresses.

Built on the official Go MCP SDK and tested against Google ADK for Go.

Why assay?

The Go ecosystem has agent frameworks (ADK, eino, trpc-agent-go) and a first-class MCP SDK — but no standalone way to evaluate what you build with them. The evaluators that exist are bundled into a single framework and only test that framework's agents. If you ship an MCP server, there's nothing in Go that answers the everyday question: does my server still expose the tools it claims, and do they still work?

assay is that missing piece — a deterministic, framework-agnostic test harness for tool-using systems:

  • Framework-agnostic — the core depends on nothing but the standard library. Anything that satisfies a tiny interface can be evaluated: MCP servers, ADK agents, or your own.
  • Deterministic — MCP server checks need no LLM and no API keys. They run free, in CI, on every commit.
  • Minimal integration — test an MCP server with zero Go code (a binary + a YAML). Test an agent with two lines.
  • CI-ready — every run returns a proper exit code, so a failing suite breaks the build.
  • Open source — MIT licensed.
  • Production grade — keyless-testable core, structured reports (console + JSON), clean multi-module architecture.

Architecture

Architecture

assay is built as two parallel tracks over one shared core.

The core (the root assay package) is completely agnostic — it imports no framework SDK. It defines two contracts:

  • Agent — anything with Run(ctx, input) (Result, error). This is the agent track: give it a prompt, see which tools it chose and what it returned.
  • ToolServer — anything with ListTools(ctx) and CallTool(ctx, name, args). This is the MCP track: probe a server's contract directly, no LLM involved.

Adapters live in their own modules so their heavy dependency trees never reach the core:

  • adk/ wraps a Google ADK agent and satisfies Agent.
  • mcp/ wraps the official MCP SDK client and satisfies ToolServer.

Both tracks feed the same shared machinery — the runner, the scoring, the console/JSON reporters, the YAML suite loader, and the CLI with its exit codes. Adding support for a new framework (LangChain-Go, a vanilla LLM loop, another MCP transport) means writing one thin adapter against an existing interface — nothing in the core changes.

The dependency arrow only ever points one way: adapters depend on the core's interfaces, never the reverse. If the core's go.mod ever needs to import an SDK, the boundary has been violated.

What it evaluates

MCP server checks (deterministic, no API key)
Check Description
tools_list Asserts the server exposes the expected set of tools
tool_call Calls a tool with given args and asserts on the outcome — expect_no_error, result_contains, max_latency
Agent metrics
Metric Description
tool_correctness Did the agent call the expected tools (optionally ordered, with argument matching)?
output_contains Does the response contain the expected substrings?
output_regex Does the response match the expected patterns?
max_latency Did the run complete within the latency ceiling?

Each check/metric returns a score in [0,1]; a suite passes when the aggregate meets its threshold.

Quickstart — test an MCP server (zero Go code)

Build your server to a binary, write a checks.yaml, and run assay. No integration code required.

# checks.yaml
threshold: 1.0
checks:
  - name: lists_all_tools
    type: tools_list
    expect_tools: [get_visa_bulletin, check_priority_date, explain_term]

  - name: explain_term_works
    type: tool_call
    tool: explain_term
    args: { term: "priority date" }
    expect_no_error: true
    max_latency: 10s

assay mcp --server "./immigration-mcp-server" --suite checks.yaml

CASE                      SCORE  RESULT
tools_list                1.00   ✓
  tools_list              1.00   3/3 expected tools exposed
tool_call:explain_term    1.00   ✓
  tool_call:explain_term  1.00   2/2 assertions passed
SUITE SCORE: 1.00   threshold 1.00   PASS

Exit code is 0 on pass, 1 on failure — drop it straight into a CI job.

Quickstart — test an agent (a few lines of Go)

Agents are constructed in code, so you wrap your agent in the matching adapter, register it, and hand off to the assay CLI. Here, an ADK agent:

package main

import (
    "github.com/tushariitr-19/assay/adk"
    "github.com/tushariitr-19/assay/cli/app"
)

func main() {
    adapter, _ := adk.New(rootAgent) // wrap your ADK agent
    app.SetAgent(adapter)            // register it
    app.Main()                       // hand off to the assay CLI
}

Build it and run the agent track through the CLI:

go build -o myeval .
./myeval agent --suite cases.yaml

# cases.yaml
threshold: 0.80
cases:
  - name: search_recent_papers
    input: "Find recent papers on retrieval-augmented generation"
    expect_tools: [TopicSearchAgent]
    output_contains: ["RAG"]
    max_latency: 60s

CASE                  SCORE  RESULT
search_recent_papers  1.00   ✓
  tool_correctness    1.00   1/1 expected tools called
  output_contains     1.00   1/1 substrings present
  max_latency         1.00   17527ms <= 60000ms
SUITE SCORE: 1.00   threshold 0.80   PASS

A runnable version lives in examples/research-agent/, dogfooded against research-agent-adk.

Bring your own agent: if you're not on ADK, implement the one-method Agent interface yourself — everything else (metrics, runner, reports, CLI) works unchanged.

type MyAgent struct{ /* ... */ }

func (a MyAgent) Run(ctx context.Context, input string) (assay.Result, error) {
    // call your agent, return Output + the tool calls it made
}

You can also call the runner directly from a _test.go instead of building a CLI:

adapter, _ := adk.New(rootAgent)
suite, _ := assay.LoadSuite("cases.yaml")
report := assay.Runner{}.Run(ctx, adapter, suite)
report.WriteText(os.Stdout)

Prerequisites

  • Go 1.25+
  • For MCP server checks: no API keys — assay only talks to your server.
  • For the agent example: whatever your agent needs (e.g. a Google AI Studio key for the ADK example).

Install

# Library — to embed in Go tests or programs
go get github.com/tushariitr-19/assay

# CLI — to test MCP servers from the command line
go install github.com/tushariitr-19/assay/cli/cmd/assay@latest

The installed assay binary tests MCP servers out of the box (assay mcp ...). To test an agent, build your own binary that registers it via app.SetAgent — see the agent quickstart above — since agents are constructed in code.

Project Structure

assay/
├── agent.go             ← Agent interface, Result, ToolCall
├── toolserver.go        ← ToolServer interface, ToolOutcome
├── metric.go            ← agent metrics (tool_correctness, output_contains, …)
├── check.go             ← MCP checks (tools_list, tool_call)
├── runner.go            ← agent runner
├── server_runner.go     ← MCP server runner
├── reporter.go          ← console + JSON reports (shared by both tracks)
├── suite_yaml.go        ← agent suite loader
├── server_yaml.go       ← MCP suite loader
├── config/              ← run configuration
├── logger/              ← structured logging via zap
├── internal/
│   ├── fakeagent/       ← keyless agent for tests
│   └── faketoolserver/  ← keyless server for tests
├── adk/                 ← ADK adapter (own module)
├── mcp/                 ← MCP adapter (own module)
├── cli/                 ← the assay binary (own module)
│   ├── app/             ← importable CLI: SetAgent, Main
│   └── cmd/assay/       ← the assay binary entry point
├── examples/
│   └── research-agent/  ← ADK dogfood example
├── tests/               ← integration tests
└── docs/                ← architecture diagram + screenshots

This is a multi-module repository: the root is the agnostic core library; adk, mcp, and cli are separate modules so their SDK dependencies stay out of the core.

Makefile Targets

make build             # Build the assay CLI binary
make test              # Unit tests (keyless, no network)
make test-integration  # Integration tests (spawns real servers / agents)
make test-all          # All tests
make lint              # go vet + gofmt check
make clean             # Remove build artifacts

Design Decisions

Agnostic core, thin adapters — the core knows two interfaces and nothing about ADK or MCP. Framework specifics live in dedicated modules, so the published library stays dependency-light and any new framework is a small additive adapter.

Keyless, deterministic core — every metric and check is unit-tested against in-memory fakes (fakeagent, faketoolserver) with no network or credentials, so the whole core runs green in CI for free.

MCP testing is contract testing, not LLM testing — an MCP server doesn't decide anything; it provides tools. So assay probes it directly (tools/list, tools/call) rather than putting a model in the loop. Deterministic, cheap, and exactly what "does my server work?" means.

Two tracks, one report — the agent runner and the server runner produce the identical report structure, so the console/JSON reporters and CI exit codes are shared, untouched, across both.

Transport vs. tool errors are distinct — a crashed connection surfaces as a Go error; a tool that ran but reported failure surfaces as IsError, scored by the expect_no_error check. The two are never conflated.

Roadmap

  • MCP server evaluation — tools_list and tool_call checks
  • Agent evaluation — tool correctness, output contains/regex, latency
  • Official MCP SDK adapter (stdio transport)
  • Google ADK adapter
  • Unified assay CLI with CI exit codes
  • Console + JSON reports, YAML suites
  • assay agent subcommand via register-and-build (app.SetAgent + Main)
  • LLM-as-judge metric for response quality (behind a pluggable Judge interface)
  • Nested trajectory granularity — score leaf tool calls inside sub-agents, not just root routing
  • MCP resources and prompts checks
  • MCP HTTP transport (in addition to stdio)
  • More adapters — LangChain-Go, vanilla LLM loops
  • Synthetic test-case generation

Contributing

PRs welcome. To support a new framework, you only write an adapter:

  1. Create a new module (e.g. langchain/) with its own go.mod.
  2. Implement either assay.Agent (for agents) or assay.ToolServer (for tool servers).
  3. Translate the framework's output into assay.Result / assay.ToolOutcome.
  4. Add an example under examples/ and an integration test under tests/.

The core never needs to change.

License

MIT — see LICENSE for details.

Acknowledgements

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Agent 

type Agent interface {
	Run(ctx context.Context, input string) (Result, error)
}

Agent is the only contract the assay core depends on. Any agent — ADK, MCP, or anything else — is evaluated by wrapping it in an adapter that satisfies this interface. The core never imports a framework; it only knows this.

type Case 

type Case struct {
	Name        string
	Input       string
	Expectation Expectation
}

Case is a single evaluation scenario: an input to send the agent and the expectations its result is scored against.

type CaseResult 

type CaseResult struct {
	Case    Case
	Scores  map[string]float64 // metric name -> score
	Details map[string]string  // metric name -> human-readable detail
	Score   float64            // mean of the metric scores
	Err     error
}

CaseResult is the outcome of running one case: its per-metric scores and the averaged case score. Err is set if the agent itself failed.

type Expectation 

type Expectation struct {
	Tools        []string // expected tool names (for ToolCorrectness)
	OutputSubstr []string // substrings expected in Output
	OutputRegex  []string // regex patterns Output should match
	MaxLatencyMS int64    // latency ceiling in milliseconds (0 = no check)
}

Expectation describes what a single case expects. Fields left empty are simply not checked by the metrics that read them.

type MaxLatency 

type MaxLatency struct{}

MaxLatency is a pass/fail check on run duration.

func (MaxLatency) Name 

func (MaxLatency) Name() string

func (MaxLatency) Score 

func (MaxLatency) Score(result Result, exp Expectation) (float64, string)

type Metric 

type Metric interface {
	// Name identifies the metric in reports.
	Name() string

	// Score returns a value in [0,1] plus a short human-readable detail.
	Score(result Result, exp Expectation) (score float64, detail string)
}

Metric scores one aspect of a Result against a case's expectations. Every metric — including future ones like an LLM judge — satisfies this same interface, which is what keeps the runner agnostic to what it's scoring.

func DefaultMetrics 

func DefaultMetrics() []Metric

DefaultMetrics returns the standard v1 metric set.

type OutputContains 

type OutputContains struct{}

OutputContains scores how many expected substrings appear in Output.

func (OutputContains) Name 

func (OutputContains) Name() string

func (OutputContains) Score 

func (OutputContains) Score(result Result, exp Expectation) (float64, string)

type OutputRegex 

type OutputRegex struct{}

OutputRegex scores how many expected patterns match Output.

func (OutputRegex) Name 

func (OutputRegex) Name() string

func (OutputRegex) Score 

func (OutputRegex) Score(result Result, exp Expectation) (float64, string)

type Result 

type Result struct {
	Output    string
	ToolCalls []ToolCall
	Latency   time.Duration
}

Result is what an agent returns from a single run: its final text output, the trace of tools it called, and how long the run took.

type RunReport 

type RunReport struct {
	Results   []CaseResult
	Score     float64 // mean of all case scores
	Threshold float64
	Passed    bool
}

RunReport is the outcome of running a whole suite.

func RunServer 

func RunServer(ctx context.Context, ts ToolServer, suite ServerSuite) RunReport

RunServer executes every check against the ToolServer and aggregates scores into the same RunReport shape the agent runner produces — so the existing reporters (WriteText / WriteJSON) work unchanged.

func (RunReport) WriteJSON 

func (r RunReport) WriteJSON(w io.Writer) error

WriteJSON renders the report as indented JSON to w.

func (RunReport) WriteText 

func (r RunReport) WriteText(w io.Writer)

WriteText renders the report as a human-readable table to w.

type Runner 

type Runner struct {
	Metrics []Metric
}

Runner evaluates an Agent against a Suite using a fixed set of metrics.

func (Runner) Run 

func (r Runner) Run(ctx context.Context, agent Agent, suite Suite) RunReport

Run executes every case in the suite and aggregates the scores.

type ServerCheck 

type ServerCheck interface {
	Name() string
	Run(ctx context.Context, ts ToolServer) (score float64, detail string)
}

ServerCheck scores one aspect of a ToolServer. Each check performs its own operation against the server (list or call) and returns a score in [0,1].

type ServerSuite 

type ServerSuite struct {
	Threshold float64
	Checks    []ServerCheck
}

ServerSuite is a named set of checks with a pass threshold in [0,1].

func LoadServerSuite 

func LoadServerSuite(path string) (ServerSuite, error)

LoadServerSuite reads a YAML file into a ServerSuite of checks.

type Suite 

type Suite struct {
	Threshold float64
	Cases     []Case
}

Suite is a named collection of cases with a pass threshold in [0,1]. A run passes when its aggregate score meets or exceeds Threshold.

func LoadSuite 

func LoadSuite(path string) (Suite, error)

LoadSuite reads and parses a YAML suite file into a Suite.

type ToolCall 

type ToolCall struct {
	Name string
	Args map[string]any
}

ToolCall records a single tool invocation: the tool's name and the arguments the agent chose to pass it.

type ToolCallCheck 

type ToolCallCheck struct {
	Tool           string
	Args           map[string]any
	ExpectNoError  bool     // outcome.IsError must be false
	ResultContains []string // substrings expected in outcome.Text
	MaxLatencyMS   int64    // 0 = no latency check
}

ToolCallCheck calls one tool and asserts on the outcome.

func (ToolCallCheck) Name 

func (c ToolCallCheck) Name() string

func (ToolCallCheck) Run 

func (c ToolCallCheck) Run(ctx context.Context, ts ToolServer) (float64, string)

type ToolCorrectness 

type ToolCorrectness struct {
	Ordered bool // if true, call order must match expectation order
}

ToolCorrectness scores whether the agent called the expected tools. In its simplest form it checks set membership (did each expected tool get called, ignoring order and extras).

func (ToolCorrectness) Name 

func (ToolCorrectness) Name() string

func (ToolCorrectness) Score 

func (m ToolCorrectness) Score(result Result, exp Expectation) (float64, string)

type ToolOutcome 

type ToolOutcome struct {
	IsError bool          // the server reported the tool call as failed
	Text    string        // concatenated text content from the response
	Latency time.Duration // wall-clock time for this call
}

ToolOutcome is the normalized result of a single tool call, flattened from whatever the underlying protocol returns into what checks need to inspect.

type ToolServer 

type ToolServer interface {
	// ListTools returns the names of the tools the server exposes.
	ListTools(ctx context.Context) ([]string, error)

	// CallTool invokes one tool by name with the given arguments.
	CallTool(ctx context.Context, name string, args map[string]any) (ToolOutcome, error)
}

ToolServer is the contract for evaluating a tool-exposing server (e.g. an MCP server) directly, with no LLM. An adapter (e.g. the mcp package) wraps a live server connection and satisfies this interface; the core never imports an MCP SDK, so it stays protocol-agnostic.

type ToolsListCheck 

type ToolsListCheck struct {
	Expected []string
}

ToolsListCheck verifies the server exposes the expected tools.

func (ToolsListCheck) Name 

func (ToolsListCheck) Name() string

func (ToolsListCheck) Run 

func (c ToolsListCheck) Run(ctx context.Context, ts ToolServer) (float64, string)

Source Files

View all Source files

Directories

Path Synopsis
internal
fakeagent
Package fakeagent provides an in-memory Agent implementation for tests.
 Package fakeagent provides an in-memory Agent implementation for tests.
faketoolserver
Package faketoolserver provides an in-memory ToolServer for tests.
 Package faketoolserver provides an in-memory ToolServer for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.