evalkit

README

EvalKit

evalkit is a small host-side evaluation harness for Go agent applications. It defines task, trial, trace, grader, and summary contracts without importing goagent or any provider package.

It is intentionally not an agent runtime. Hosts adapt their own agent, workflow, or service call into the Harness interface, then run repeatable evaluation suites around that boundary.

Use

runner := evalkit.Runner{
    Harness: evalkit.HarnessFunc(func(ctx context.Context, task evalkit.Task) (*evalkit.RunResult, error) {
        result, err := agent.RunDetailed(ctx, agentcore.RunRequest{Input: task.Input})
        if err != nil {
            return nil, err
        }
        return &evalkit.RunResult{
            Output: result.Content,
            Outcome: evalkit.Outcome{
                Status:    "succeeded",
                OutputRef: "artifact:run-output",
            },
        }, nil
    }),
    TrialsPerTask: 3,
}

suite := evalkit.Suite{
    Name: "tool-selection-smoke",
    Tasks: []evalkit.Task{{
        ID: "lookup-account",
        Input: "Look up account A-123 and summarize the status.",
        SuccessCriteria: "Uses the account lookup tool and returns a concise status.",
    }},
    Graders: []evalkit.Grader{
        evalkit.GraderFunc{
            GraderName: "contains-status",
            Fn: func(ctx context.Context, req evalkit.GradeRequest) (*evalkit.GradeResult, error) {
                passed := strings.Contains(req.Trial.Output, "status")
                return &evalkit.GradeResult{Passed: passed, Score: 1}, nil
            },
        },
    },
}

report, err := runner.Run(ctx, suite)

Boundary

evalkit owns:

• task and suite definitions
• repeated trials for non-deterministic agent behavior
• grader contracts and assertion aggregation
• trace DTOs suitable for host adapters
• suite-level pass/fail summaries

It does not own:

• model calls
• tool execution
• durable storage
• online production monitoring
• hosted dashboards
• model-as-judge implementations

Outcome and sanitized traces

RunResult.Outcome separates the evaluated system's domain result from a Harness execution error. Use Status, OutputRef, and a stable ErrorCode for graders; keep raw provider errors in neither Outcome nor Trace. Outcome metadata is defensively copied into each Trial.

Host adapters decide how persisted runtime data becomes TraceStep values. They should use fixed metadata and label allowlists, store only host-owned artifact references, and exclude prompts, tool inputs/outputs, credentials, absolute trust paths, and arbitrary event messages. The host-api example maps its existing SQLite workflow and run stores this way without adding a production HTTP endpoint.

Use runkit or artifactkit when a host wants durable trace or artifact storage. Use a host adapter to map goagent.RunDetailed, workflow runs, or service endpoints into evalkit.Harness.

See ../examples/evalkit-goagent-regression for a runnable goagent regression evaluation that uses a mock LLM, repeated trials, and graders.

Verify

go test ./...

Documentation

Index

• type Assertion
• type GradeRequest
• type GradeResult
• type Grader
• type GraderFunc
  • func (g GraderFunc) Grade(ctx context.Context, req GradeRequest) (*GradeResult, error)
  • func (g GraderFunc) Name() string
• type Harness
• type HarnessFunc
  • func (f HarnessFunc) RunTask(ctx context.Context, task Task) (*RunResult, error)
• type Outcome
• type RunResult
• type Runner
  • func (r Runner) Run(ctx context.Context, suite Suite) (*SuiteResult, error)
• type Suite
• type SuiteResult
• type Summary
• type Task
• type Trace
• type TraceStep
• type Trial
• type TrialResult
• type Usage

Types

type Assertion

type Assertion struct {
	Name    string
	Passed  bool
	Message string
}

type GradeRequest

type GradeRequest struct {
	Task  Task
	Trial Trial
}

type GradeResult

type GradeResult struct {
	Name       string
	Score      float64
	Passed     bool
	Message    string
	Assertions []Assertion
	Metadata   map[string]any
}

GradeResult is one grader's decision for a trial. Assertions, when present, are aggregated into Passed so callers do not need to duplicate that logic.

type Grader

type Grader interface {
	Name() string
	Grade(context.Context, GradeRequest) (*GradeResult, error)
}

type GraderFunc

type GraderFunc struct {
	GraderName string
	Fn         func(context.Context, GradeRequest) (*GradeResult, error)
}

func (GraderFunc) Grade

func (g GraderFunc) Grade(ctx context.Context, req GradeRequest) (*GradeResult, error)

func (GraderFunc) Name

func (g GraderFunc) Name() string

type Harness

type Harness interface {
	RunTask(context.Context, Task) (*RunResult, error)
}

Harness adapts an agent, workflow, or service endpoint into evalkit.

type HarnessFunc

type HarnessFunc func(context.Context, Task) (*RunResult, error)

func (HarnessFunc) RunTask

func (f HarnessFunc) RunTask(ctx context.Context, task Task) (*RunResult, error)

type Outcome

type Outcome struct {
	Status    string
	OutputRef string
	ErrorCode string
	Metadata  map[string]any
}

Outcome is the evaluated system's domain result. OutputRef points to host-owned content without copying that content into an eval report.

type RunResult

type RunResult struct {
	Output   string
	Outcome  Outcome
	Trace    Trace
	Metadata map[string]any
}

type Runner

type Runner struct {
	Harness       Harness
	TrialsPerTask int
	Now           func() time.Time
}

Runner executes a suite sequentially. It intentionally does not own concurrency, durable storage, or model-as-judge infrastructure.

func (Runner) Run

func (r Runner) Run(ctx context.Context, suite Suite) (*SuiteResult, error)

type Suite

type Suite struct {
	Name    string
	Tasks   []Task
	Graders []Grader
}

Suite groups tasks and graders that should be evaluated together.

type SuiteResult

type SuiteResult struct {
	Name       string
	StartedAt  time.Time
	FinishedAt time.Time
	Summary    Summary
	Trials     []TrialResult
}

type Summary

type Summary struct {
	TotalTrials  int
	PassedTrials int
	FailedTrials int
	TaskCount    int
	GraderCount  int
	Duration     time.Duration
}

type Task

type Task struct {
	ID              string
	Input           string
	SuccessCriteria string
	Metadata        map[string]any
}

Task is one eval case with stable input and reader-facing success criteria.

type Trace

type Trace struct {
	RunID  string
	Steps  []TraceStep
	Usage  Usage
	Labels map[string]string
}

Trace is the bounded trajectory record used by graders. Hosts decide how much detail to map in from runtime events, logs, or workflow records.

type TraceStep

type TraceStep struct {
	Type      string
	Name      string
	Status    string
	Message   string
	StartedAt time.Time
	EndedAt   time.Time
	Metadata  map[string]any
}

type Trial

type Trial struct {
	TaskID     string
	Index      int
	Output     string
	Outcome    Outcome
	Trace      Trace
	Error      string
	StartedAt  time.Time
	FinishedAt time.Time
	Duration   time.Duration
	Metadata   map[string]any
}

type TrialResult

type TrialResult struct {
	Trial   Trial
	Grades  []GradeResult
	Passed  bool
	Message string
}

TrialResult combines one harness attempt with all grader outputs.

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int
	ToolCalls    int
	LLMCalls     int
}