evals

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2026 License: MIT Imports: 18 Imported by: 0

README

Pydantic Evals for Go

Evaluate non-deterministic functions — LLM calls, agents, pipelines — with type-safe, idiomatic Go.

Documentation: pkg.go.dev/github.com/Kludex/pydantic-evals-go

Source: github.com/Kludex/pydantic-evals-go

A Go port of Pydantic Evals. You bring a function to test; it runs over a set of cases, scores each result with evaluators you choose, and gives you a report you can print, diff, serialize, or ship to Pydantic Logfire.

It's like a test framework — but for code whose output you can't pin down with a single ==.

Install

go get github.com/Kludex/pydantic-evals-go

Example

Bind your input, output, and metadata types once with For, build a dataset, and run it. Point an OTLP exporter at Pydantic Logfire and every run shows up in Logfire's evaluation views — the experiment, each case, each task call, each evaluator:

package main

import (
	"context"
	"os"

	evals "github.com/Kludex/pydantic-evals-go"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	ctx := context.Background()

	// Send the evaluation trace tree to Logfire (set LOGFIRE_TOKEN to a write token).
	exporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpointURL("https://logfire-us.pydantic.dev/v1/traces"),
		otlptracehttp.WithHeaders(map[string]string{"Authorization": os.Getenv("LOGFIRE_TOKEN")}),
	)
	if err != nil {
		panic(err)
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	otel.SetTracerProvider(tp)
	defer tp.Shutdown(ctx) // flush before exit

	s := evals.For[string, string, any]()

	dataset := s.Dataset("capitals",
		s.Case("What is the capital of France?").Name("france").Expect("Paris"),
	).With(s.EqualsExpected())

	answer := func(ctx context.Context, question string) (string, error) {
		return "Paris", nil
	}

	report, err := dataset.Evaluate(ctx, answer, evals.Config{Name: "capitals-experiment"})
	if err != nil {
		panic(err)
	}
	report.Print(evals.RenderOptions{IncludeInput: true, IncludeOutput: true, IncludeAverages: true})
}

It prints the summary locally and sends the full trace to Logfire:

          Evaluation Summary: capitals-experiment
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Case ID  ┃ Inputs                         ┃ Outputs ┃ Assertions ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━┩
│ france   │ What is the capital of France? │ Paris   │ ✔          │
├──────────┼────────────────────────────────┼─────────┼────────────┤
│ Averages │                                │         │ 100.0% ✔   │
└──────────┴────────────────────────────────┴─────────┴────────────┘

Tracing is optional and zero-cost until you configure a provider — drop the exporter lines and the same program just prints the table. The OpenTelemetry packages live in their own modules:

go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
       go.opentelemetry.io/otel/sdk

Custom evaluators

An evaluator is any type with an Evaluate method. Return Score, Assertion, or Category:

type Closeness struct{}

func (Closeness) Evaluate(ctx context.Context, ec *evals.EvaluatorContext[string, string, any]) (evals.Output, error) {
	if ec.Output == ec.ExpectedOutput {
		return evals.Score(1.0), nil
	}
	return evals.Score(0.0).WithReason("not an exact match"), nil
}

Add it with .With(Closeness{}). Booleans become assertions, numbers become scores, strings become labels — and the report figures out the columns.

What's in the box

  • Built-in evaluators — Equals, EqualsExpected, Contains, IsInstance, MaxDuration.
  • Concurrent evaluation with Config{MaxConcurrency, Repeat, ...}.
  • Metrics & attributes recorded from inside your task (IncrementMetric, SetAttribute).
  • Lifecycle hooks for per-case setup and teardown.
  • YAML/JSON dataset save & load.
  • OpenTelemetry tracing, exportable to Logfire (shown above) — no-op until you configure a provider.

See the package documentation for the full guide and runnable examples, and examples/ for complete programs — including agents built on the OpenAI Go SDK, Genkit, and Eino, each evaluated and traced to Logfire.

Scope

This port covers the core of Pydantic Evals: cases, datasets, the evaluation engine, the non-LLM built-in evaluators, metrics, lifecycle hooks, reporting, serialization, and OpenTelemetry tracing. It omits the Python-infrastructure-specific pieces — the in-evaluator span-tree query API (and HasMatchingSpan), the LLMJudge evaluator, online evaluation, and statistical report evaluators. The Registry lets you add your own.

License

MIT, same as Pydantic Evals. See LICENSE.

Documentation

Overview

Package evals is a small, type-safe toolkit for evaluating non-deterministic functions — LLM calls, agents, retrieval pipelines, anything whose output you can't assert with a single ==.

It's a Go port of Pydantic Evals. You bring a function to test; the library runs it over a set of cases, scores each result with evaluators you choose, and gives you a report you can print, diff, serialize, or ship to an OpenTelemetry backend like Pydantic Logfire.

The mental model

Think of it as a test framework for probabilistic code:

  • A Case is one scenario: an input, an optional expected output, optional metadata.
  • A Dataset is a named collection of cases plus the evaluators that score them — your test suite.
  • A TaskFunc is the function under test: func(ctx, input) (output, error).
  • An Evaluator looks at a task's output and returns a result. Booleans become pass/fail assertions, numbers become scores, strings become labels.
  • An EvaluationReport collects every result, with per-case detail and an aggregate summary.

Everything is generic over three type parameters — the input (I), the output (O), and the metadata (M) — so your evaluators receive already-typed values with no casting.

Your first evaluation

Bind the three types once with For, then build a dataset and run it. Use [any] for any type parameter you don't need to pin down: 

s := evals.For[string, string, any]()

dataset := s.Dataset("capitals",
	s.Case("What is the capital of France?").Expect("Paris"),
).With(s.EqualsExpected())

answer := func(ctx context.Context, q string) (string, error) {
	return "Paris", nil
}

report, err := dataset.Evaluate(context.Background(), answer)
if err != nil {
	log.Fatal(err)
}
report.Print()

For returns a Suite that carries the type parameters, so you never repeat [string, string, any] at the call sites. See the Suite examples for the full builder.

Built-in evaluators

The package ships evaluators for the common deterministic checks: Equals, EqualsExpected, Contains, IsInstance, and MaxDuration. Add them to a dataset with Dataset.With, or to a single case with CaseBuilder.Eval.

Custom evaluators

An evaluator is any type with an Evaluate method. Return a result with Score, Assertion, or Category — add a “why” with WithReason, or return several named results at once with Named: 

type Closeness struct{}

func (Closeness) Evaluate(ctx context.Context, ec *evals.EvaluatorContext[string, string, any]) (evals.Output, error) {
	if ec.Output == ec.ExpectedOutput {
		return evals.Score(1.0), nil
	}
	return evals.Score(0.0).WithReason("not an exact match"), nil
}

That's the whole contract. The evaluator's name in the report defaults to its Go type name; implement NamedEvaluator to choose another, VersionedEvaluator to tag results with a version, or SpecEvaluator to make it serializable (see below).

Reports

Dataset.Evaluate returns an EvaluationReport. Render it as a table with EvaluationReport.Render, write it anywhere with EvaluationReport.Fprint, or print it to stdout with EvaluationReport.Print. RenderOptions controls which columns appear. EvaluationReport.Averages gives you the aggregate summary, and EvaluationReport.CaseGroups groups repeated runs when you set Config.Repeat.

Metrics and attributes

Inside a task, record extra data that flows into the report and to evaluators: IncrementMetric for numeric counters (averaged across cases) and SetAttribute for arbitrary values. Both are no-ops outside an evaluation, so instrumented code is safe to call anywhere.

Lifecycle hooks

For per-case setup and teardown — spin up a fixture, enrich the context before evaluators run, clean up afterwards — implement Lifecycle (embed BaseLifecycle for no-op defaults) and pass a factory to Dataset.EvaluateWithLifecycle.

Saving and loading datasets

Datasets serialize to YAML or JSON with Dataset.Save, and load back with LoadDataset. Loading rebuilds evaluators through a Registry: call Registry.RegisterDefaults for the built-ins, or Registry.Register your own. Custom evaluators that should round-trip implement SpecEvaluator.

OpenTelemetry and Logfire

Dataset.Evaluate emits an OpenTelemetry span tree — an experiment span, a span per case, a span per task run, and a span per evaluator — using the global tracer provider. It costs nothing until you configure one. Point an OTLP exporter at Logfire and call otel.SetTracerProvider to see your experiments in Logfire's evaluation views; see the Logfire example below for the full wiring. The repository's examples directory also has end-to-end programs evaluating agents built on the OpenAI Go SDK, Genkit, and Eino.

Example

Example shows the shortest path to an evaluation: bind the types once with For, build a dataset, run a task, and print the report. 

package main

import (
	"context"

	evals "github.com/Kludex/pydantic-evals-go"
)

// renderOpts hides durations so example output is deterministic. The title is
// omitted because its centering depends on terminal width.
var renderOpts = evals.RenderOptions{
	IncludeInput:    true,
	IncludeOutput:   true,
	IncludeAverages: true,
	OmitTitle:       true,
}

func main() {
	s := evals.For[string, string, any]()

	dataset := s.Dataset("capitals",
		s.Case("What is the capital of France?").Name("france").Expect("Paris"),
	).With(s.EqualsExpected())

	answer := func(_ context.Context, q string) (string, error) {
		return "Paris", nil
	}

	report, err := dataset.Evaluate(context.Background(), answer)
	if err != nil {
		panic(err)
	}
	report.Print(renderOpts)
}

Output:
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Case ID  ┃ Inputs                         ┃ Outputs ┃ Assertions ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━┩
│ france   │ What is the capital of France? │ Paris   │ ✔          │
├──────────┼────────────────────────────────┼─────────┼────────────┤
│ Averages │                                │         │ 100.0% ✔   │
└──────────┴────────────────────────────────┴─────────┴────────────┘
Example (Logfire)

Example_logfire wires an OTLP exporter to Pydantic Logfire so that the span tree emitted by Evaluate — an experiment span, a span per case, per task run, and per evaluator — shows up in Logfire's evaluation views.

Instrumentation uses the global tracer provider, so configuring one is all it takes; the library code is unchanged. Set LOGFIRE_TOKEN to a Logfire write token (the US endpoint is used here; use logfire-eu for the EU region). 

package main

import (
	"context"
	"os"

	evals "github.com/Kludex/pydantic-evals-go"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	ctx := context.Background()

	exporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpointURL("https://logfire-us.pydantic.dev/v1/traces"),
		otlptracehttp.WithHeaders(map[string]string{
			"Authorization": os.Getenv("LOGFIRE_TOKEN"),
		}),
	)
	if err != nil {
		panic(err)
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	otel.SetTracerProvider(tp)
	defer tp.Shutdown(ctx) // flush buffered spans before exit

	s := evals.For[string, string, any]()
	dataset := s.Dataset("capitals",
		s.Case("What is the capital of France?").Name("france").Expect("Paris"),
	).With(s.EqualsExpected())

	answer := func(ctx context.Context, q string) (string, error) {
		return "Paris", nil
	}

	report, err := dataset.Evaluate(ctx, answer, evals.Config{Name: "capitals-experiment"})
	if err != nil {
		panic(err)
	}
	report.Print()
}

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Assertion 

func Assertion(passed bool) single

Assertion returns a boolean assertion Output (pass/fail).

func Category 

func Category(label string) single

Category returns a categorical label Output.

func IncrementMetric 

func IncrementMetric(ctx context.Context, name string, amount float64)

IncrementMetric increments a numeric metric on the current task run by amount.

It is a no-op if ctx is not a task-run context. The accumulated metrics are exposed to evaluators via EvaluatorContext.Metrics and on the resulting ReportCase, and are averaged in the report summary.

Example

ExampleIncrementMetric records a metric from inside the task; it surfaces on the report and is averaged across cases. 

package main

import (
	"context"
	"fmt"

	evals "github.com/Kludex/pydantic-evals-go"
)

func main() {
	s := evals.For[string, string, any]()
	dataset := s.Dataset("metered", s.Case("hello").Name("c1"))

	report, _ := dataset.Evaluate(context.Background(),
		func(ctx context.Context, in string) (string, error) {
			evals.IncrementMetric(ctx, "chars", float64(len(in)))
			return in, nil
		})

	fmt.Println("chars:", report.Cases[0].Metrics["chars"])
}

Output:
chars: 5

func Score 

func Score(value float64) single

Score returns a numeric score Output. Use [single.WithReason] to explain it.

Example

ExampleScore demonstrates a custom evaluator that returns a numeric score. The evaluator's report name defaults to its Go type name ("matchAnswer"). 

package main

import (
	"context"
	"fmt"
	"strings"

	evals "github.com/Kludex/pydantic-evals-go"
)

// matchAnswer is a custom evaluator returning a graded score.
type matchAnswer struct{}

func (matchAnswer) Evaluate(_ context.Context, ec *evals.EvaluatorContext[string, string, any]) (evals.Output, error) {
	switch {
	case ec.Output == ec.ExpectedOutput:
		return evals.Score(1.0), nil
	case strings.Contains(strings.ToLower(ec.Output), strings.ToLower(ec.ExpectedOutput)):
		return evals.Score(0.8).WithReason("substring match"), nil
	default:
		return evals.Score(0.0), nil
	}
}

func main() {
	s := evals.For[string, string, any]()
	dataset := s.Dataset("quiz",
		s.Case("capital of France?").Name("france").Expect("Paris"),
	).With(matchAnswer{})

	report, _ := dataset.Evaluate(context.Background(),
		func(_ context.Context, q string) (string, error) { return "Paris", nil })

	for _, c := range report.Cases {
		fmt.Printf("%s: %v\n", c.Name, c.Scores["matchAnswer"].Value)
	}
}

Output:
france: 1

func ScoreInt 

func ScoreInt(value int) single

ScoreInt returns an integer score Output, rendered without decimals.

func SetAttribute 

func SetAttribute(ctx context.Context, name string, value any)

SetAttribute records an arbitrary attribute on the current task run.

It is a no-op if ctx is not a task-run context (i.e. not the context passed to a task during Dataset.Evaluate). The recorded attributes are exposed to evaluators via EvaluatorContext.Attributes and on the resulting ReportCase.

Types

type BaseLifecycle 

type BaseLifecycle[I, O, M any] struct{}

BaseLifecycle is a no-op Lifecycle suitable for embedding.

func (BaseLifecycle[I, O, M]) PrepareContext 

func (BaseLifecycle[I, O, M]) PrepareContext(_ context.Context, ec *EvaluatorContext[I, O, M]) (*EvaluatorContext[I, O, M], error)

func (BaseLifecycle[I, O, M]) Setup 

func (BaseLifecycle[I, O, M]) Setup(context.Context) error

func (BaseLifecycle[I, O, M]) Teardown 

func (BaseLifecycle[I, O, M]) Teardown(context.Context, *ReportCase[I, O, M], *ReportCaseFailure[I, O, M]) error

type Bool 

type Bool bool

Bool is a Scalar assertion result.

func (Bool) String 

func (b Bool) String() string

type Case 

type Case[I, O, M any] struct {
	// Name identifies the case in reports and for filtering. When empty, a
	// generic name ("Case N") is assigned during evaluation.
	Name string
	// Inputs to the task being evaluated.
	Inputs I
	// Metadata associated with the case, available to evaluators.
	Metadata M
	// HasMetadata reports whether Metadata was provided.
	HasMetadata bool
	// ExpectedOutput of the task, for comparison by evaluators.
	ExpectedOutput O
	// HasExpectedOutput reports whether ExpectedOutput was provided.
	HasExpectedOutput bool
	// Evaluators run only on this case, in addition to dataset-level evaluators.
	Evaluators []Evaluator[I, O, M]
}

Case is a single row of a Dataset.

Each case represents one test scenario. Inputs is required; Name, Metadata, ExpectedOutput, and case-specific Evaluators are optional. Use NewCase with option functions to construct a case, which sets the Has* flags consistently.

func NewCase 

func NewCase[I, O, M any](inputs I, opts ...CaseOption[I, O, M]) Case[I, O, M]

NewCase creates a Case with the given inputs and options.

type CaseBuilder 

type CaseBuilder[I, O, M any] struct {
	// contains filtered or unexported fields
}

CaseBuilder is a fluent builder for a Case, returned by Suite.Case.

func (*CaseBuilder[I, O, M]) Build 

func (b *CaseBuilder[I, O, M]) Build() Case[I, O, M]

Build returns the configured Case.

func (*CaseBuilder[I, O, M]) Eval 

func (b *CaseBuilder[I, O, M]) Eval(evaluators ...Evaluator[I, O, M]) *CaseBuilder[I, O, M]

Eval appends case-specific evaluators.

func (*CaseBuilder[I, O, M]) Expect 

func (b *CaseBuilder[I, O, M]) Expect(expected O) *CaseBuilder[I, O, M]

Expect sets the expected output.

func (*CaseBuilder[I, O, M]) Meta 

func (b *CaseBuilder[I, O, M]) Meta(metadata M) *CaseBuilder[I, O, M]

Meta sets the case metadata.

func (*CaseBuilder[I, O, M]) Name 

func (b *CaseBuilder[I, O, M]) Name(name string) *CaseBuilder[I, O, M]

Name sets the case name.

type CaseOption 

type CaseOption[I, O, M any] func(*Case[I, O, M])

CaseOption configures a Case built with NewCase.

func WithCaseEvaluators 

func WithCaseEvaluators[I, O, M any](evaluators ...Evaluator[I, O, M]) CaseOption[I, O, M]

WithCaseEvaluators appends case-specific evaluators.

func WithCaseName 

func WithCaseName[I, O, M any](name string) CaseOption[I, O, M]

WithCaseName sets the case name.

func WithExpectedOutput 

func WithExpectedOutput[I, O, M any](expected O) CaseOption[I, O, M]

WithExpectedOutput sets the case expected output and marks it as present.

func WithMetadata 

func WithMetadata[I, O, M any](metadata M) CaseOption[I, O, M]

WithMetadata sets the case metadata and marks it as present.

type Config 

type Config struct {
	// Name of the experiment; defaults to TaskName.
	Name string
	// TaskName is the displayed task name; defaults to "task".
	TaskName string
	// MaxConcurrency limits concurrent case evaluations; <= 0 means unlimited.
	MaxConcurrency int
	// Repeat runs each case this many times (default/zero is treated as 1).
	// When > 1, results are grouped by the original case name for aggregation.
	Repeat int
	// Metadata is arbitrary experiment metadata recorded on the report.
	Metadata map[string]any
}

Config configures a call to Dataset.Evaluate. The zero value is valid: it runs each case once, unbounded, with the experiment named after the task.

Config is intentionally non-generic so that Evaluate calls need no type parameters. For per-case lifecycle hooks, use Dataset.EvaluateWithLifecycle.

Example (Repeat)

ExampleConfig_repeat runs each case several times and reads the aggregate. 

package main

import (
	"context"
	"fmt"

	evals "github.com/Kludex/pydantic-evals-go"
)

func main() {
	s := evals.For[string, string, any]()
	dataset := s.Dataset("repeated",
		s.Case("hi").Name("c1").Expect("hi"),
	).With(s.EqualsExpected())

	report, _ := dataset.Evaluate(context.Background(),
		func(_ context.Context, in string) (string, error) { return in, nil },
		evals.Config{Repeat: 3})

	groups := report.CaseGroups()
	fmt.Printf("%d run(s) of %q\n", len(groups[0].Runs), groups[0].Name)
}

Output:
3 run(s) of "c1"

type Contains 

type Contains[I, O, M any] struct {
	// Value to look for within the output.
	Value any
	// CaseSensitive controls string comparison (default false: case-insensitive).
	CaseSensitive bool
	// AsStrings forces both sides to be compared as strings.
	AsStrings bool
	// Name overrides the result name in reports (default "Contains").
	Name string
}

Contains checks whether the output contains the provided value.

For strings it checks substring containment; for slices/arrays, element membership; for maps, that every key/value in Value (when Value is a map) is present, or that Value is a key. CaseSensitive applies only to string checks.

func (Contains[I, O, M]) Evaluate 

func (c Contains[I, O, M]) Evaluate(_ context.Context, ec *EvaluatorContext[I, O, M]) (Output, error)

func (Contains[I, O, M]) EvaluationName 

func (c Contains[I, O, M]) EvaluationName() string

func (Contains[I, O, M]) Spec 

func (c Contains[I, O, M]) Spec() EvaluatorSpec

type Dataset 

type Dataset[I, O, M any] struct {
	// Name of the dataset.
	Name string
	// Cases in the dataset.
	Cases []Case[I, O, M]
	// Evaluators applied to all cases.
	Evaluators []Evaluator[I, O, M]
}

Dataset is a collection of test [Case]s evaluated against a task.

func LoadDataset 

func LoadDataset[I, O, M any](data []byte, reg *Registry[I, O, M], opts LoadOptions[I, O, M]) (*Dataset[I, O, M], error)

LoadDataset parses a Dataset from YAML or JSON bytes, constructing case-level and dataset-level evaluators via the registry.

Inputs, expected outputs and metadata are decoded with the Decode* hooks in opts; when a hook is nil the deserialized value is converted to the target type via a JSON round-trip.

Example

ExampleLoadDataset rebuilds a dataset from YAML through a registry of the built-in evaluators. 

package main

import (
	"fmt"

	evals "github.com/Kludex/pydantic-evals-go"
)

func main() {
	data := []byte(`
name: greetings
cases:
  - name: c1
    inputs: hi
    expected_output: hello
evaluators:
  - EqualsExpected
`)

	reg := evals.NewRegistry[string, string, any]()
	reg.RegisterDefaults()

	dataset, err := evals.LoadDataset(data, reg, evals.LoadOptions[string, string, any]{})
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s: %d case(s), %d evaluator(s)\n", dataset.Name, len(dataset.Cases), len(dataset.Evaluators))
}

Output:
greetings: 1 case(s), 1 evaluator(s)

func NewDataset 

func NewDataset[I, O, M any](name string, cases []Case[I, O, M], evaluators ...Evaluator[I, O, M]) (*Dataset[I, O, M], error)

NewDataset creates a Dataset, validating that case names are unique.

func (*Dataset[I, O, M]) AddCase 

func (d *Dataset[I, O, M]) AddCase(c Case[I, O, M]) error

AddCase appends a case, validating that its name is unique.

func (*Dataset[I, O, M]) AddEvaluator 

func (d *Dataset[I, O, M]) AddEvaluator(e Evaluator[I, O, M])

AddEvaluator adds an evaluator to all cases in the dataset.

func (*Dataset[I, O, M]) AddEvaluatorForCase 

func (d *Dataset[I, O, M]) AddEvaluatorForCase(caseName string, e Evaluator[I, O, M]) error

AddEvaluatorForCase adds an evaluator to the case with the given name. It returns an error if no such case exists.

func (*Dataset[I, O, M]) Evaluate 

func (d *Dataset[I, O, M]) Evaluate(ctx context.Context, task TaskFunc[I, O], cfg ...Config) (*EvaluationReport[I, O, M], error)

Evaluate runs the task against every case in the dataset, applying the dataset-level and case-level evaluators, and returns an EvaluationReport.

Cases run concurrently (bounded by cfg.MaxConcurrency). A task error produces a ReportCaseFailure; an evaluator error produces an EvaluatorFailure on the case rather than failing the case. Evaluate itself only returns an error if the config is invalid or ctx is cancelled.

cfg is optional; pass at most one Config. For per-case lifecycle hooks, use Dataset.EvaluateWithLifecycle.

func (*Dataset[I, O, M]) EvaluateWithLifecycle 

func (d *Dataset[I, O, M]) EvaluateWithLifecycle(ctx context.Context, task TaskFunc[I, O], newLifecycle func(c Case[I, O, M]) Lifecycle[I, O, M], cfg ...Config) (*EvaluationReport[I, O, M], error)

EvaluateWithLifecycle is like Dataset.Evaluate but invokes newLifecycle once per case to obtain its Lifecycle hooks (Setup, PrepareContext, Teardown).

func (*Dataset[I, O, M]) Save 

func (d *Dataset[I, O, M]) Save(opts SaveOptions) ([]byte, error)

Save serializes the dataset to YAML or JSON bytes, using the short form for each evaluator spec.

Example

ExampleDataset_Save serializes a dataset to YAML, using the short form for each evaluator. 

package main

import (
	"fmt"

	evals "github.com/Kludex/pydantic-evals-go"
)

func main() {
	s := evals.For[string, string, any]()
	dataset := s.Dataset("greetings",
		s.Case("hi").Name("c1").Expect("hello"),
	).With(s.EqualsExpected(), s.IsInstance("string"))

	out, _ := dataset.Save(evals.SaveOptions{Format: "yaml"})
	fmt.Print(string(out))
}

Output:
name: greetings
cases:
  - name: c1
    inputs: hi
    expected_output: hello
evaluators:
  - EqualsExpected
  - IsInstance: string

func (*Dataset[I, O, M]) With 

func (d *Dataset[I, O, M]) With(evaluators ...Evaluator[I, O, M]) *Dataset[I, O, M]

With appends dataset-level evaluators and returns the dataset for chaining.

type Equals 

type Equals[I, O, M any] struct {
	// Value to compare the output against.
	Value O
	// Name overrides the result name in reports (default "Equals").
	Name string
}

Equals checks whether the output exactly equals the provided value.

func (Equals[I, O, M]) Evaluate 

func (e Equals[I, O, M]) Evaluate(_ context.Context, ec *EvaluatorContext[I, O, M]) (Output, error)

func (Equals[I, O, M]) EvaluationName 

func (e Equals[I, O, M]) EvaluationName() string

func (Equals[I, O, M]) Spec 

func (e Equals[I, O, M]) Spec() EvaluatorSpec

type EqualsExpected 

type EqualsExpected[I, O, M any] struct {
	// Name overrides the result name in reports (default "EqualsExpected").
	Name string
}

EqualsExpected checks whether the output exactly equals the case's expected output. If the case has no expected output, it yields no result.

func (EqualsExpected[I, O, M]) Evaluate 

func (e EqualsExpected[I, O, M]) Evaluate(_ context.Context, ec *EvaluatorContext[I, O, M]) (Output, error)

func (EqualsExpected[I, O, M]) EvaluationName 

func (e EqualsExpected[I, O, M]) EvaluationName() string

func (EqualsExpected[I, O, M]) Spec 

func (e EqualsExpected[I, O, M]) Spec() EvaluatorSpec

type EvaluationReason 

type EvaluationReason struct {
	Value  Scalar
	Reason string
}

EvaluationReason is a Scalar result with an optional explanation.

type EvaluationReport 

type EvaluationReport[I, O, M any] struct {
	// Name of the report (the experiment name).
	Name string
	// Cases that were evaluated successfully.
	Cases []ReportCase[I, O, M]
	// Failures are cases whose task execution raised an error.
	Failures []ReportCaseFailure[I, O, M]
	// ExperimentMetadata associated with this experiment, if any.
	ExperimentMetadata map[string]any
}

EvaluationReport is the result of evaluating a task over a Dataset.

func (*EvaluationReport[I, O, M]) Averages 

func (r *EvaluationReport[I, O, M]) Averages() *ReportCaseAggregate

Averages returns the overall summary aggregate for the report, or nil if there are no cases. For multi-run experiments it averages the per-group summaries.

func (*EvaluationReport[I, O, M]) CaseGroups 

func (r *EvaluationReport[I, O, M]) CaseGroups() []ReportCaseGroup[I, O, M]

CaseGroups groups cases by SourceCaseName and computes per-group aggregates. It returns nil for a single-run experiment (no case has a SourceCaseName).

func (*EvaluationReport[I, O, M]) Fprint 

func (r *EvaluationReport[I, O, M]) Fprint(w io.Writer, opts ...RenderOptions)

Fprint writes the rendered report to w.

func (*EvaluationReport[I, O, M]) Print 

func (r *EvaluationReport[I, O, M]) Print(opts ...RenderOptions)

Print writes the rendered report to standard output. To write elsewhere (a file, a buffer, a test), use EvaluationReport.Fprint.

func (*EvaluationReport[I, O, M]) Render 

func (r *EvaluationReport[I, O, M]) Render(opts ...RenderOptions) string

Render returns the report rendered as a box-drawing table string.

type EvaluationResult 

type EvaluationResult struct {
	Name             string
	Value            Scalar
	Reason           string
	Source           EvaluatorSpec
	EvaluatorVersion string
}

EvaluationResult is the detail of a single named evaluation result.

type Evaluator 

type Evaluator[I, O, M any] interface {
	// Evaluate scores the task output described by ec, returning an [Output]
	// built with [Score], [Assertion], [Category] or [Named].
	Evaluate(ctx context.Context, ec *EvaluatorContext[I, O, M]) (Output, error)
}

Evaluator assesses the result of running a task against a single Case.

Implementations only need Evaluate. The name used for an evaluator's results in reports defaults to its Go type name; implement NamedEvaluator to override it, SpecEvaluator to make the evaluator serializable to YAML/JSON, and VersionedEvaluator to tag its results with a version.

type EvaluatorContext 

type EvaluatorContext[I, O, M any] struct {
	// Name is the name of the case, if any.
	Name string
	// Inputs is the input provided to the task for this case.
	Inputs I
	// Metadata associated with the case.
	Metadata M
	// HasMetadata reports whether Metadata was set on the case.
	HasMetadata bool
	// ExpectedOutput is the expected output for the case.
	ExpectedOutput O
	// HasExpectedOutput reports whether ExpectedOutput was set on the case.
	HasExpectedOutput bool
	// Output is the actual output produced by the task.
	Output O
	// Duration is how long the task run took.
	Duration time.Duration
	// Attributes recorded during the task run via [SetAttribute].
	Attributes map[string]any
	// Metrics recorded during the task run via [IncrementMetric].
	Metrics map[string]float64
}

EvaluatorContext carries everything an Evaluator needs to score one case.

It holds the case inputs, the actual and expected outputs, metadata, the task duration, and any attributes and metrics recorded during the task run via SetAttribute and IncrementMetric.

type EvaluatorFactory 

type EvaluatorFactory[I, O, M any] func(spec EvaluatorSpec) (Evaluator[I, O, M], error)

EvaluatorFactory builds an Evaluator from a deserialized EvaluatorSpec.

type EvaluatorFailure 

type EvaluatorFailure struct {
	Name             string
	ErrorMessage     string
	Source           EvaluatorSpec
	EvaluatorVersion string
	ErrorType        string
}

EvaluatorFailure represents an error raised while running an Evaluator.

type EvaluatorSpec 

type EvaluatorSpec struct {
	// Name of the evaluator class to construct.
	Name string
	// Args holds the single positional argument, if any (length 0 or 1).
	Args []any
	// Kwargs holds the keyword arguments, if any.
	Kwargs map[string]any
}

EvaluatorSpec is the serializable specification of an evaluator.

It supports three short forms when (de)serialized to YAML/JSON, matching the Python `EvaluatorSpec`:

  • "MyEvaluator" — the bare name, when there are no arguments;
  • {"MyEvaluator": arg} — a single positional argument;
  • {"MyEvaluator": {k1: v1, k2: v2}} — keyword arguments.

Args holds a single positional argument (length 0 or 1); Kwargs holds keyword arguments. At most one of them is non-empty.

func NewSpec 

func NewSpec(name string) EvaluatorSpec

NewSpec builds an EvaluatorSpec with no arguments.

func NewSpecArg 

func NewSpecArg(name string, arg any) EvaluatorSpec

NewSpecArg builds an EvaluatorSpec with a single positional argument.

func NewSpecKwargs 

func NewSpecKwargs(name string, kwargs map[string]any) EvaluatorSpec

NewSpecKwargs builds an EvaluatorSpec with keyword arguments.

type Float 

type Float float64

Float is a Scalar score result. Inf and NaN are not valid scores and are rejected when an evaluator's output is normalized.

func (Float) String 

func (f Float) String() string

type Int 

type Int int

Int is a Scalar score result.

func (Int) String 

func (i Int) String() string

type IsInstance 

type IsInstance[I, O, M any] struct {
	// TypeName is the unqualified Go type name the output must have.
	TypeName string
	// Name overrides the result name in reports (default "IsInstance").
	Name string
}

IsInstance checks whether the output's runtime type name matches TypeName.

The match is against the unqualified type name (e.g. "string", "int", or a struct's name), making it useful when O is an interface type like `any`.

func (IsInstance[I, O, M]) Evaluate 

func (e IsInstance[I, O, M]) Evaluate(_ context.Context, ec *EvaluatorContext[I, O, M]) (Output, error)

func (IsInstance[I, O, M]) EvaluationName 

func (e IsInstance[I, O, M]) EvaluationName() string

func (IsInstance[I, O, M]) Spec 

func (e IsInstance[I, O, M]) Spec() EvaluatorSpec

type Label 

type Label string

Label is a Scalar categorical result.

func (Label) String 

func (l Label) String() string

type Lifecycle 

type Lifecycle[I, O, M any] interface {
	// Setup runs before task execution.
	Setup(ctx context.Context) error
	// PrepareContext runs after the task, before evaluators, and may enrich the
	// evaluator context (e.g. add metrics or attributes).
	PrepareContext(ctx context.Context, ec *EvaluatorContext[I, O, M]) (*EvaluatorContext[I, O, M], error)
	// Teardown runs after evaluators complete. result is nil if the case ended
	// without producing a report (e.g. cancellation).
	Teardown(ctx context.Context, result *ReportCase[I, O, M], failure *ReportCaseFailure[I, O, M]) error
}

Lifecycle provides per-case setup, context preparation, and teardown hooks. A fresh instance is created for each case via the factory passed to [WithLifecycle]. All methods are optional; embed BaseLifecycle to get no-op defaults.

type LoadOptions 

type LoadOptions[I, O, M any] struct {
	// Format is "yaml" or "json". When empty, it is inferred from the data: a
	// leading '{' implies JSON, otherwise YAML.
	Format string
	// DefaultName is used when the serialized data has no name.
	DefaultName string
	// DecodeInputs converts a deserialized inputs value into I.
	DecodeInputs func(any) (I, error)
	// DecodeOutput converts a deserialized expected_output value into O.
	DecodeOutput func(any) (O, error)
	// DecodeMetadata converts a deserialized metadata value into M.
	DecodeMetadata func(any) (M, error)
}

LoadOptions configures dataset loading.

type MaxDuration 

type MaxDuration[I, O, M any] struct {
	// Max is the inclusive upper bound on the task duration.
	Max time.Duration
}

MaxDuration checks whether the task ran in at most Max.

func (MaxDuration[I, O, M]) Evaluate 

func (e MaxDuration[I, O, M]) Evaluate(_ context.Context, ec *EvaluatorContext[I, O, M]) (Output, error)

func (MaxDuration[I, O, M]) Spec 

func (e MaxDuration[I, O, M]) Spec() EvaluatorSpec

type NamedEvaluator 

type NamedEvaluator interface {
	EvaluationName() string
}

NamedEvaluator is an optional interface an Evaluator may implement to override the name used for its results in reports.

type Output 

type Output interface {
	// contains filtered or unexported methods
}

Output is the value an Evaluator returns from Evaluate.

Build one with Score, Assertion or Category for a single result, or Named for several named results from one evaluator. A bare Score becomes a numeric score in the report, Assertion a boolean assertion, and Category a string label.

func Named 

func Named(pairs ...any) Output

Named groups several results from one evaluator under explicit names. Pass alternating name and Output arguments; each Output must be a single result (from Score/Assertion/Category), not another Named. 

return evals.Named(
    "length", evals.Score(0.8).WithReason("close"),
    "sentiment", evals.Category("neutral"),
), nil

Named panics if given an odd number of arguments or a non-string name, which are programming errors rather than runtime conditions.

Example

ExampleNamed shows one evaluator returning several named results at once. 

package main

import (
	"context"
	"fmt"

	evals "github.com/Kludex/pydantic-evals-go"
)

func main() {
	type quality struct{}
	_ = quality{}

	eval := evals.For[string, string, any]()
	dataset := eval.Dataset("multi",
		eval.Case("hello").Name("c1"),
	).With(namedEvaluator{})

	report, _ := dataset.Evaluate(context.Background(),
		func(_ context.Context, in string) (string, error) { return in, nil })

	c := report.Cases[0]
	fmt.Println("score:", c.Scores["length"].Value)
	fmt.Println("label:", c.Labels["shape"].Value)
}

type namedEvaluator struct{}

func (namedEvaluator) Evaluate(_ context.Context, ec *evals.EvaluatorContext[string, string, any]) (evals.Output, error) {
	shape := "short"
	if len(ec.Output) > 10 {
		shape = "long"
	}
	return evals.Named(
		"length", evals.ScoreInt(len(ec.Output)),
		"shape", evals.Category(shape),
	), nil
}

Output:
score: 5
label: short

func NoResult 

func NoResult() Output

NoResult returns an Output that records nothing, for evaluators that conditionally produce no result (e.g. when there is no expected output).

type Registry 

type Registry[I, O, M any] struct {
	// contains filtered or unexported fields
}

Registry maps evaluator names to factories used when loading a Dataset from YAML or JSON. Construct one with NewRegistry and register built-ins via Registry.RegisterDefaults or custom evaluators via Registry.Register.

func NewRegistry 

func NewRegistry[I, O, M any]() *Registry[I, O, M]

NewRegistry returns an empty Registry.

func (*Registry[I, O, M]) Register 

func (r *Registry[I, O, M]) Register(name string, factory EvaluatorFactory[I, O, M])

Register associates an evaluator name with a factory. It overrides any existing registration for that name.

func (*Registry[I, O, M]) RegisterDefaults 

func (r *Registry[I, O, M]) RegisterDefaults()

RegisterDefaults registers the built-in evaluators that do not require an LLM or telemetry: Equals, EqualsExpected, Contains, IsInstance and MaxDuration.

type RenderOptions 

type RenderOptions struct {
	IncludeInput          bool
	IncludeMetadata       bool
	IncludeExpectedOutput bool
	IncludeOutput         bool
	IncludeDurations      bool
	IncludeTotalDuration  bool
	IncludeAverages       bool
	IncludeReasons        bool

	// Title overrides the default "Evaluation Summary: <name>" title. An empty
	// Title uses the default; set OmitTitle to render no title at all.
	Title string
	// OmitTitle renders the table without a title row.
	OmitTitle bool
}

RenderOptions configures EvaluationReport.Render and EvaluationReport.Print.

func DefaultRenderOptions 

func DefaultRenderOptions() RenderOptions

DefaultRenderOptions returns the options used when none are supplied: durations and averages included, matching the Python defaults.

type ReportCase 

type ReportCase[I, O, M any] struct {
	// Name of the case in the report.
	Name string
	// Inputs to the task.
	Inputs I
	// Metadata associated with the case.
	Metadata    M
	HasMetadata bool
	// ExpectedOutput of the task.
	ExpectedOutput    O
	HasExpectedOutput bool
	// Output produced by the task.
	Output O

	// Metrics recorded during the task run.
	Metrics map[string]float64
	// Attributes recorded during the task run.
	Attributes map[string]any

	// Scores are evaluation results with numeric values.
	Scores map[string]EvaluationResult
	// Labels are evaluation results with string values.
	Labels map[string]EvaluationResult
	// Assertions are evaluation results with boolean values.
	Assertions map[string]EvaluationResult

	// TaskDuration is the duration of the task run.
	TaskDuration time.Duration
	// TotalDuration includes evaluator execution time.
	TotalDuration time.Duration

	// SourceCaseName is the original case name before run-indexing, used as the
	// aggregation key for multi-run (repeat) experiments. Empty when repeat == 1.
	SourceCaseName string

	// EvaluatorFailures are evaluators that errored for this case.
	EvaluatorFailures []EvaluatorFailure
}

ReportCase is a single successfully-evaluated case in an EvaluationReport.

type ReportCaseAggregate 

type ReportCaseAggregate struct {
	Name string

	Scores  map[string]float64
	Labels  map[string]map[string]float64
	Metrics map[string]float64
	// Assertions is the pass rate across all assertions, or nil if there were none.
	Assertions    *float64
	TaskDuration  time.Duration
	TotalDuration time.Duration
}

ReportCaseAggregate summarizes a set of cases by averaging their quantitative attributes. The synthetic "Averages" row in a report is a ReportCaseAggregate.

type ReportCaseFailure 

type ReportCaseFailure[I, O, M any] struct {
	Name              string
	Inputs            I
	Metadata          M
	HasMetadata       bool
	ExpectedOutput    O
	HasExpectedOutput bool

	// ErrorMessage is the message of the error that caused the failure.
	ErrorMessage string
	// ErrorType is the Go type name of the error.
	ErrorType string

	// SourceCaseName mirrors [ReportCase.SourceCaseName].
	SourceCaseName string
}

ReportCaseFailure is a case whose task execution raised an error.

type ReportCaseGroup 

type ReportCaseGroup[I, O, M any] struct {
	Name              string
	Inputs            I
	Metadata          M
	HasMetadata       bool
	ExpectedOutput    O
	HasExpectedOutput bool

	Runs     []ReportCase[I, O, M]
	Failures []ReportCaseFailure[I, O, M]
	Summary  ReportCaseAggregate
}

ReportCaseGroup is the grouped result of running the same case multiple times (when repeat > 1). It is a computed view obtained via EvaluationReport.CaseGroups.

type SaveOptions 

type SaveOptions struct {
	// Format is "yaml" or "json". Defaults to "yaml".
	Format string
	// Schema, when set, is written as the "$schema" reference.
	Schema string
}

SaveOptions configures dataset serialization.

type Scalar 

type Scalar interface {

	// String returns the human-readable form used when rendering reports.
	String() string
	// contains filtered or unexported methods
}

Scalar is the most primitive output allowed from an Evaluator.

The concrete kinds are Bool, Int, Float and Label (a string). A Bool is treated as an assertion, an Int or finite Float as a score, and a Label as a categorical label. This mirrors the Python `EvaluationScalar` union of `bool | int | float | str`.

type SpecEvaluator 

type SpecEvaluator interface {
	Spec() EvaluatorSpec
}

SpecEvaluator is an optional interface an Evaluator may implement so it can be serialized to and reconstructed from a Dataset file. Evaluators that are only used in-memory do not need it.

type Suite 

type Suite[I, O, M any] struct{}

Suite is a typed entry point that captures the inputs, output and metadata types once so you don't repeat them at every call site. Obtain one with For: 

s := evals.For[string, string, any]()
c := s.Case("hi").Name("greeting").Expect("hello")
ds, _ := s.Dataset("quiz", c).With(s.IsInstance("string"))

Suite is a zero-size value; create as many as you like.

func For 

func For[I, O, M any]() Suite[I, O, M]

For returns a Suite bound to the given inputs (I), output (O) and metadata (M) types.

func (Suite[I, O, M]) Case 

func (Suite[I, O, M]) Case(inputs I) *CaseBuilder[I, O, M]

Case starts building a Case with the given inputs. Chain Name/Expect/Meta/ Eval to configure it; the CaseBuilder is usable directly as a Case argument or via CaseBuilder.Build.

func (Suite[I, O, M]) Contains 

func (Suite[I, O, M]) Contains(value any) Contains[I, O, M]

Contains builds a Contains evaluator bound to this suite's types.

func (Suite[I, O, M]) Dataset 

func (Suite[I, O, M]) Dataset(name string, cases ...*CaseBuilder[I, O, M]) *Dataset[I, O, M]

Dataset creates a Dataset from the given case builders. Add dataset-level evaluators with Dataset.With. It panics if two cases share a name, which is a test-definition error; use NewDataset if you need to handle that as a value.

func (Suite[I, O, M]) Equals 

func (Suite[I, O, M]) Equals(value O) Equals[I, O, M]

Equals builds an Equals evaluator bound to this suite's types.

func (Suite[I, O, M]) EqualsExpected 

func (Suite[I, O, M]) EqualsExpected() EqualsExpected[I, O, M]

EqualsExpected builds an EqualsExpected evaluator bound to this suite's types.

func (Suite[I, O, M]) IsInstance 

func (Suite[I, O, M]) IsInstance(typeName string) IsInstance[I, O, M]

IsInstance builds an IsInstance evaluator bound to this suite's types.

func (Suite[I, O, M]) MaxDuration 

func (Suite[I, O, M]) MaxDuration(max time.Duration) MaxDuration[I, O, M]

MaxDuration builds a MaxDuration evaluator bound to this suite's types.

type TaskFunc 

type TaskFunc[I, O any] func(ctx context.Context, inputs I) (O, error)

TaskFunc is the function under evaluation. It receives the task-run context (on which SetAttribute and IncrementMetric may be called) and a case's inputs, and returns the produced output.

type VersionedEvaluator 

type VersionedEvaluator interface {
	EvaluatorVersion() string
}

VersionedEvaluator is an optional interface an Evaluator may implement to tag its results with a version string for downstream filtering.

Source Files

View all Source files

Directories

Path Synopsis
examples
capitals command
Command capitals demonstrates evaluating a simple function with the evals library.
 Command capitals demonstrates evaluating a simple function with the evals library.
logfire command
Command logfire runs a multi-faceted evaluation and exports the traces to Pydantic Logfire via OTLP, so the experiment shows up in Logfire's eval views.
 Command logfire runs a multi-faceted evaluation and exports the traces to Pydantic Logfire via OTLP, so the experiment shows up in Logfire's eval views.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.