evalgo

README

Eval-Go

A native-Go framework for evaluating LLM, RAG, and agent outputs — 27 metrics, synthetic-data generation, red-teaming, caching, cost tracking, and cross-version regression gating, as both a library and a CLI.

Python dominates LLM evaluation with heavy frameworks (DeepEval, RAGAS, DeepTeam). Eval-Go covers the same evaluation surface but leans on Go's strengths — concurrency, determinism, zero-cost heuristics, and go test. Metrics split into families:

• Deterministic (code-based): JSON validity, regex, citation, tool-set match, refusal detection. Fast, free, no LLM — run them first as cheap gates.
• Semantic (LLM-as-a-judge): RAG, agent, safety, conversational, and red-team metrics, decomposed RAGAS-style (extract atomic units → verify each) rather than one vague 0–1 score.

The core package has zero third-party dependencies (stdlib only). The judge adapter for agent-go lives in a separate package (./llmjudge), so deterministic-only users pay no dependency cost.

Agent and conversational metrics are record-then-evaluate: they score a run the system already produced (carried on the Sample), so any framework — or none — can emit the data, with no live @observe-style instrumentation.

It also ships AgentBench — a tool-calling benchmark that ranks LLMs (local Ollama or cloud) across 30 real-world scenarios on tool choice, conditional orchestration, distractor resistance, and knowing when not to call a tool.

New here? docs/overview.md explains the two layers (scoring an output vs. driving an agent), which one answers which question, and where each approach stops being valid. This README is the reference.

Install

go get github.com/liliang-cn/eval-go                          # library
go install github.com/liliang-cn/eval-go/cmd/evalgo@latest    # CLI

Quick start

# deterministic metrics — free, offline
evalgo -d examples/golden.json -m nonempty,citation,valid_json

# add LLM-as-a-judge (any OpenAI-compatible endpoint)
export LLM_BASE_URL=... LLM_API_KEY=... LLM_MODEL=...
evalgo -d examples/golden.json --judge env \
  -m faithfulness,answer_relevancy,context_precision,rubric \
  -f json -o report.json --fail-under 0.8

The dataset is a JSON / JSONL / CSV array of samples. Exit code is 1 when the CI gate fails (--fail-under, or any failure by default) and 2 on a config error.

Metrics

Names below are the CLI -m identifiers. A few deterministic metrics are library-only (they take constructor arguments) and are marked as such.

Metric	Category	What it checks
nonempty	deterministic	output has non-whitespace content
valid_json	deterministic	output is structurally valid JSON
exact_match	deterministic	trimmed output equals expected
citation	deterministic	output carries a [SOURCE]-style citation
JSONHasFields, Contains, MatchesRegex, ForbidsRegex	deterministic (library)	required keys / substring / format / safety boundary
rubric	semantic	G-Eval-style pass/score against a per-sample rubric
faithfulness	RAG	answer's claims are grounded in context (catches hallucination)
answer_relevancy	RAG	answer statements actually address the question
context_precision	RAG	retrieved chunks are relevant to the question
contextual_recall	RAG	context covers everything the reference answer needs
contextual_relevancy	RAG	statements in the context are relevant to the question
tool_correctness	agent (deterministic)	invoked tools match the expected set
argument_correctness	agent	each tool call's arguments are correct for the task
task_completion	agent	the agent actually accomplished the task
step_efficiency	agent	goal reached without redundant / repeated steps
plan_quality	agent	the agent's plan is logical, complete, efficient
plan_adherence	agent	the agent's execution followed its own plan
hallucination	safety	output does not contradict the trusted context
bias, toxicity	safety	output statements are unbiased / non-toxic
pii_leakage	safety	output leaks no personally identifiable information
summarization	quality	summary is faithful to and covers the source text
conversation_completeness	multi-turn	assistant fulfilled the user's intentions across turns
knowledge_retention	multi-turn	assistant remembered facts given earlier
conversation_relevancy	multi-turn	each assistant turn is relevant in context
role_adherence	multi-turn	assistant stayed in its assigned persona
attack_resistance	red-team	system resisted (refused / didn't leak) an adversarial input
refusal	red-team (deterministic)	output contains refusal / deflection language

Plus DAG (library) — build a metric from a decision tree of judge questions instead of one score; see Custom metrics.

The Sample

One JSON object per row. Fields are optional; a metric whose inputs are absent skips and passes, so you can mix RAG, agent, and conversational rows in one file.

{
  "name": "weather-agent",
  "input": "What's the weather in Tokyo tomorrow?",
  "output": "Tokyo tomorrow: sunny, high around 24C.",
  "expected": "...",                          // reference answer (exact_match, contextual_recall)
  "context": ["..."],                          // retrieved chunks (RAG / hallucination)
  "rubric": "States the forecast accurately.", // for the rubric judge
  "meta": {"team": "search"},                  // labels; filterable with --where

  // agent run (record-then-evaluate)
  "plan": "1) geocode, 2) fetch forecast, 3) report",
  "trajectory": ["geocoded Tokyo", "fetched forecast", "reported"],
  "tool_calls": [{"name": "geocode", "args": {"query": "Tokyo"}}],
  "expected_tools": ["geocode", "weather"],

  // multi-turn conversation
  "turns": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
  "persona": "a concise banking support agent"
}

Library quick start

import (
    evalgo "github.com/liliang-cn/eval-go"
    "github.com/liliang-cn/eval-go/llmjudge"
)

judge, _ := llmjudge.FromEnv()                 // LLM_BASE_URL / LLM_API_KEY / LLM_MODEL
judge = evalgo.RateLimit(judge, 4, 2)          // token-bucket; avoid 429s
judge = evalgo.Cache(judge, ".eval-cache")     // disk cache; re-runs spend no tokens

suite := evalgo.Suite{
    Concurrency: 4,
    Metrics: []evalgo.Metric{
        evalgo.CitationPresent(),              // deterministic
        evalgo.Faithfulness(judge),            // RAG
        evalgo.AnswerRelevancy(judge, 0.5),
        evalgo.RubricJudge(judge, 0.7),
    },
    Samples: []evalgo.Sample{{
        Name:    "grounded-answer",
        Input:   "What is the savings account interest rate?",
        Context: []string{"The savings account pays 0.30% below 50000 and 0.55% above."},
        Output:  "The rate is tiered: 0.30% below 50000 and 0.55% above [KB-001].",
    }},
}

report := suite.Run(context.Background())
report.WriteConsole(os.Stdout)
if report.Failed() { os.Exit(1) }              // CI gate

Samples run concurrently (bounded by Concurrency); metrics within a sample run sequentially so a shared rate-limited judge paces cleanly.

go test integration

RunT turns a suite into table-driven, parallel subtests so go test exit codes and CI reporting work for free. Gate semantic evals behind an env flag so plain go test spends no tokens:

func TestRAG(t *testing.T) {
    if os.Getenv("RUN_EVALS") != "true" { t.Skip("set RUN_EVALS=true") }
    judge, _ := llmjudge.FromEnv()
    evalgo.RunT(t, evalgo.Suite{ /* Metrics, Samples */ })
}

go test ./...                                   # deterministic only, free & offline
RUN_EVALS=true LLM_BASE_URL=... LLM_API_KEY=... LLM_MODEL=... go test ./... -v

Agent evaluation

Agent metrics score a recorded run carried on the Sample (plan, trajectory, tool_calls, expected_tools) — across the same three layers DeepEval names: action (tool_correctness, argument_correctness), execution (task_completion, step_efficiency), reasoning (plan_quality, plan_adherence).

evalgo -d examples/agent.json -m tool_correctness          # free, offline
evalgo -d examples/agent.json --judge env \
  -m tool_correctness,argument_correctness,task_completion,step_efficiency,plan_quality,plan_adherence

Run the agent (end-to-end)

The metrics above score a run you already recorded. eval-go can also run the agent for you: define Tasks, plug the system under test in as a Target, and Bench executes every task, captures the run into a Sample, and scores it — one pipeline, no external glue.

bench := evalgo.Bench{
    Target:  myAgent,        // a Target: Run(ctx, Task) (RunOutput, error)
    Tasks:   tasks,          // each: Input + ExpectedTools + Rubric (+ Files seed)
    Metrics: metrics,        // evalgo.BuildMetrics(["tool_correctness","rubric",...], spec)
}
report, samples := bench.Run(ctx)

A Target can be in-process (wrap a Go agent) or ExecTarget, which drives an agent that runs as a subprocess in any language — the task is passed via EVAL_* env vars (EVAL_INPUT, EVAL_RUBRIC, EVAL_EXPECTED_TOOLS, EVAL_FILES, …) and the program prints a JSON RunOutput (or a bare Sample) to stdout. Runner (run only, no scoring) and a per-task Timeout are also available; a target error becomes a failed Sample rather than aborting the batch.

Compare several agents (evalgo bench)

Run the same tasks through several agents and get a task × agent PASS/FAIL grid — a cross-framework or cross-config benchmark, all from two config files:

evalgo bench --tasks tasks.json --targets targets.json --judge env \
  -m tool_correctness,task_completion,step_efficiency,rubric \
  --gate task_completion,rubric -o grid.json

tasks.json is an array of Task (name, input, expected_tools, rubric, and optional files seed fixtures). targets.json declares the agents:

[
  {"name": "agent-go",   "command": ["go","run","./examples/eval-bench"], "dir": "../agent-go", "env": {"GOWORK": "off"}},
  {"name": "miniagent",  "command": ["python","run.py"], "dir": "../mini"},
  {"name": "harness-rs", "command": ["./target/debug/eval-bench"]}
]

Each agent runs as a subprocess (the EVAL_* contract above), so Go, Python and Rust agents compete on identical tasks. --gate picks which metrics decide PASS/FAIL per cell (default task_completion,rubric, so a correct-but-inefficient run still passes). The library form is evalgo.Comparison{Targets, Tasks, Metrics, Gate}.

Synthesize a dataset

Generate goldens from your sources with an LLM instead of hand-writing them; the output is a normal samples array:

evalgo gen --docs handbook.txt -n 3 -o golden.json              # chunk a doc, 3 goldens/chunk
evalgo gen --contexts groups.json --evolutions 1 -o hard.json   # from context groups, + complexity evolution

Library: Synthesizer{Judge: ...}.FromDocuments(ctx, docs, chunkSize) (or FromContexts) returns []Sample with Input / Expected / Context set.

Red-teaming (safety probes)

Generate an adversarial dataset to check your own system upholds its safety boundaries — authorized defensive testing. Generation is offline; the probes carry no operational detail (the danger would live in the response, which a safe system must refuse).

evalgo redteam -o attacks.json                      # offline: injection / jailbreak / PII / harmful
evalgo redteam --kinds prompt_injection,jailbreak --enhance 1 --judge env  # LLM-obfuscated probes
# run attacks.json through your system, fill each "output", then:
evalgo -d attacks.json --judge env -m attack_resistance,pii_leakage,toxicity,refusal

refusal is a free deterministic gate (does the output decline?); attack_resistance is the judge call that decides whether the attack actually succeeded.

Datasets & regression gating

Load goldens from JSON, JSONL, or CSV (the CLI picks by extension), filter by metadata, and diff two runs to catch regressions between versions:

evalgo -d goldens.csv -m faithfulness --judge env             # CSV in, columns → Sample fields
evalgo -d attacks.json -m refusal --where attack=jailbreak    # only matching samples

evalgo -d goldens.json --judge env -f json -o new.json        # save a report
evalgo diff old.json new.json                                 # exit 1 if any metric regressed

diff pairs results by sample+metric and flags each as regressed / fixed / improved / added / removed (regressions first), so a PR that quietly makes an eval worse fails CI. Library: LoadJSON/LoadJSONL/LoadCSV, FilterMeta, DiffReports.

Cost & token tracking

With --judge env, every run ends with a judge usage block — calls, estimated tokens, cumulative judge time, and (with --cost-*) an estimated cost. The meter sits inside the cache, so cache hits aren't counted: the number reflects only real provider calls.

evalgo -d goldens.json --judge env -m faithfulness,rubric \
  --cache .eval-cache --cost-in 0.4 --cost-out 1.2

--- judge usage ---
calls         : 20
est. tokens   : 2089 prompt + 725 completion = 2814
judge time    : 377.4s (cumulative)
est. cost     : $0.0017

Library: evalgo.NewMeter(costInPerM, costOutPerM) then judge = meter.Wrap(judge); read meter.Usage(). Token counts are heuristic (EstimateTokens, ~4 chars/token), not a provider tokenizer.

Custom metrics (DAG)

RubricJudge is one judge call. DAG builds a metric from a decision tree, so the judge makes small, local decisions and pass/fail logic stays deterministic and explainable:

root := evalgo.YesNo("Is the OUTPUT valid JSON?",
    evalgo.YesNo("Does it contain a 'name' field?",
        evalgo.Leaf(1, "ok"),
        evalgo.Leaf(0.5, "missing name")),
    evalgo.Leaf(0, "not JSON"))
m := evalgo.DAG(judge, "json_shape", 0.99, root)

CLI reference

evalgo -d <dataset> [flags]     evaluate a dataset (root command)
evalgo bench ...                run tasks through several agents, print a PASS/FAIL grid
evalgo gen ...                  synthesize a golden dataset from docs/contexts
evalgo redteam ...              generate an adversarial dataset (offline)
evalgo diff <old> <new>         compare two reports, gate on regressions
evalgo metrics                  list registered metric names

evalgo bench --tasks tasks.json --targets targets.json runs every task through every agent (each an EVAL_*-driven subprocess) and prints a task × agent grid. Key flags: --gate (metrics that decide PASS/FAIL, default task_completion,rubric), --timeout (per task), -m, --judge, --concurrency, -o.

Flag (eval)	Purpose
-d, --dataset	dataset JSON/JSONL/CSV (- for stdin)
-m, --metrics	comma list of metric names
--judge	env (LLM via LLM_*) or none
-f, --format	console or json
-o, --out	also write a JSON report file
--concurrency / --rps	parallel samples / judge rate limit
--threshold / --fail-under	judge pass threshold / suite CI gate
--cache	dir to cache judge responses (re-runs spend no tokens)
--cost-in / --cost-out	USD per 1M prompt / completion tokens
--where	only evaluate samples whose meta matches key=value

License

MIT

Documentation

Overview

Package evalgo is a small, native-Go LLM evaluation framework.

Philosophy (vs. Python's DeepEval/RAGAS): extend Go's strengths — concurrency, determinism, zero-cost heuristics, and `go test` — instead of importing a heavy framework. Metrics split into two families:

• Deterministic (code-based): JSON/regex/contains/exact — fast, free, no LLM.
• Semantic (LLM-as-a-judge): rubric, faithfulness, answer relevancy, context precision — capture intent via a Judge.

The core package has zero third-party dependencies (stdlib only). The agent-go judge adapter lives in the sibling package ./llmjudge so library users who only want deterministic metrics pay no dependency cost.

Example

Example shows Eval-Go used as a library: build a Suite of deterministic metrics over an in-memory golden set and inspect the aggregated report. (Semantic metrics additionally take a judge — see package llmjudge.)

package main

import (
	"context"
	"fmt"

	evalgo "github.com/liliang-cn/eval-go"
)

func main() {
	suite := evalgo.Suite{
		Metrics: []evalgo.Metric{
			evalgo.ValidJSON(),
			evalgo.JSONHasFields("name", "age"),
		},
		Samples: []evalgo.Sample{
			{Name: "ok", Output: `{"name":"Jane","age":30}`},
			{Name: "missing-age", Output: `{"name":"Jane"}`},
		},
	}

	report := suite.Run(context.Background())
	for _, ms := range report.Summary() {
		fmt.Printf("%s: %d/%d passed\n", ms.Metric, ms.Passed, ms.Total)
	}
	fmt.Println("failed:", report.Failed())
}

Output:
json_has_fields: 1/2 passed
valid_json: 2/2 passed
failed: true

Index

• Variables
• func AttackKinds() []string
• func EstimateTokens(s string) int
• func RegisteredMetrics() []string
• func RunT(t *testing.T, s Suite)
• type AlignmentReport
  • func (a AlignmentReport) Failing(minKappa, minF1 float64) []string
  • func (a AlignmentReport) WriteConsole(w io.Writer, maxDisagreements int)
  • func (a AlignmentReport) WriteJSON(w io.Writer) error
• type AlignmentResult
  • func AlignMetric(ctx context.Context, samples []Sample, metric Metric, labelKey string) (AlignmentResult, error)
• type AttackKind
• type Bench
  • func (b Bench) Run(ctx context.Context) (Report, []Sample)
• type ChangeStatus
• type Comparison
  • func (c Comparison) Run(ctx context.Context) ComparisonReport
• type ComparisonReport
  • func (cr ComparisonReport) Grid() map[string]map[string]bool
  • func (cr ComparisonReport) RenderGrid(w io.Writer)
• type DAGNode
  • func Branch(question string, choices map[string]DAGNode, fallback DAGNode) DAGNode
  • func Leaf(score float64, reason string) DAGNode
  • func YesNo(question string, yes, no DAGNode) DAGNode
• type Diff
  • func DiffReports(old, new Report) Diff
  • func (d Diff) Regressions() int
  • func (d Diff) WriteConsole(w io.Writer)
• type Disagreement
• type ExecTarget
  • func (e ExecTarget) Run(ctx context.Context, task Task) (RunOutput, error)
• type ExecTargetSpec
• type JudgeFunc
  • func Cache(j JudgeFunc, dir string) JudgeFunc
  • func RateLimit(j JudgeFunc, rps float64, burst int) JudgeFunc
• type Meter
  • func NewMeter(costInPerM, costOutPerM float64) *Meter
  • func (m *Meter) Usage() Usage
  • func (m *Meter) Wrap(j JudgeFunc) JudgeFunc
• type Metric
  • func AnswerRelevancy(judge JudgeFunc, passThreshold float64) Metric
  • func ArgumentCorrectness(judge JudgeFunc, passThreshold float64) Metric
  • func AttackResistance(judge JudgeFunc, passThreshold float64) Metric
  • func Bias(judge JudgeFunc, passThreshold float64) Metric
  • func BuildMetrics(names []string, spec MetricSpec) ([]Metric, error)
  • func CitationPresent() Metric
  • func Contains(substr string) Metric
  • func ContextualPrecision(judge JudgeFunc, passThreshold float64) Metric
  • func ContextualRecall(judge JudgeFunc, passThreshold float64) Metric
  • func ContextualRelevancy(judge JudgeFunc, passThreshold float64) Metric
  • func ConversationCompleteness(judge JudgeFunc, passThreshold float64) Metric
  • func ConversationRelevancy(judge JudgeFunc, passThreshold float64) Metric
  • func DAG(judge JudgeFunc, name string, passThreshold float64, root DAGNode) Metric
  • func ExactMatch() Metric
  • func Faithfulness(judge JudgeFunc) Metric
  • func ForbidsRegex(pattern string) Metric
  • func Hallucination(judge JudgeFunc, passThreshold float64) Metric
  • func JSONHasFields(fields ...string) Metric
  • func KnowledgeRetention(judge JudgeFunc, passThreshold float64) Metric
  • func MatchesRegex(pattern string) Metric
  • func NonEmpty() Metric
  • func PIILeakage(judge JudgeFunc) Metric
  • func PlanAdherence(judge JudgeFunc, passThreshold float64) Metric
  • func PlanQuality(judge JudgeFunc, passThreshold float64) Metric
  • func RefusalPresent() Metric
  • func RoleAdherence(judge JudgeFunc, passThreshold float64) Metric
  • func RubricJudge(judge JudgeFunc, passThreshold float64) Metric
  • func StepEfficiency(judge JudgeFunc, passThreshold float64) Metric
  • func Summarization(judge JudgeFunc, passThreshold float64) Metric
  • func TaskCompletion(judge JudgeFunc, passThreshold float64) Metric
  • func ToolCorrectness() Metric
  • func Toxicity(judge JudgeFunc, passThreshold float64) Metric
  • func ValidJSON() Metric
• type MetricFunc
  • func (m MetricFunc) Name() string
  • func (m MetricFunc) Score(ctx context.Context, s Sample) (Result, error)
• type MetricSpec
• type MetricSummary
• type NamedTarget
  • func LoadTargets(r io.Reader) ([]NamedTarget, error)
• type RedTeam
  • func (rt RedTeam) Generate(ctx context.Context) ([]Sample, error)
• type Report
  • func LoadReport(r io.Reader) (Report, error)
  • func (r Report) Failed() bool
  • func (r Report) Summary() []MetricSummary
  • func (r Report) WriteConsole(w io.Writer)
  • func (r Report) WriteJSON(w io.Writer) error
• type Result
• type RunOutput
• type Runner
  • func (r Runner) Run(ctx context.Context, tasks []Task) []Sample
• type Sample
  • func Filter(samples []Sample, keep func(Sample) bool) []Sample
  • func FilterMeta(samples []Sample, key, value string) []Sample
  • func LoadCSV(r io.Reader) ([]Sample, error)
  • func LoadJSON(r io.Reader) ([]Sample, error)
  • func LoadJSONL(r io.Reader) ([]Sample, error)
• type SampleReport
• type ScoreDelta
• type Suite
  • func (s Suite) Run(ctx context.Context) Report
• type Synthesizer
  • func (sy Synthesizer) FromContexts(ctx context.Context, groups [][]string) ([]Sample, error)
  • func (sy Synthesizer) FromDocuments(ctx context.Context, docs []string, chunkSize int) ([]Sample, error)
• type Target
• type TargetFunc
  • func (f TargetFunc) Run(ctx context.Context, task Task) (RunOutput, error)
• type TargetReport
• type Task
  • func LoadTasks(r io.Reader) ([]Task, error)
  • func (t Task) Sample(out RunOutput) Sample
• type ToolCall
• type Turn
• type Usage

Examples

• Package
• Bench

Variables

var AllAttackKinds = []AttackKind{PromptInjection, Jailbreak, PIIExtraction, HarmfulRequest}

AllAttackKinds is the default set probed when none is specified.

var ErrNoLabels = errors.New("no samples carry a label for this metric")

ErrNoLabels is returned by AlignMetric when no sample in the set carries a human label for the metric — so the caller can skip it cleanly.

Functions

func AttackKinds

func AttackKinds() []string

AttackKinds returns the known attack-kind names, for CLI help and validation.

func EstimateTokens

func EstimateTokens(s string) int

EstimateTokens approximates BPE token count as ~4 characters per token — a rough, model-agnostic heuristic (no tokenizer dependency, keeps the core stdlib-only). Use it for relative comparison and ballpark cost, not billing.

func RegisteredMetrics

func RegisteredMetrics() []string

RegisteredMetrics returns the metric names known to BuildMetrics, sorted.

func RunT

func RunT(t *testing.T, s Suite)

RunT drives a Suite through Go's native test runner: one subtest per sample, executed with t.Parallel() so hundreds of rows evaluate concurrently. Failed metrics become t.Error, so `go test` exit codes and CI reporting work for free.

func TestRAG(t *testing.T) {
    if os.Getenv("RUN_EVALS") != "true" { t.Skip("set RUN_EVALS=true") }
    evalgo.RunT(t, suite)
}

Types

type AlignmentReport

type AlignmentReport []AlignmentResult

AlignmentReport is the alignment of several metrics, ready to write or gate.

func (AlignmentReport) Failing

func (a AlignmentReport) Failing(minKappa, minF1 float64) []string

Failing returns the metrics whose Kappa < minKappa or F1 < minF1. A floor of 0 disables that check. Use as the CI gate on judge quality.

func (AlignmentReport) WriteConsole

func (a AlignmentReport) WriteConsole(w io.Writer, maxDisagreements int)

WriteConsole emits a human-readable per-metric agreement summary, with up to maxDisagreements example mismatches per metric (0 = none).

func (AlignmentReport) WriteJSON

func (a AlignmentReport) WriteJSON(w io.Writer) error

WriteJSON emits the machine-readable alignment report.

type AlignmentResult

type AlignmentResult struct {
	Metric  string `json:"metric"`
	N       int    `json:"n"`       // labeled samples successfully scored
	Errored int    `json:"errored"` // labeled samples the judge errored on (excluded from N)

	// binary agreement — confusion matrix
	TP int `json:"tp"`
	FP int `json:"fp"`
	TN int `json:"tn"`
	FN int `json:"fn"`

	Accuracy  float64 `json:"accuracy"`
	Precision float64 `json:"precision"`
	Recall    float64 `json:"recall"`
	F1        float64 `json:"f1"`
	Kappa     float64 `json:"kappa"` // Cohen's kappa (chance-corrected agreement)

	// continuous agreement (judge.Score vs human score)
	Pearson  float64 `json:"pearson"`
	Spearman float64 `json:"spearman"`
	MAE      float64 `json:"mae"`
	RMSE     float64 `json:"rmse"`

	// BestThreshold is the pass-threshold on judge.Score that best matches the
	// human binary labels — use it to re-calibrate the metric's threshold.
	BestThreshold float64 `json:"best_threshold"`

	Disagreements []Disagreement `json:"disagreements,omitempty"`
}

AlignmentResult reports how well a metric's judge agrees with human labels on the labeled subset. Binary stats compare judge.Passed against (label >= 0.5); continuous stats compare judge.Score against the raw label.

func AlignMetric

func AlignMetric(ctx context.Context, samples []Sample, metric Metric, labelKey string) (AlignmentResult, error)

AlignMetric runs metric over the samples that carry a human label under labelKey (Sample.Labels[labelKey]) and measures judge-vs-human agreement. labelKey is the name the user labels by — typically the same name used to request the metric (the CLI `-m` name), which can differ from metric.Name(). It reuses Suite.Run, so the same bounded concurrency and rate-limited judge apply; it only post-processes the produced Results against the labels. Returns ErrNoLabels when the set has no labeled samples for labelKey.

type AttackKind

type AttackKind string

AttackKind is a category of adversarial probe.

const (
	PromptInjection AttackKind = "prompt_injection" // override the system's instructions
	Jailbreak       AttackKind = "jailbreak"        // roleplay / persona escapes
	PIIExtraction   AttackKind = "pii_extraction"   // coax out secrets or personal data
	HarmfulRequest  AttackKind = "harmful_request"  // requests a safe system must refuse
)

type Bench

type Bench struct {
	Target      Target
	Tasks       []Task
	Metrics     []Metric
	Concurrency int
	Timeout     time.Duration
	OnResult    func(task Task, s Sample, err error)
}

Bench is the end-to-end agent-eval entrypoint: run every Task through Target to produce Samples, then score them with Metrics. Returns the scored Report and the Samples (so you can persist or inspect the raw runs).

Example

ExampleBench shows the end-to-end agent-eval flow: define Tasks, plug an agent in as a Target, and let Bench run + score it. Here the Target is an in-process stub and the metrics are deterministic (no LLM judge) so the output is stable.

package main

import (
	"context"
	"fmt"

	evalgo "github.com/liliang-cn/eval-go"
)

func main() {
	// The system under test: any func that runs a Task and reports the run.
	agent := evalgo.TargetFunc(func(_ context.Context, t evalgo.Task) (evalgo.RunOutput, error) {
		return evalgo.RunOutput{
			Output:    "handled " + t.Name,
			ToolCalls: []evalgo.ToolCall{{Name: "write_file"}},
		}, nil
	})

	bench := evalgo.Bench{
		Target: agent,
		Tasks: []evalgo.Task{
			{Name: "task-a", Input: "do A", ExpectedTools: []string{"write_file"}},
		},
		Metrics: []evalgo.Metric{evalgo.NonEmpty(), evalgo.ToolCorrectness()},
	}

	report, samples := bench.Run(context.Background())
	fmt.Println("ran samples:", len(samples))
	for _, sr := range report.Samples {
		fmt.Printf("%s passed=%v\n", sr.Sample, sr.Passed)
	}
}

Output:
ran samples: 1
task-a passed=true

func (Bench) Run

func (b Bench) Run(ctx context.Context) (Report, []Sample)

Run produces Samples from Tasks via the Target, then scores them.

type ChangeStatus

type ChangeStatus string

ChangeStatus classifies one metric result's movement between two reports.

const (
	Regressed ChangeStatus = "regressed" // was passing, now failing
	Fixed     ChangeStatus = "fixed"     // was failing, now passing
	Declined  ChangeStatus = "declined"  // same pass state, lower score
	Improved  ChangeStatus = "improved"  // same pass state, higher score
	Unchanged ChangeStatus = "unchanged"
	Added     ChangeStatus = "added"   // present only in the new report
	Removed   ChangeStatus = "removed" // present only in the old report
)

type Comparison

type Comparison struct {
	Targets     []NamedTarget
	Tasks       []Task
	Metrics     []Metric
	Gate        []string // metric names that must pass for a cell to PASS; empty = all metrics
	Concurrency int
	Timeout     time.Duration
	OnResult    func(target string, task Task, s Sample, err error)
}

Comparison runs every Task through every Target and scores each run, so agents can be compared on identical tasks. Targets run sequentially (one agent at a time); a target's tasks run with Concurrency.

func (Comparison) Run

func (c Comparison) Run(ctx context.Context) ComparisonReport

Run executes the comparison and returns the full report.

type ComparisonReport

type ComparisonReport struct {
	Targets []TargetReport `json:"targets"`
	Gate    []string       `json:"gate,omitempty"`
}

ComparisonReport holds every agent's results plus the gate used to derive PASS/FAIL, and can render a task x agent grid.

func (ComparisonReport) Grid

func (cr ComparisonReport) Grid() map[string]map[string]bool

Grid returns passed[taskName][targetName] derived from the gate.

func (ComparisonReport) RenderGrid

func (cr ComparisonReport) RenderGrid(w io.Writer)

RenderGrid writes a task x agent PASS/FAIL table to w.

type DAGNode

type DAGNode interface {
	// contains filtered or unexported methods
}

DAGNode is one node in a DAG metric's decision tree.

func Branch

func Branch(question string, choices map[string]DAGNode, fallback DAGNode) DAGNode

Branch asks the judge to answer question with exactly one of the choice labels and routes to that child. If the answer matches no label, it routes to fallback; a nil fallback with no match scores 0.

func Leaf

func Leaf(score float64, reason string) DAGNode

Leaf is a terminal node that assigns a fixed score and reason.

func YesNo

func YesNo(question string, yes, no DAGNode) DAGNode

YesNo is the common binary Branch: answer "yes" → yes, "no" → no.

type Diff

type Diff []ScoreDelta

Diff is the full comparison of two reports.

func DiffReports

func DiffReports(old, new Report) Diff

DiffReports compares old vs new by (sample, metric).

func (Diff) Regressions

func (d Diff) Regressions() int

Regressions counts results that started passing and now fail.

func (Diff) WriteConsole

func (d Diff) WriteConsole(w io.Writer)

WriteConsole prints the diff, regressions first, ending with a verdict.

type Disagreement

type Disagreement struct {
	Sample      string  `json:"sample"`
	Human       float64 `json:"human"`        // human gold score 0..1
	JudgeScore  float64 `json:"judge_score"`  // the judge's raw score 0..1
	JudgePassed bool    `json:"judge_passed"` // the judge's pass verdict
	Reason      string  `json:"reason,omitempty"`
}

Disagreement is one labeled sample where the judge and the human disagreed (on the pass/fail boundary), surfaced for inspection.

type ExecTarget

type ExecTarget struct {
	Command []string          // argv, e.g. ["python", "runner.py"] or ["./agent"]
	Dir     string            // working directory (optional)
	Env     map[string]string // extra environment (e.g. API keys); merged over os.Environ
}

ExecTarget drives an agent that runs as a subprocess — so eval-go can evaluate an agent written in any language. The Task is passed both as JSON on stdin and as EVAL_* environment variables; the program prints a JSON RunOutput (or a bare Sample, or a one-element array of either) to stdout.

Environment passed to the program:

EVAL_NAME, EVAL_INPUT, EVAL_RUBRIC, EVAL_EXPECTED, EVAL_EXPECTED_TOOLS (CSV),
EVAL_CONTEXT (JSON array), EVAL_FILES (JSON object of seed fixtures)

func (ExecTarget) Run

func (e ExecTarget) Run(ctx context.Context, task Task) (RunOutput, error)

Run implements Target by executing the configured command.

type ExecTargetSpec

type ExecTargetSpec struct {
	Name    string            `json:"name"`
	Command []string          `json:"command"`
	Dir     string            `json:"dir,omitempty"`
	Env     map[string]string `json:"env,omitempty"`
}

ExecTargetSpec is the JSON-config form of an ExecTarget, so the agents under comparison can be declared in a file rather than in Go.

type JudgeFunc

type JudgeFunc func(ctx context.Context, prompt string) (string, error)

JudgeFunc executes one LLM-as-a-judge call: given a prompt, return raw text (expected to contain JSON). Wrap with RateLimit to pace provider QPS. Build one from agent-go via the sibling package ./llmjudge.

func Cache

func Cache(j JudgeFunc, dir string) JudgeFunc

Cache wraps a JudgeFunc so identical prompts are served from an on-disk cache instead of re-calling (and re-billing) the provider. Responses are keyed by a SHA-256 of the prompt and persisted as files under dir, so they survive across runs — a re-run of an unchanged dataset spends no tokens. Concurrency-safe.

Only successful responses are cached; errors always re-call. An empty dir disables caching (returns the judge unchanged), mirroring RateLimit.

Cache and RateLimit compose; wrap with Cache outermost so cache hits skip the rate limiter entirely: RateLimit(Cache(judge, dir), rps, burst) caches, then paces only the misses — or Cache(RateLimit(judge,...), dir) to pace nothing on a hit. The CLI uses the latter.

func RateLimit

func RateLimit(j JudgeFunc, rps float64, burst int) JudgeFunc

RateLimit wraps a JudgeFunc with a lazy token-bucket limiter so concurrent samples don't trigger 429 Too Many Requests from the judge provider.

rps is tokens refilled per second; burst is the bucket capacity. Production code may prefer golang.org/x/time/rate; this is an equivalent, dependency-free implementation kept in-tree to honor the framework's stdlib-only core.

type Meter

type Meter struct {
	// contains filtered or unexported fields
}

Meter wraps a JudgeFunc to record usage — call count, (estimated) tokens, cumulative judge time, and optional cost — so a CI run can report what an evaluation spent. Concurrency-safe.

Wrap the Meter INSIDE the cache so cache hits aren't billed:

judge = RateLimit(judge, rps, burst)
judge = meter.Wrap(judge)        // counts only real provider calls
judge = Cache(judge, dir)        // a hit returns before reaching the meter

Token counts are heuristic (see EstimateTokens), not a provider tokenizer — good for relative comparison and rough cost, not billing-exact.

func NewMeter

func NewMeter(costInPerM, costOutPerM float64) *Meter

NewMeter creates a Meter. costInPerM / costOutPerM are USD per 1,000,000 prompt / completion tokens; pass 0 to omit cost from the report.

func (*Meter) Usage

func (m *Meter) Usage() Usage

Usage snapshots the metered totals.

func (*Meter) Wrap

func (m *Meter) Wrap(j JudgeFunc) JudgeFunc

Wrap returns a JudgeFunc that records each call into the Meter.

type Metric

type Metric interface {
	Name() string
	Score(ctx context.Context, s Sample) (Result, error)
}

Metric scores a single sample. Deterministic metrics ignore ctx; semantic metrics use it for the judge call (cancellation / deadlines).

func AnswerRelevancy

func AnswerRelevancy(judge JudgeFunc, passThreshold float64) Metric

AnswerRelevancy measures how much of Output actually addresses Input: extract the answer's statements, classify each as relevant to the question. Score = relevant statements / total. Catches rambling / off-topic answers.

func ArgumentCorrectness

func ArgumentCorrectness(judge JudgeFunc, passThreshold float64) Metric

ArgumentCorrectness asks the judge, for EACH recorded tool call, whether the arguments are correct and sufficient to accomplish the INPUT task. Score = correct calls / total. Catches right-tool-wrong-arguments failures.

func AttackResistance

func AttackResistance(judge JudgeFunc, passThreshold float64) Metric

AttackResistance is the headline red-team metric: given the adversarial Input and the system's Output, judge whether the system RESISTED — refused, deflected, or otherwise stayed safe — rather than complying with the attack. Pass = resisted. An empty Output is treated as a (degenerate) refusal.

func Bias

func Bias(judge JudgeFunc, passThreshold float64) Metric

Bias decomposes Output into statements and flags any that carry bias (gender, political, racial, etc.). We score the UNBIASED fraction so higher is better: score = unbiased statements / total.

func BuildMetrics

func BuildMetrics(names []string, spec MetricSpec) ([]Metric, error)

BuildMetrics resolves metric names into Metrics. It errors on an unknown name, or on a semantic metric requested without a judge — so misconfiguration fails fast instead of silently skipping evaluation.

func CitationPresent

func CitationPresent() Metric

CitationPresent passes when Output contains at least one [SOURCE]-style citation — a deterministic proxy for grounded, attributable RAG answers.

func Contains

func Contains(substr string) Metric

Contains passes when Output contains substr (case-insensitive).

func ContextualPrecision

func ContextualPrecision(judge JudgeFunc, passThreshold float64) Metric

ContextualPrecision measures retrieval quality: of the Context chunks supplied, how many are actually relevant to the Input. Score = relevant chunks / total. Low precision means the retriever is surfacing noise.

func ContextualRecall

func ContextualRecall(judge JudgeFunc, passThreshold float64) Metric

ContextualRecall measures whether the retrieved Context contains everything the reference answer needs: decompose Expected into statements, then check each can be attributed to the retrieval context. Score = attributable statements / total. Low recall means the retriever missed information the answer depends on.

func ContextualRelevancy

func ContextualRelevancy(judge JudgeFunc, passThreshold float64) Metric

ContextualRelevancy measures retrieval noise: of the statements present in the retrieved Context, how many are relevant to the Input. Score = relevant statements / total. Low relevancy means the context is padded with off-topic material.

func ConversationCompleteness

func ConversationCompleteness(judge JudgeFunc, passThreshold float64) Metric

ConversationCompleteness judges whether the assistant fulfilled all of the user's intentions and requests across the whole conversation.

func ConversationRelevancy

func ConversationRelevancy(judge JudgeFunc, passThreshold float64) Metric

ConversationRelevancy asks the judge, for EACH assistant message, whether it is relevant given the conversation so far. Score = relevant / total.

func DAG

func DAG(judge JudgeFunc, name string, passThreshold float64, root DAGNode) Metric

DAG builds a metric from a decision tree of judge questions instead of one vague 0-1 score. Each interior node asks the judge a categorical question about the Sample and routes to a child by the answer; each leaf assigns a concrete score. This makes complex pass/fail logic deterministic and explainable — the judge only makes small, local decisions (the DeepEval "DAG metric" idea).

root := YesNo("Is the OUTPUT valid JSON?",
    YesNo("Does it contain a 'name' field?", Leaf(1, "ok"), Leaf(0.5, "missing name")),
    Leaf(0, "not JSON"))
m := DAG(judge, "json_shape", 0.99, root)

passThreshold is the minimum leaf score (0..1) required to pass.

func ExactMatch

func ExactMatch() Metric

ExactMatch passes when trimmed Output equals trimmed Expected.

func Faithfulness

func Faithfulness(judge JudgeFunc) Metric

Faithfulness measures how grounded Output is in Context: extract the claims the answer makes, then check each against the retrieval context. Score = non-contradicted claims / total claims. Catches hallucination.

func ForbidsRegex

func ForbidsRegex(pattern string) Metric

ForbidsRegex passes when Output does NOT match the pattern — a safety boundary (no secrets, no banned phrases, no leaked PII).

func Hallucination

func Hallucination(judge JudgeFunc, passThreshold float64) Metric

Hallucination treats Context as trusted source facts and checks that Output does not contradict them. We invert the usual sense so higher = MORE factually consistent, matching the rest of the framework (higher is always better).

func JSONHasFields

func JSONHasFields(fields ...string) Metric

JSONHasFields passes when Output is a JSON object containing all given keys.

func KnowledgeRetention

func KnowledgeRetention(judge JudgeFunc, passThreshold float64) Metric

KnowledgeRetention judges whether the assistant RETAINED information the user already provided earlier. A low score means it asked again for facts already given, or forgot or contradicted earlier context.

func MatchesRegex

func MatchesRegex(pattern string) Metric

MatchesRegex passes when Output matches the pattern (e.g. a required format).

func NonEmpty

func NonEmpty() Metric

NonEmpty passes when Output has non-whitespace content.

func PIILeakage

func PIILeakage(judge JudgeFunc) Metric

PIILeakage is a single-judgment check: does Output leak personally identifiable information (names tied to data, emails, phone numbers, SSNs, credit cards, addresses)? Passes when the text is clean.

func PlanAdherence

func PlanAdherence(judge JudgeFunc, passThreshold float64) Metric

PlanAdherence judges whether the agent's actual Trajectory followed its own Plan. Needs both a plan and a trajectory; skipped (pass) otherwise.

func PlanQuality

func PlanQuality(judge JudgeFunc, passThreshold float64) Metric

PlanQuality judges the agent's stated Plan for the INPUT: is it logical, complete, and efficient? Skipped (pass) when the sample carries no plan.

func RefusalPresent

func RefusalPresent() Metric

RefusalPresent is a deterministic proxy for attack resistance: the Output contains refusal/deflection language. Free and offline — a cheap first gate before spending a judge call on attack_resistance.

func RoleAdherence

func RoleAdherence(judge JudgeFunc, passThreshold float64) Metric

RoleAdherence judges whether the assistant consistently stayed in its assigned Persona throughout the conversation. Skipped (pass) when no persona is given.

func RubricJudge

func RubricJudge(judge JudgeFunc, passThreshold float64) Metric

RubricJudge is a G-Eval-style metric: the judge scores Output against Sample.Rubric with chain-of-thought, returning {passed, score, reason}. passThreshold is the minimum score (0..1) required to pass.

func StepEfficiency

func StepEfficiency(judge JudgeFunc, passThreshold float64) Metric

StepEfficiency judges whether the agent reached the goal without unnecessary, redundant, or repeated steps. A correct-but-wasteful run scores low.

func Summarization

func Summarization(judge JudgeFunc, passThreshold float64) Metric

Summarization judges whether Output is a faithful AND complete summary of the source text in Input: alignment (no contradictions/hallucinations vs source) and coverage (captures the key points). Input is the ORIGINAL TEXT, Output the SUMMARY.

func TaskCompletion

func TaskCompletion(judge JudgeFunc, passThreshold float64) Metric

TaskCompletion judges whether the agent actually accomplished the INPUT task, given everything it did (trajectory + tool calls) and its final OUTPUT.

func ToolCorrectness

func ToolCorrectness() Metric

ToolCorrectness checks the tools the agent actually invoked against the ground-truth ExpectedTools (order-independent set comparison). Score is the Jaccard overlap so missing AND extraneous tool calls both cost points; the metric passes only on an exact set match. Deterministic — no LLM.

func Toxicity

func Toxicity(judge JudgeFunc, passThreshold float64) Metric

Toxicity decomposes Output into statements and flags any that are toxic (insults, threats, hate, harassment). We score the NON-TOXIC fraction so higher is better: score = non-toxic statements / total.

func ValidJSON

func ValidJSON() Metric

ValidJSON passes when Output is structurally valid JSON.

type MetricFunc

type MetricFunc struct {
	MetricName string
	Fn         func(ctx context.Context, s Sample) (Result, error)
}

MetricFunc adapts a function into a Metric.

func (MetricFunc) Name

func (m MetricFunc) Name() string

func (MetricFunc) Score

func (m MetricFunc) Score(ctx context.Context, s Sample) (Result, error)

type MetricSpec

type MetricSpec struct {
	Judge     JudgeFunc // required for semantic metrics; may be nil for deterministic-only
	Threshold float64   // pass threshold for relevancy / context_precision / rubric (default 0.5)
}

MetricSpec configures the metrics built by BuildMetrics.

type MetricSummary

type MetricSummary struct {
	Metric    string  `json:"metric"`
	PassRate  float64 `json:"pass_rate"` // 0..1
	MeanScore float64 `json:"mean_score"`
	Passed    int     `json:"passed"`
	Total     int     `json:"total"`
}

MetricSummary aggregates one metric across all samples.

type NamedTarget

type NamedTarget struct {
	Name   string
	Target Target
}

NamedTarget pairs a label with a Target so a Comparison can attribute results.

func LoadTargets

func LoadTargets(r io.Reader) ([]NamedTarget, error)

LoadTargets reads a JSON array of ExecTargetSpec and builds NamedTargets, each backed by an ExecTarget.

type RedTeam

type RedTeam struct {
	Judge   JudgeFunc    // optional; required only when Enhance > 0
	Kinds   []AttackKind // categories to include (default: AllAttackKinds)
	Enhance int          // LLM obfuscation passes applied to each probe
}

RedTeam generates attack Samples. With a Judge set and Enhance > 0, each probe is additionally rewritten to evade naive keyword filters (stress-tests robustness); without a Judge, generation is fully offline.

func (RedTeam) Generate

func (rt RedTeam) Generate(ctx context.Context) ([]Sample, error)

Generate builds the adversarial dataset.

type Report

type Report struct {
	Samples []SampleReport `json:"samples"`
	Usage   *Usage         `json:"usage,omitempty"` // judge usage, when metered (set by the caller)
}

Report is the full outcome of a Suite run.

func LoadReport

func LoadReport(r io.Reader) (Report, error)

LoadReport parses a JSON report produced by Report.WriteJSON. Only the per-sample results are needed for diffing (summary/usage are ignored).

func (Report) Failed

func (r Report) Failed() bool

Failed reports whether any sample failed any metric — use as the CI exit gate.

func (Report) Summary

func (r Report) Summary() []MetricSummary

Summary computes per-metric aggregates, ordered by metric name.

func (Report) WriteConsole

func (r Report) WriteConsole(w io.Writer)

WriteConsole emits a human-readable report: per-sample metric grid + per-metric aggregates + overall verdict.

func (Report) WriteJSON

func (r Report) WriteJSON(w io.Writer) error

WriteJSON emits the machine-readable report (for CI artifacts / dashboards).

type Result

type Result struct {
	Metric string  `json:"metric"`
	Score  float64 `json:"score"`
	Passed bool    `json:"passed"`
	Reason string  `json:"reason,omitempty"`
	Err    string  `json:"error,omitempty"`
}

Result is the outcome of one metric on one sample. Score is normalized 0..1.

type RunOutput

type RunOutput struct {
	Output     string     `json:"output"`               // the final answer the agent produced
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"` // tools invoked, in order
	Trajectory []string   `json:"trajectory,omitempty"` // ordered action/reasoning steps
	Plan       string     `json:"plan,omitempty"`       // the agent's stated plan, if any
	Context    []string   `json:"context,omitempty"`    // evidence the judge should see (e.g. final files)
}

RunOutput is what a Target reports after executing one Task: everything the metrics need to grade the run.

type Runner

type Runner struct {
	Target      Target
	Concurrency int           // tasks run in parallel; default 4
	Timeout     time.Duration // per-task wall clock; 0 = no per-task timeout
	OnResult    func(task Task, s Sample, err error)
}

Runner drives a set of Tasks through a Target, concurrently, and returns one Sample per Task in input order. A Target error never aborts the batch: it becomes a Sample whose Output records the failure and whose Meta["run_error"] is set, so the metrics grade it as a failed run rather than silently dropping.

func (Runner) Run

func (r Runner) Run(ctx context.Context, tasks []Task) []Sample

Run executes every Task and returns the resulting Samples (input order).

type Sample

type Sample struct {
	Name     string            `json:"name"`               // unique-ish label for reports
	Input    string            `json:"input"`              // the question / prompt / task given to the system
	Output   string            `json:"output"`             // the actual answer produced by the system under test
	Expected string            `json:"expected,omitempty"` // optional reference answer (for ExactMatch etc.)
	Context  []string          `json:"context,omitempty"`  // retrieved evidence chunks (for RAG metrics)
	Rubric   string            `json:"rubric,omitempty"`   // pass/fail criterion for a rubric judge
	Meta     map[string]string `json:"meta,omitempty"`     // free-form labels carried into reports

	// human gold judgment per metric name, normalized 0..1 (binary: 1=pass, 0=fail).
	// Only used by AlignMetric to measure judge-vs-human agreement; ignored by scoring.
	Labels map[string]float64 `json:"labels,omitempty"`

	// --- agent execution (recorded from an agent run; for agentic metrics) ---
	Plan          string     `json:"plan,omitempty"`           // the agent's stated plan (for plan_quality / plan_adherence)
	Trajectory    []string   `json:"trajectory,omitempty"`     // ordered reasoning/action steps the agent took
	ToolCalls     []ToolCall `json:"tool_calls,omitempty"`     // tools the agent actually invoked, in order
	ExpectedTools []string   `json:"expected_tools,omitempty"` // ground-truth tool names (for tool_correctness)

	// --- multi-turn conversation (for conversational metrics) ---
	Turns   []Turn `json:"turns,omitempty"`   // the full chat history, in order
	Persona string `json:"persona,omitempty"` // the role the assistant should hold (for role_adherence)
}

Sample is one row of a golden dataset plus the system's actual output.

The first block of fields describes a single-turn output (RAG / generation). The agent block records what an agent actually did during a run — Eval-Go evaluates that recorded trajectory rather than instrumenting a live runtime, so any framework (or none) can emit these fields and be scored the same way.

func Filter

func Filter(samples []Sample, keep func(Sample) bool) []Sample

Filter returns the samples for which keep returns true.

func FilterMeta

func FilterMeta(samples []Sample, key, value string) []Sample

FilterMeta returns the samples whose Meta[key] equals value — e.g. select one red-team attack kind with FilterMeta(samples, "attack", "jailbreak").

func LoadCSV

func LoadCSV(r io.Reader) ([]Sample, error)

LoadCSV reads Samples from a CSV with a header row. Recognized columns (case-insensitive) map to Sample fields: name, input, output, expected, rubric, plan, persona. "context" splits on '|' into chunks; "expected_tools" splits on ','. Any other column becomes a Meta entry — so a "tag" or "category" column is filterable via FilterMeta.

func LoadJSON

func LoadJSON(r io.Reader) ([]Sample, error)

LoadJSON reads a JSON array of Samples.

func LoadJSONL

func LoadJSONL(r io.Reader) ([]Sample, error)

LoadJSONL reads one Sample per line (JSON Lines) — the convenient append-only format for large or streamed datasets. Blank lines are skipped.

type SampleReport

type SampleReport struct {
	Sample  string            `json:"sample"`
	Meta    map[string]string `json:"meta,omitempty"`
	Results []Result          `json:"results"`
	Passed  bool              `json:"passed"`
}

SampleReport holds every metric Result for one sample.

type ScoreDelta

type ScoreDelta struct {
	Sample    string       `json:"sample"`
	Metric    string       `json:"metric"`
	Old       float64      `json:"old"`
	New       float64      `json:"new"`
	OldPassed bool         `json:"old_passed"`
	NewPassed bool         `json:"new_passed"`
	Status    ChangeStatus `json:"status"`
}

ScoreDelta is one metric's movement between two reports.

type Suite

type Suite struct {
	Samples     []Sample
	Metrics     []Metric
	Concurrency int // samples evaluated in parallel; default 4
}

Suite is a dataset + the metrics to apply, run concurrently across samples.

func (Suite) Run

func (s Suite) Run(ctx context.Context) Report

Run evaluates every metric against every sample. Samples run concurrently (bounded by Concurrency); metrics within a sample run sequentially so a shared rate-limited judge paces cleanly. A metric error becomes a failed Result (Err set) rather than aborting the whole run.

type Synthesizer

type Synthesizer struct {
	Judge      JudgeFunc // the generating LLM (same shape as a judge)
	PerContext int       // goldens to generate per context group (default 2)
	Evolutions int       // complexity-evolution passes applied to each question (default 0)
	Rubric     string    // rubric stamped on every Sample (default: grounded-answer criterion)
}

Synthesizer generates golden Samples with an LLM, so you can build an evaluation dataset from source documents instead of hand-writing JSON. The generated Samples carry Input / Expected / Context (and a default Rubric), ready to feed straight into a Suite — including the RAG and faithfulness metrics that score against Context.

It uses only a JudgeFunc, so the stdlib-only core stays dependency-free.

func (Synthesizer) FromContexts

func (sy Synthesizer) FromContexts(ctx context.Context, groups [][]string) ([]Sample, error)

FromContexts generates Samples for each group of retrieval-context chunks: the LLM writes realistic questions answerable from that context plus grounded reference answers. Each group becomes that many Samples' Context.

func (Synthesizer) FromDocuments

func (sy Synthesizer) FromDocuments(ctx context.Context, docs []string, chunkSize int) ([]Sample, error)

FromDocuments splits each document into ~chunkSize-rune chunks (on word boundaries), treats each chunk as one context group, and generates from them.

type Target

type Target interface {
	Run(ctx context.Context, task Task) (RunOutput, error)
}

Target is the system under test. It runs one Task and reports what happened. Implement it in-process for a Go agent, or use ExecTarget to drive any agent that runs as a subprocess (Python, Rust, a shell script, ...).

type TargetFunc

type TargetFunc func(ctx context.Context, task Task) (RunOutput, error)

TargetFunc adapts a plain function into a Target.

func (TargetFunc) Run

func (f TargetFunc) Run(ctx context.Context, task Task) (RunOutput, error)

Run implements Target.

type TargetReport

type TargetReport struct {
	Name    string   `json:"name"`
	Report  Report   `json:"report"`
	Samples []Sample `json:"samples"`
}

TargetReport is one agent's scored results across all tasks.

type Task

type Task struct {
	Name          string            `json:"name"`
	Input         string            `json:"input"`                    // the prompt / goal handed to the agent
	ExpectedTools []string          `json:"expected_tools,omitempty"` // ground-truth tools (for tool_correctness)
	Rubric        string            `json:"rubric,omitempty"`         // pass/fail criterion (for the rubric judge)
	Expected      string            `json:"expected,omitempty"`       // optional reference answer
	Context       []string          `json:"context,omitempty"`        // evidence carried into the Sample if the run adds none
	Files         map[string]string `json:"files,omitempty"`          // optional seed fixtures (path -> content) for the target
	Meta          map[string]string `json:"meta,omitempty"`           // labels carried into the report
}

Task is one unit of agentic work plus the ground truth needed to grade the resulting run. It is the input side of an agent eval; the Target turns it into a RunOutput, and Task.Sample merges the two into a gradeable Sample.

func LoadTasks

func LoadTasks(r io.Reader) ([]Task, error)

LoadTasks reads a JSON array of Task (the agent-benchmark input set).

func (Task) Sample

func (t Task) Sample(out RunOutput) Sample

Sample merges a Task with the Target's RunOutput into a gradeable Sample.

type ToolCall

type ToolCall struct {
	Name   string         `json:"name"`             // tool / function name invoked
	Args   map[string]any `json:"args,omitempty"`   // arguments the agent passed
	Output string         `json:"output,omitempty"` // tool result, if captured (helps the judge)
}

ToolCall is one tool invocation recorded from an agent run.

type Turn

type Turn struct {
	Role    string `json:"role"`    // "user" or "assistant"
	Content string `json:"content"` // the message text
}

Turn is one message in a multi-turn conversation.

type Usage

type Usage struct {
	Calls            int     `json:"calls"`
	PromptTokens     int     `json:"prompt_tokens"`
	CompletionTokens int     `json:"completion_tokens"`
	TotalTokens      int     `json:"total_tokens"`
	JudgeSeconds     float64 `json:"judge_seconds"`
	Cost             float64 `json:"cost,omitempty"`
}

Usage is a snapshot of judge usage, suitable for reports.