evalgo

README

Eval-Go

A small, native-Go framework for evaluating LLM and RAG outputs.

While Python dominates LLM evaluation with heavy frameworks (DeepEval, RAGAS), Eval-Go takes the opposite approach: lean on Go's own strengths — concurrency, determinism, zero-cost heuristics, and go test — instead of importing a large framework. Metrics split into two families:

• Deterministic (code-based): JSON validity, regex, forbidden-pattern, exact match, citation presence. Fast, free, no LLM. Run them first as cheap gates.
• Semantic (LLM-as-a-judge): rubric/G-Eval, faithfulness, answer relevancy, contextual precision — capture intent via a pluggable judge.

The core package has zero third-party dependencies (stdlib only). The judge adapter for agent-go lives in a separate package (./llmjudge) so deterministic-only users pay no dependency cost.

Eval-Go is both a library and a CLI: import the evalgo package to embed evaluation in your own Go code and tests, or run the evalgo binary against a JSON golden dataset in CI.

Install

# as a library
go get github.com/liliang-cn/eval-go

# as a CLI
go install github.com/liliang-cn/eval-go/cmd/evalgo@latest

CLI

# deterministic metrics — free, offline
evalgo -dataset examples/golden.json -metrics nonempty,citation,valid_json

# add LLM-as-a-judge (any OpenAI-compatible endpoint)
export LLM_BASE_URL=... LLM_API_KEY=... LLM_MODEL=...
evalgo -dataset examples/golden.json -judge env \
  -metrics faithfulness,answer_relevancy,context_precision,rubric \
  -format json -out report.json -fail-under 0.8

The dataset is a JSON array of samples (name, input, output, context, expected, rubric). Exit code is 1 when the CI gate fails (-fail-under, or any failure by default) and 2 on a config error — drop it straight into a pipeline.

Flag	Purpose
-dataset	golden dataset JSON (- for stdin)
-metrics	comma list (see table below)
-judge	env (LLM via LLM_*) or none
-format	console or json
-out	also write a JSON report file
-concurrency / -rps	parallel samples / judge rate limit
-threshold / -fail-under	metric pass threshold / suite CI gate

Library quick start

import (
    evalgo "github.com/liliang-cn/eval-go"
    "github.com/liliang-cn/eval-go/llmjudge"
)

judge, _ := llmjudge.FromEnv()          // LLM_BASE_URL / LLM_API_KEY / LLM_MODEL
judge = evalgo.RateLimit(judge, 4, 2)   // token-bucket; avoid 429s

suite := evalgo.Suite{
    Concurrency: 4,
    Metrics: []evalgo.Metric{
        evalgo.CitationPresent(),               // deterministic
        evalgo.Faithfulness(judge),             // semantic (RAG)
        evalgo.AnswerRelevancy(judge, 0.5),
        evalgo.ContextualPrecision(judge, 0.5),
        evalgo.RubricJudge(judge, 0.7),
    },
    Samples: []evalgo.Sample{{
        Name:    "grounded-answer",
        Input:   "What is the savings account interest rate?",
        Context: []string{"The savings account pays 0.30% below 50000 and 0.55% above."},
        Output:  "The rate is tiered: 0.30% below 50000 and 0.55% above [KB-001].",
    }},
}

report := suite.Run(context.Background())
report.WriteConsole(os.Stdout)
if report.Failed() { os.Exit(1) }       // CI gate

Native go test integration

RunT turns a suite into table-driven, parallel subtests so go test exit codes and CI reporting work for free. Gate semantic evals behind an env flag so plain go test spends no tokens:

func TestRAG(t *testing.T) {
    if os.Getenv("RUN_EVALS") != "true" { t.Skip("set RUN_EVALS=true") }
    judge, _ := llmjudge.FromEnv()
    evalgo.RunT(t, evalgo.Suite{Judge stuff..., Metrics: ..., Samples: ...})
}

go test ./...                                   # deterministic only, free & offline
RUN_EVALS=true LLM_BASE_URL=... LLM_API_KEY=... LLM_MODEL=... go test ./... -v

Metrics

Metric	Type	What it checks
ValidJSON, JSONHasFields	deterministic	output is JSON / has required keys
MatchesRegex, ForbidsRegex	deterministic	required format / safety boundary (no leaked secrets)
Contains, ExactMatch, NonEmpty	deterministic	substring / reference / non-empty
CitationPresent	deterministic	output carries [SOURCE] attribution
RubricJudge	semantic	G-Eval-style pass/score against a rubric
Faithfulness	semantic (RAG)	claims extracted from the answer are grounded in context (catches hallucination)
AnswerRelevancy	semantic (RAG)	answer statements actually address the question
ContextualPrecision	semantic (RAG)	retrieved context chunks are relevant to the question

RAG metrics use the DeepEval/RAGAS-style decomposition (extract atomic units → verify each) rather than one vague 0–1 score — more reliable and explainable.

examples/golden.json deliberately includes a hallucinated answer; faithfulness scores it 0.00 even though it carries a citation that fools the deterministic checks.

License

MIT

Documentation

Overview

Package evalgo is a small, native-Go LLM evaluation framework.

Philosophy (vs. Python's DeepEval/RAGAS): extend Go's strengths — concurrency, determinism, zero-cost heuristics, and `go test` — instead of importing a heavy framework. Metrics split into two families:

• Deterministic (code-based): JSON/regex/contains/exact — fast, free, no LLM.
• Semantic (LLM-as-a-judge): rubric, faithfulness, answer relevancy, context precision — capture intent via a Judge.

The core package has zero third-party dependencies (stdlib only). The agent-go judge adapter lives in the sibling package ./llmjudge so library users who only want deterministic metrics pay no dependency cost.

Example

Example shows Eval-Go used as a library: build a Suite of deterministic metrics over an in-memory golden set and inspect the aggregated report. (Semantic metrics additionally take a judge — see package llmjudge.)

package main

import (
	"context"
	"fmt"

	evalgo "github.com/liliang-cn/eval-go"
)

func main() {
	suite := evalgo.Suite{
		Metrics: []evalgo.Metric{
			evalgo.ValidJSON(),
			evalgo.JSONHasFields("name", "age"),
		},
		Samples: []evalgo.Sample{
			{Name: "ok", Output: `{"name":"Jane","age":30}`},
			{Name: "missing-age", Output: `{"name":"Jane"}`},
		},
	}

	report := suite.Run(context.Background())
	for _, ms := range report.Summary() {
		fmt.Printf("%s: %d/%d passed\n", ms.Metric, ms.Passed, ms.Total)
	}
	fmt.Println("failed:", report.Failed())
}

Output:
json_has_fields: 1/2 passed
valid_json: 2/2 passed
failed: true

Index

• func RegisteredMetrics() []string
• func RunT(t *testing.T, s Suite)
• type JudgeFunc
  • func RateLimit(j JudgeFunc, rps float64, burst int) JudgeFunc
• type Metric
  • func AnswerRelevancy(judge JudgeFunc, passThreshold float64) Metric
  • func BuildMetrics(names []string, spec MetricSpec) ([]Metric, error)
  • func CitationPresent() Metric
  • func Contains(substr string) Metric
  • func ContextualPrecision(judge JudgeFunc, passThreshold float64) Metric
  • func ExactMatch() Metric
  • func Faithfulness(judge JudgeFunc) Metric
  • func ForbidsRegex(pattern string) Metric
  • func JSONHasFields(fields ...string) Metric
  • func MatchesRegex(pattern string) Metric
  • func NonEmpty() Metric
  • func RubricJudge(judge JudgeFunc, passThreshold float64) Metric
  • func ValidJSON() Metric
• type MetricFunc
  • func (m MetricFunc) Name() string
  • func (m MetricFunc) Score(ctx context.Context, s Sample) (Result, error)
• type MetricSpec
• type MetricSummary
• type Report
  • func (r Report) Failed() bool
  • func (r Report) Summary() []MetricSummary
  • func (r Report) WriteConsole(w io.Writer)
  • func (r Report) WriteJSON(w io.Writer) error
• type Result
• type Sample
• type SampleReport
• type Suite
  • func (s Suite) Run(ctx context.Context) Report

Examples

• Package

Functions

func RegisteredMetrics

func RegisteredMetrics() []string

RegisteredMetrics returns the metric names known to BuildMetrics, sorted.

func RunT

func RunT(t *testing.T, s Suite)

RunT drives a Suite through Go's native test runner: one subtest per sample, executed with t.Parallel() so hundreds of rows evaluate concurrently. Failed metrics become t.Error, so `go test` exit codes and CI reporting work for free.

func TestRAG(t *testing.T) {
    if os.Getenv("RUN_EVALS") != "true" { t.Skip("set RUN_EVALS=true") }
    evalgo.RunT(t, suite)
}

Types

type JudgeFunc

type JudgeFunc func(ctx context.Context, prompt string) (string, error)

JudgeFunc executes one LLM-as-a-judge call: given a prompt, return raw text (expected to contain JSON). Wrap with RateLimit to pace provider QPS. Build one from agent-go via the sibling package ./llmjudge.

func RateLimit

func RateLimit(j JudgeFunc, rps float64, burst int) JudgeFunc

RateLimit wraps a JudgeFunc with a lazy token-bucket limiter so concurrent samples don't trigger 429 Too Many Requests from the judge provider.

rps is tokens refilled per second; burst is the bucket capacity. Production code may prefer golang.org/x/time/rate; this is an equivalent, dependency-free implementation kept in-tree to honor the framework's stdlib-only core.

type Metric

type Metric interface {
	Name() string
	Score(ctx context.Context, s Sample) (Result, error)
}

Metric scores a single sample. Deterministic metrics ignore ctx; semantic metrics use it for the judge call (cancellation / deadlines).

func AnswerRelevancy

func AnswerRelevancy(judge JudgeFunc, passThreshold float64) Metric

AnswerRelevancy measures how much of Output actually addresses Input: extract the answer's statements, classify each as relevant to the question. Score = relevant statements / total. Catches rambling / off-topic answers.

func BuildMetrics

func BuildMetrics(names []string, spec MetricSpec) ([]Metric, error)

BuildMetrics resolves metric names into Metrics. It errors on an unknown name, or on a semantic metric requested without a judge — so misconfiguration fails fast instead of silently skipping evaluation.

func CitationPresent

func CitationPresent() Metric

CitationPresent passes when Output contains at least one [SOURCE]-style citation — a deterministic proxy for grounded, attributable RAG answers.

func Contains

func Contains(substr string) Metric

Contains passes when Output contains substr (case-insensitive).

func ContextualPrecision

func ContextualPrecision(judge JudgeFunc, passThreshold float64) Metric

ContextualPrecision measures retrieval quality: of the Context chunks supplied, how many are actually relevant to the Input. Score = relevant chunks / total. Low precision means the retriever is surfacing noise.

func ExactMatch

func ExactMatch() Metric

ExactMatch passes when trimmed Output equals trimmed Expected.

func Faithfulness

func Faithfulness(judge JudgeFunc) Metric

Faithfulness measures how grounded Output is in Context: extract the claims the answer makes, then check each against the retrieval context. Score = non-contradicted claims / total claims. Catches hallucination.

func ForbidsRegex

func ForbidsRegex(pattern string) Metric

ForbidsRegex passes when Output does NOT match the pattern — a safety boundary (no secrets, no banned phrases, no leaked PII).

func JSONHasFields

func JSONHasFields(fields ...string) Metric

JSONHasFields passes when Output is a JSON object containing all given keys.

func MatchesRegex

func MatchesRegex(pattern string) Metric

MatchesRegex passes when Output matches the pattern (e.g. a required format).

func NonEmpty

func NonEmpty() Metric

NonEmpty passes when Output has non-whitespace content.

func RubricJudge

func RubricJudge(judge JudgeFunc, passThreshold float64) Metric

RubricJudge is a G-Eval-style metric: the judge scores Output against Sample.Rubric with chain-of-thought, returning {passed, score, reason}. passThreshold is the minimum score (0..1) required to pass.

func ValidJSON

func ValidJSON() Metric

ValidJSON passes when Output is structurally valid JSON.

type MetricFunc

type MetricFunc struct {
	MetricName string
	Fn         func(ctx context.Context, s Sample) (Result, error)
}

MetricFunc adapts a function into a Metric.

func (MetricFunc) Name

func (m MetricFunc) Name() string

func (MetricFunc) Score

func (m MetricFunc) Score(ctx context.Context, s Sample) (Result, error)

type MetricSpec

type MetricSpec struct {
	Judge     JudgeFunc // required for semantic metrics; may be nil for deterministic-only
	Threshold float64   // pass threshold for relevancy / context_precision / rubric (default 0.5)
}

MetricSpec configures the metrics built by BuildMetrics.

type MetricSummary

type MetricSummary struct {
	Metric    string  `json:"metric"`
	PassRate  float64 `json:"pass_rate"` // 0..1
	MeanScore float64 `json:"mean_score"`
	Passed    int     `json:"passed"`
	Total     int     `json:"total"`
}

MetricSummary aggregates one metric across all samples.

type Report

type Report struct {
	Samples []SampleReport `json:"samples"`
}

Report is the full outcome of a Suite run.

func (Report) Failed

func (r Report) Failed() bool

Failed reports whether any sample failed any metric — use as the CI exit gate.

func (Report) Summary

func (r Report) Summary() []MetricSummary

Summary computes per-metric aggregates, ordered by metric name.

func (Report) WriteConsole

func (r Report) WriteConsole(w io.Writer)

WriteConsole emits a human-readable report: per-sample metric grid + per-metric aggregates + overall verdict.

func (Report) WriteJSON

func (r Report) WriteJSON(w io.Writer) error

WriteJSON emits the machine-readable report (for CI artifacts / dashboards).

type Result

type Result struct {
	Metric string  `json:"metric"`
	Score  float64 `json:"score"`
	Passed bool    `json:"passed"`
	Reason string  `json:"reason,omitempty"`
	Err    string  `json:"error,omitempty"`
}

Result is the outcome of one metric on one sample. Score is normalized 0..1.

type Sample

type Sample struct {
	Name     string            `json:"name"`               // unique-ish label for reports
	Input    string            `json:"input"`              // the question / prompt given to the system
	Output   string            `json:"output"`             // the actual answer produced by the system under test
	Expected string            `json:"expected,omitempty"` // optional reference answer (for ExactMatch etc.)
	Context  []string          `json:"context,omitempty"`  // retrieved evidence chunks (for RAG metrics)
	Rubric   string            `json:"rubric,omitempty"`   // pass/fail criterion for a rubric judge
	Meta     map[string]string `json:"meta,omitempty"`     // free-form labels carried into reports
}

Sample is one row of a golden dataset plus the system's actual output.

type SampleReport

type SampleReport struct {
	Sample  string            `json:"sample"`
	Meta    map[string]string `json:"meta,omitempty"`
	Results []Result          `json:"results"`
	Passed  bool              `json:"passed"`
}

SampleReport holds every metric Result for one sample.

type Suite

type Suite struct {
	Samples     []Sample
	Metrics     []Metric
	Concurrency int // samples evaluated in parallel; default 4
}

Suite is a dataset + the metrics to apply, run concurrently across samples.

func (Suite) Run

func (s Suite) Run(ctx context.Context) Report

Run evaluates every metric against every sample. Samples run concurrently (bounded by Concurrency); metrics within a sample run sequentially so a shared rate-limited judge paces cleanly. A metric error becomes a failed Result (Err set) rather than aborting the whole run.