eval

package module
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package eval defines a subject-agnostic quality-evaluation kernel. Evaluator is generic over the subject, Metric carries structured identity and measurement semantics, and Report independently represents an optional verdict, normalized quality score, raw numeric measurement, feedback, and child reports. SuiteEvaluator preserves heterogeneous results while CompositeEvaluator explicitly aggregates comparable scored verdicts. Dataset owns case identity, Experiment executes it with bounded concurrency, and ExperimentReport.Compare reports exact aggregate deltas without inventing statistical claims. ProjectionEvaluator adapts aggregate subjects to narrow evaluator inputs.

Domain vocabularies live outside the kernel: judge supplies generic model-backed evaluation, text owns generated-text metrics, ranking owns provider-neutral ranking metrics, and trajectory owns deterministic Agent execution evaluation. New domains implement Evaluator directly and do not depend on another domain's vocabulary.

Index

Examples

Constants

View Source 
const DefaultMaxConcurrency = 4

DefaultMaxConcurrency bounds evaluation fan-out when a host does not choose an explicit limit.

View Source 
const MaxReportDepth = 64

MaxReportDepth bounds recursive detail trees at every public trust boundary.

Variables

View Source 
var (
	ErrInvalidEvaluatorConfig = errors.New("eval: evaluator configuration is invalid")
	ErrInvalidMetric          = errors.New("eval: invalid metric")
	ErrInvalidScore           = errors.New("eval: invalid score")
	ErrInvalidReport          = errors.New("eval: invalid report")
	ErrInvalidCase            = errors.New("eval: invalid case")
	ErrInvalidDataset         = errors.New("eval: invalid dataset")
	ErrInvalidExperiment      = errors.New("eval: invalid experiment")
	ErrInvalidComparison      = errors.New("eval: invalid comparison")
	ErrCaseNotEvaluated       = errors.New("eval: case was not evaluated")
)

Evaluation sentinels classify invalid values at their owning aggregate boundary without introducing domain-specific error taxonomies.

Functions

This section is empty.

Types

type Case 

type Case[T any] struct {
	ID       CaseID
	Subject  T
	Metadata metadata.Map
}

Case gives a stable identity to one evaluation subject.

func (Case[T]) Validate 

func (c Case[T]) Validate() error

type CaseID 

type CaseID string

CaseID is a stable identity within one Dataset.

func (CaseID) String 

func (c CaseID) String() string

func (CaseID) Validate 

func (c CaseID) Validate() error

type CaseResult 

type CaseResult struct {
	ID       CaseID
	Metadata metadata.Map
	Report   Report
	Err      error
}

CaseResult preserves Dataset identity whether evaluation produced a report or an error.

type Comparison 

type Comparison struct {
	Baseline       ExperimentSummary
	Candidate      ExperimentSummary
	EvaluatedDelta int
	PassedDelta    int
	FailedDelta    int
	UnjudgedDelta  int
	ErrorDelta     int
	Metrics        []MetricComparison
}

Comparison reports candidate-minus-baseline deltas without inventing statistical significance.

type Component 

type Component[T any] struct {
	Evaluator Evaluator[T]
	Weight    float64
	Required  bool
}

Component assigns score weight and pass criticality to one evaluator. A zero Weight selects 1. Required components must pass independently of the aggregate pass policy.

type CompositeConfig 

type CompositeConfig[T any] struct {
	Components     []Component[T]
	PassPolicy     PassPolicy
	MinimumPassed  int
	MaxConcurrency int
}

CompositeConfig defines score aggregation, pass semantics, and bounded concurrency. A zero MaxConcurrency selects DefaultMaxConcurrency.

type CompositeEvaluator 

type CompositeEvaluator[T any] struct {
	// contains filtered or unexported fields
}

CompositeEvaluator combines only scored, decided child reports under one explicit weighting and pass policy.

func NewCompositeEvaluator 

func NewCompositeEvaluator[T any](config CompositeConfig[T]) (*CompositeEvaluator[T], error)

NewCompositeEvaluator snapshots components and resolves all zero-value defaults before evaluation begins.

func (*CompositeEvaluator[T]) Evaluate 

func (c *CompositeEvaluator[T]) Evaluate(ctx context.Context, subject T) (Report, error)

type Dataset 

type Dataset[T any] struct {
	// contains filtered or unexported fields
}

Dataset is an immutable ordered set of uniquely identified cases. Evaluators must not mutate subjects; metadata is owned and cloned by the Dataset.

func NewDataset 

func NewDataset[T any](cases ...Case[T]) (Dataset[T], error)

NewDataset snapshots cases and rejects duplicate identity before experiment scheduling can make result correlation ambiguous.

func (Dataset[T]) Cases 

func (d Dataset[T]) Cases() []Case[T]

Cases returns an owned copy in deterministic declaration order.

func (Dataset[T]) Len 

func (d Dataset[T]) Len() int

type Direction 

type Direction string

Direction describes how a raw measurement relates to quality. It is kept separate from Score, whose direction is always higher-is-better. 

const (
	// DirectionUnspecified means the metric has no raw measurement direction.
	DirectionUnspecified Direction = ""
	// DirectionHigherIsBetter marks increasing raw measurements as improvement.
	DirectionHigherIsBetter Direction = "higher_is_better"
	// DirectionLowerIsBetter marks decreasing raw measurements as improvement.
	DirectionLowerIsBetter Direction = "lower_is_better"
)

func (Direction) Validate 

func (d Direction) Validate() error

type Distribution 

type Distribution struct {
	Count   int
	Mean    float64
	Minimum float64
	P10     float64
	P50     float64
	P90     float64
	Maximum float64
}

Distribution summarizes one homogeneous numeric signal. Count distinguishes an absent distribution from a real distribution whose values are all zero.

type DistributionDelta 

type DistributionDelta struct {
	Present bool
	Mean    float64
}

DistributionDelta is candidate mean minus baseline mean. Present is false when either side has no values, so absence cannot be mistaken for zero.

type ErrorPolicy 

type ErrorPolicy string

ErrorPolicy controls whether independent case failures are collected or stop new scheduling. 

const (
	ErrorCollect  ErrorPolicy = "collect"
	ErrorFailFast ErrorPolicy = "fail_fast"
)

Experiment error policies never hide failures from CaseResult.

type Evaluator 

type Evaluator[T any] interface {
	// Evaluate inspects one subject without mutating it and returns a valid,
	// owned report for the evaluator's metric. Implementations must honor ctx;
	// a non-nil error means the report must not be consumed.
	Evaluate(ctx context.Context, subject T) (Report, error)
}

Evaluator evaluates one subject and returns a valid report.

type EvaluatorFunc 

type EvaluatorFunc[T any] func(context.Context, T) (Report, error)

EvaluatorFunc adapts a function to Evaluator without introducing another evaluation call path.

func (EvaluatorFunc[T]) Evaluate 

func (e EvaluatorFunc[T]) Evaluate(ctx context.Context, subject T) (Report, error)

type Experiment 

type Experiment[T any] struct {
	// contains filtered or unexported fields
}

Experiment is an immutable plan for evaluating one Dataset. It owns bounded scheduling and error semantics, but no persistence, artifacts, or product identity.

func NewExperiment 

func NewExperiment[T any](config ExperimentConfig[T]) (Experiment[T], error)

NewExperiment snapshots the Dataset and resolves bounded scheduling before a run starts.

func (Experiment[T]) Run 

func (e Experiment[T]) Run(ctx context.Context) (ExperimentReport, error)
Example 
package main

import (
	"context"
	"fmt"

	"github.com/Tangerg/scope/eval"
)

func main() {
	metric, err := eval.NewMetric(eval.MetricConfig{
		Namespace: "example",
		Name:      "non_empty",
	})
	if err != nil {
		panic(err)
	}
	evaluator := eval.EvaluatorFunc[string](func(_ context.Context, subject string) (eval.Report, error) {
		return eval.Report{Metric: metric, Verdict: eval.VerdictPass}, nil
	})
	dataset, err := eval.NewDataset(
		eval.Case[string]{ID: "first", Subject: "answer"},
	)
	if err != nil {
		panic(err)
	}
	experiment, err := eval.NewExperiment(eval.ExperimentConfig[string]{
		Dataset: dataset, Evaluator: evaluator,
	})
	if err != nil {
		panic(err)
	}
	report, err := experiment.Run(context.Background())
	if err != nil {
		panic(err)
	}
	summary := report.Summary()

	fmt.Println(summary.Total, summary.Passed)
}

Output:
1 1

type ExperimentConfig 

type ExperimentConfig[T any] struct {
	Dataset        Dataset[T]
	Evaluator      Evaluator[T]
	MaxConcurrency int
	ErrorPolicy    ErrorPolicy
}

ExperimentConfig binds one immutable Dataset to one Evaluator and scheduling policy.

type ExperimentReport 

type ExperimentReport struct {
	// contains filtered or unexported fields
}

ExperimentReport owns ordered case results and the summary derived from them.

func (ExperimentReport) Cases 

func (e ExperimentReport) Cases() []CaseResult

Cases returns owned results in Dataset order.

func (ExperimentReport) Compare 

func (e ExperimentReport) Compare(candidate ExperimentReport) (Comparison, error)

Compare keeps the baseline authoritative: only reports over the same ordered Dataset and Metric identities are comparable. Exact deltas avoid inventing statistical significance or a synthetic score across unlike units.

func (ExperimentReport) Summary 

func (e ExperimentReport) Summary() ExperimentSummary

Summary returns the owned aggregate calculated from Cases.

type ExperimentSummary 

type ExperimentSummary struct {
	Total     int
	Evaluated int
	Passed    int
	Failed    int
	Unjudged  int
	Errors    int
	Metrics   []MetricSummary
}

ExperimentSummary aggregates categorical outcomes and homogeneous metric distributions without collapsing unlike metrics.

type Metric 

type Metric struct {
	// contains filtered or unexported fields
}

Metric identifies an evaluation without encoding configuration into a string. Parameters holds owned, structured identity for calculation and decision rules. Unit and Direction describe optional raw measurements; normalized scores are always unitless and higher-is-better.

func NewMetric 

func NewMetric(config MetricConfig) (Metric, error)

NewMetric snapshots structured parameters so later mutation cannot change report comparability.

func (Metric) Clone 

func (m Metric) Clone() Metric

func (Metric) Direction 

func (m Metric) Direction() Direction

func (Metric) MarshalJSON added in v0.13.0 

func (m Metric) MarshalJSON() ([]byte, error)

func (Metric) Name 

func (m Metric) Name() MetricName

func (Metric) Namespace 

func (m Metric) Namespace() string

func (Metric) Parameters 

func (m Metric) Parameters() metadata.Map

func (Metric) String 

func (m Metric) String() string

func (Metric) Unit 

func (m Metric) Unit() string

func (*Metric) UnmarshalJSON added in v0.13.0 

func (m *Metric) UnmarshalJSON(data []byte) error

func (Metric) Validate 

func (m Metric) Validate() error

type MetricComparison 

type MetricComparison struct {
	Metric           Metric
	Baseline         MetricSummary
	Candidate        MetricSummary
	EvaluatedDelta   int
	PassedDelta      int
	FailedDelta      int
	UnjudgedDelta    int
	ScoreDelta       DistributionDelta
	MeasurementDelta DistributionDelta
}

MetricComparison keeps exact aggregate deltas attached to one full metric identity.

type MetricConfig 

type MetricConfig struct {
	Namespace  string
	Name       MetricName
	Unit       string
	Direction  Direction
	Parameters metadata.Map
}

MetricConfig supplies the full comparison identity of one metric.

type MetricName 

type MetricName string

MetricName identifies one quality calculation within a namespace. 

const MetricNameComposite MetricName = "composite"

MetricNameComposite identifies the explicit score-aggregating evaluator. 

const MetricNameSuite MetricName = "suite"

MetricNameSuite identifies a heterogeneous report that preserves child metrics instead of aggregating unlike scores.

type MetricSummary 

type MetricSummary struct {
	Metric       Metric
	Evaluated    int
	Passed       int
	Failed       int
	Unjudged     int
	Scores       Distribution
	Measurements Distribution
}

MetricSummary keeps score and measurement distributions attached to their full Metric identity so unrelated units, directions, and configurations are never aggregated together. Experiment summarizes both top-level reports and their Details.

type PassPolicy 

type PassPolicy string

PassPolicy controls categorical aggregation independently from score weights. 

const (
	PassAll     PassPolicy = "all"
	PassAny     PassPolicy = "any"
	PassAtLeast PassPolicy = "at_least"
)

Composite pass policies remain explicit so required components and minimum counts cannot be encoded in magic thresholds.

type Projection 

type Projection[T, Subject any] func(T) (Subject, error)

Projection narrows an aggregate case to the subject owned by one evaluator.

type ProjectionEvaluator 

type ProjectionEvaluator[T, Subject any] struct {
	// contains filtered or unexported fields
}

ProjectionEvaluator adapts one aggregate case to the narrower subject a domain evaluator consumes.

func NewProjectionEvaluator 

func NewProjectionEvaluator[T, Subject any](
	evaluator Evaluator[Subject],
	projection Projection[T, Subject],
) (*ProjectionEvaluator[T, Subject], error)

NewProjectionEvaluator keeps domain projection at the edge instead of adding domain nouns to the evaluation kernel.

func (*ProjectionEvaluator[T, Subject]) Evaluate 

func (p *ProjectionEvaluator[T, Subject]) Evaluate(ctx context.Context, value T) (Report, error)

type Report 

type Report struct {
	Metric      Metric       `json:"metric"`
	Verdict     Verdict      `json:"verdict,omitzero"`
	Score       *Score       `json:"score,omitzero"`
	Measurement *float64     `json:"measurement,omitzero"`
	Feedback    string       `json:"feedback,omitzero"`
	Metadata    metadata.Map `json:"metadata,omitzero"`
	Details     []Report     `json:"details,omitzero"`
}

Report is one evaluation result. Verdict, normalized Score, and raw Measurement are independent and optional so measurement-only and qualitative evaluations do not need to invent a pass threshold or quality score. Details contains owned child reports instead of convention-based metadata keys.

func (Report) Clone 

func (r Report) Clone() (Report, error)

Clone validates the complete detail tree before allocating its detached copy.

func (Report) MarshalJSON 

func (r Report) MarshalJSON() ([]byte, error)

func (*Report) UnmarshalJSON 

func (r *Report) UnmarshalJSON(data []byte) error

func (Report) Validate 

func (r Report) Validate() error

type Score 

type Score float64

Score is a normalized quality score in the closed interval [0, 1], where a higher value is always better.

func NewScore 

func NewScore(value float64) (Score, error)

NewScore rejects non-finite and out-of-range values at construction.

func (Score) Float64 

func (s Score) Float64() float64

func (Score) Validate 

func (s Score) Validate() error

func (Score) Verdict 

func (s Score) Verdict(threshold Score) (Verdict, error)

Verdict returns the categorical judgment for a valid threshold.

Example 
package main

import (
	"fmt"

	"github.com/Tangerg/scope/eval"
)

func main() {
	score, err := eval.NewScore(0.82)
	if err != nil {
		panic(err)
	}
	threshold, err := eval.NewScore(0.8)
	if err != nil {
		panic(err)
	}
	verdict, err := score.Verdict(threshold)
	if err != nil {
		panic(err)
	}

	fmt.Println(verdict, score.Float64())
}

Output:
pass 0.82

type SuiteConfig 

type SuiteConfig[T any] struct {
	Evaluators     []Evaluator[T]
	MaxConcurrency int
}

SuiteConfig groups heterogeneous evaluators without collapsing their results into one score. A zero MaxConcurrency selects DefaultMaxConcurrency.

type SuiteEvaluator 

type SuiteEvaluator[T any] struct {
	// contains filtered or unexported fields
}

SuiteEvaluator preserves every child report and records the ordered child metrics in its own identity. Its verdict fails when any decided child fails, passes when at least one child passes and none fail, and remains unspecified when every child is measurement-only or qualitative.

func NewSuiteEvaluator 

func NewSuiteEvaluator[T any](config SuiteConfig[T]) (*SuiteEvaluator[T], error)

NewSuiteEvaluator snapshots the ordered evaluator set and bounds concurrent child evaluation.

func (*SuiteEvaluator[T]) Evaluate 

func (s *SuiteEvaluator[T]) Evaluate(ctx context.Context, subject T) (Report, error)

type Verdict 

type Verdict string

Verdict is an optional categorical judgment. An unspecified verdict is a valid outcome for measurement-only or qualitative evaluations. 

const (
	// VerdictUnspecified represents a valid result with no categorical decision.
	VerdictUnspecified Verdict = ""
	// VerdictPass records satisfaction of an evaluator's explicit rule.
	VerdictPass Verdict = "pass"
	// VerdictFail records failure of an evaluator's explicit rule.
	VerdictFail Verdict = "fail"
)

func (Verdict) Decided 

func (v Verdict) Decided() bool

func (Verdict) Validate 

func (v Verdict) Validate() error

Source Files

View all Source files

Directories

Path Synopsis
Package judge evaluates arbitrary subjects with a structured-output chat model.
 Package judge evaluates arbitrary subjects with a structured-output chat model.
Package ranking evaluates ranked outputs against graded relevance judgments.
 Package ranking evaluates ranked outputs against graded relevance judgments.
Package text evaluates generated text without imposing one shared sample on metrics with different semantic inputs.
 Package text evaluates generated text without imposing one shared sample on metrics with different semantic inputs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.