text

Documentation

Overview

Package text evaluates generated text without imposing one shared sample on metrics with different semantic inputs.

Index

• Constants
• Variables
• type AnswerRelevanceEvaluator
  • func NewAnswerRelevanceEvaluator(config ModelEvaluatorConfig) (*AnswerRelevanceEvaluator, error)
  • func (a *AnswerRelevanceEvaluator) Evaluate(ctx context.Context, sample AnswerRelevanceSample) (eval.Report, error)
• type AnswerRelevanceSample
  • func (a AnswerRelevanceSample) Validate() error
• type CorrectnessEvaluator
  • func NewCorrectnessEvaluator(config ModelEvaluatorConfig) (*CorrectnessEvaluator, error)
  • func (c *CorrectnessEvaluator) Evaluate(ctx context.Context, sample CorrectnessSample) (eval.Report, error)
• type CorrectnessSample
  • func (c CorrectnessSample) Validate() error
• type GroundednessEvaluator
  • func NewGroundednessEvaluator(config ModelEvaluatorConfig) (*GroundednessEvaluator, error)
  • func (g *GroundednessEvaluator) Evaluate(ctx context.Context, sample GroundednessSample) (eval.Report, error)
• type GroundednessSample
  • func (g GroundednessSample) Clone() GroundednessSample
  • func (g GroundednessSample) EvidenceText() string
  • func (g GroundednessSample) Validate() error
• type ModelEvaluatorConfig

Constants

const MetricAnswerRelevance eval.MetricName = "answer_relevance"

const MetricCorrectness eval.MetricName = "correctness"

const MetricGroundedness eval.MetricName = "groundedness"

Variables

var ErrInvalidSample = errors.New("eval/text: invalid sample")

Types

type AnswerRelevanceEvaluator

type AnswerRelevanceEvaluator struct {
	// contains filtered or unexported fields
}

AnswerRelevanceEvaluator scores whether generated output addresses its originating input. Groundedness is intentionally evaluated separately.

func NewAnswerRelevanceEvaluator

func NewAnswerRelevanceEvaluator(config ModelEvaluatorConfig) (*AnswerRelevanceEvaluator, error)

func (*AnswerRelevanceEvaluator) Evaluate

func (a *AnswerRelevanceEvaluator) Evaluate(ctx context.Context, sample AnswerRelevanceSample) (eval.Report, error)

type AnswerRelevanceSample

type AnswerRelevanceSample struct {
	Input  string `json:"input"`
	Output string `json:"output"`
}

func (AnswerRelevanceSample) Validate

func (a AnswerRelevanceSample) Validate() error

type CorrectnessEvaluator

type CorrectnessEvaluator struct {
	// contains filtered or unexported fields
}

func NewCorrectnessEvaluator

func NewCorrectnessEvaluator(config ModelEvaluatorConfig) (*CorrectnessEvaluator, error)

func (*CorrectnessEvaluator) Evaluate

func (c *CorrectnessEvaluator) Evaluate(ctx context.Context, sample CorrectnessSample) (eval.Report, error)

type CorrectnessSample

type CorrectnessSample struct {
	Input     string `json:"input"`
	Output    string `json:"output"`
	Reference string `json:"reference"`
}

func (CorrectnessSample) Validate

func (c CorrectnessSample) Validate() error

type GroundednessEvaluator

type GroundednessEvaluator struct {
	// contains filtered or unexported fields
}

GroundednessEvaluator scores whether generated output is supported by the supplied evidence.

func NewGroundednessEvaluator

func NewGroundednessEvaluator(config ModelEvaluatorConfig) (*GroundednessEvaluator, error)

func (*GroundednessEvaluator) Evaluate

func (g *GroundednessEvaluator) Evaluate(ctx context.Context, sample GroundednessSample) (eval.Report, error)

type GroundednessSample

type GroundednessSample struct {
	Output   string   `json:"output"`
	Evidence []string `json:"evidence"`
}

func (GroundednessSample) Clone

func (g GroundednessSample) Clone() GroundednessSample

func (GroundednessSample) EvidenceText

func (g GroundednessSample) EvidenceText() string

func (GroundednessSample) Validate

func (g GroundednessSample) Validate() error

type ModelEvaluatorConfig

type ModelEvaluatorConfig struct {
	Model          chat.Model
	PromptTemplate *chatclient.Template
	// Threshold is optional. Without one, evaluation produces a score without
	// inventing a pass/fail decision.
	Threshold *eval.Score
	Samples   int
}

ModelEvaluatorConfig configures model-backed text metrics. Each evaluator exposes only the prompt variables its own sample contains. Samples greater than one use the median judge score.