README
¶
eval
Retrieval and RAG evaluation: datasets with ground truth, ranking metrics, regression comparison, and (optionally) LLM-judged answer quality.
Retrieval evaluation
ds := eval.NewDataset("myset")
ds.Add(eval.EvalQuery{
Query: "which company designed go",
RelevantIDs: []string{"chunk-1"}, // binary relevance
Relevance: map[string]int{"chunk-1": 2}, // optional graded (NDCG)
})
suite := eval.NewBenchmarkSuite(ds, k)
report, err := suite.Run(ctx, retriever) // retriever implements Retriever
Retriever—Retrieve(ctx, query, k) ([]string, error); adapt any store with a 10-line wrapper (seeexample/e2e).- Metrics:
PrecisionAtK,RecallAtK,MRR,NDCGAtK,ComputeRetrievalMetrics;Reportaggregates means per dataset. Compare(current, baseline, tolerance)— regression gate used byrecall eval compare(exit 2 on regression, CI-friendly).
RAG (answer) evaluation
RAGEval (NewRAGEval(judge)) + RunWithAnswers scores generated
answers (faithfulness, correctness) using a pluggable Judge (LLM or
heuristic) over AnswerQuality per query.
Datasets persist as JSON: Dataset.Save(path) / LoadDataset(path).
Documentation
¶
Overview ¶
Package eval provides evaluation metrics and a benchmark suite for measuring and regression-testing retrieval and RAG quality.
The package is dependency-free. Retrieval metrics (Precision@K, Recall@K, MRR, NDCG@K) are pure functions over ranked ID lists and ground truth, and answer-quality metrics use a pluggable Judge (a deterministic lexical judge is included; an LLM judge can be plugged in via the Judge interface).
Index ¶
- func MRR(retrieved []string, relevant map[string]bool) float64
- func NDCGAtK(retrieved []string, relevance map[string]int, k int) float64
- func PrecisionAtK(retrieved []string, relevant map[string]bool, k int) float64
- func RecallAtK(retrieved []string, relevant map[string]bool, k int) float64
- type AnswerQuality
- type Answerer
- type BenchmarkSuite
- type Comparison
- type Dataset
- type EvalQuery
- type Judge
- type JudgeInput
- type MetricDelta
- type OverlapJudge
- type PerQueryResult
- type RAGEval
- type Report
- type RetrievalMetrics
- type RetrievalSystem
- type Retriever
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MRR ¶
MRR returns the reciprocal rank of the first relevant result in the retrieved list (1.0 if the first result is relevant, 0.5 if the second, ...). Returns 0 when no result is relevant. The full list is considered.
func NDCGAtK ¶
NDCGAtK returns the normalized discounted cumulative gain at cutoff K. Relevance grades come from the relevance map (grade 0 when absent). When there is no graded relevance (all grades <= 0), the result is 0.
func PrecisionAtK ¶
PrecisionAtK returns the fraction of the top-K retrieved items that are in the relevant set. Returns 0 when k <= 0.
Types ¶
type AnswerQuality ¶
type AnswerQuality struct {
// Faithfulness measures how much of the answer is supported by the context.
Faithfulness float64
// Relevance measures how much of the question the answer addresses.
Relevance float64
// Correctness measures agreement with the reference answer (0 when no
// reference is provided).
Correctness float64
// Comment is an optional explanation from the judge.
Comment string
}
AnswerQuality holds the 0-1 answer-quality scores.
type Answerer ¶
type Answerer interface {
// Answer returns an answer to the question grounded on the context.
Answer(ctx context.Context, question, context string) (string, error)
}
Answerer produces an answer for a question given the retrieved context.
type BenchmarkSuite ¶
type BenchmarkSuite struct {
// Dataset is the evaluation dataset to run.
Dataset *Dataset
// K is the number of results to retrieve per query.
K int
}
BenchmarkSuite runs an evaluation dataset against a retrieval system and supports regression comparison against a baseline report.
func NewBenchmarkSuite ¶
func NewBenchmarkSuite(ds *Dataset, k int) *BenchmarkSuite
NewBenchmarkSuite creates a BenchmarkSuite for a dataset at cutoff K.
func (*BenchmarkSuite) Run ¶
Run evaluates retrieval quality for every query in the dataset and returns an aggregate report.
func (*BenchmarkSuite) RunWithAnswers ¶
func (s *BenchmarkSuite) RunWithAnswers(ctx context.Context, sys RetrievalSystem, a Answerer, j Judge) (*Report, error)
RunWithAnswers runs retrieval, generates answers, and scores both retrieval and answer quality (faithfulness, relevance, correctness).
type Comparison ¶
type Comparison struct {
// Baseline is the reference report.
Baseline *Report
// Current is the report being checked.
Current *Report
// Deltas lists every compared metric.
Deltas []MetricDelta
// Regressions are metrics that dropped by more than the tolerance.
Regressions []MetricDelta
// Improvements are metrics that rose by more than the tolerance.
Improvements []MetricDelta
// Passed is true when there are no regressions.
Passed bool
}
Comparison is the result of comparing a current report against a baseline.
func Compare ¶
func Compare(current, baseline *Report, tolerance float64) *Comparison
Compare checks a current report against a baseline for regressions. A metric is a regression when it drops by more than tolerance (absolute); a tolerance of 0 flags any drop. Answer-quality metrics are only compared when both reports contain them.
type Dataset ¶
type Dataset struct {
// Name identifies the dataset.
Name string
// Version is an optional free-form version string.
Version string `json:"version,omitempty"`
// Queries is the ordered list of evaluation cases.
Queries []EvalQuery
}
Dataset is a named, versioned collection of evaluation queries that can be loaded from and saved to a JSON file.
func LoadDataset ¶
LoadDataset reads a JSON dataset from path.
func NewDataset ¶
NewDataset creates an empty dataset with the given name.
type EvalQuery ¶
type EvalQuery struct {
// ID is a stable identifier for this case (defaults to the query text).
ID string
// Query is the question to evaluate.
Query string
// RelevantIDs are the ground-truth relevant chunk IDs (binary relevance).
RelevantIDs []string
// Relevance is optional graded relevance (chunk ID -> grade, higher is
// better). When present, it is used for NDCG and overrides RelevantIDs.
Relevance map[string]int `json:"relevance,omitempty"`
// Context is the ground-truth supporting context, used for answer-quality
// (faithfulness) evaluation.
Context string `json:"context,omitempty"`
// ReferenceAnswer is the ground-truth answer, used for correctness
// evaluation when provided.
ReferenceAnswer string `json:"reference_answer,omitempty"`
}
EvalQuery is a single evaluation case: a query plus its ground truth.
type Judge ¶
type Judge interface {
// Judge returns answer-quality scores for the input.
Judge(ctx context.Context, in JudgeInput) (AnswerQuality, error)
}
Judge scores an answer against a question, context, and optional reference. A deterministic lexical judge (OverlapJudge) is provided; an LLM-based judge can be plugged in by implementing this interface.
type JudgeInput ¶
type JudgeInput struct {
// Question is the user's question.
Question string
// Context is the retrieved context the answer was grounded on.
Context string
// Answer is the generated answer to evaluate.
Answer string
// Reference is an optional ground-truth answer for correctness scoring.
Reference string
}
JudgeInput is the material a Judge scores for answer quality.
type MetricDelta ¶
type MetricDelta struct {
// Name is the metric name (e.g., "mean_mrr").
Name string
// Baseline is the metric value in the baseline report.
Baseline float64
// Current is the metric value in the current report.
Current float64
// Delta is Current - Baseline (negative is a drop).
Delta float64
}
MetricDelta is the change of a single metric between a baseline and a current report.
type OverlapJudge ¶
type OverlapJudge struct{}
OverlapJudge is a deterministic, dependency-free Judge that approximates answer quality using lexical overlap:
- Faithfulness: fraction of the answer's content words that appear in the context (claims supported by the context).
- Relevance: fraction of the question's content words that appear in the answer.
- Correctness: token F1 between the answer and the reference answer (0 when no reference is provided).
It is intended as a fast, reproducible baseline and for tests; plug in an LLM Judge for semantic judgment.
func NewOverlapJudge ¶
func NewOverlapJudge() *OverlapJudge
NewOverlapJudge creates a lexical OverlapJudge.
func (*OverlapJudge) Judge ¶
func (j *OverlapJudge) Judge(ctx context.Context, in JudgeInput) (AnswerQuality, error)
Judge implements Judge using lexical overlap.
type PerQueryResult ¶
type PerQueryResult struct {
// ID is the evaluation case ID.
ID string
// Query is the query text.
Query string
// Metrics holds the retrieval metrics for this query.
Metrics RetrievalMetrics
// Answer is the generated answer (empty for retrieval-only runs).
Answer string
// Quality holds the answer-quality scores (zero when not evaluated).
Quality AnswerQuality
}
PerQueryResult holds the evaluation outcome for a single query.
type RAGEval ¶
type RAGEval struct {
// Judge scores answers.
Judge Judge
}
RAGEval evaluates the quality of RAG answers using a pluggable Judge.
func NewRAGEval ¶
NewRAGEval creates a RAGEval with the given judge.
func (*RAGEval) Evaluate ¶
func (e *RAGEval) Evaluate(ctx context.Context, in JudgeInput) (AnswerQuality, error)
Evaluate scores a single (question, context, answer) triple.
type Report ¶
type Report struct {
// Dataset is the dataset name this report was produced from.
Dataset string `json:"dataset"`
// K is the cutoff used for the @K metrics.
K int `json:"k"`
// NumQueries is the number of queries evaluated.
NumQueries int `json:"num_queries"`
// MeanPrecision is the mean Precision@K across queries.
MeanPrecision float64 `json:"mean_precision"`
// MeanRecall is the mean Recall@K across queries.
MeanRecall float64 `json:"mean_recall"`
// MeanMRR is the mean MRR across queries.
MeanMRR float64 `json:"mean_mrr"`
// MeanNDCG is the mean NDCG@K across queries.
MeanNDCG float64 `json:"mean_ndcg"`
// HasAnswerMetrics reports whether answer-quality metrics are present.
HasAnswerMetrics bool `json:"has_answer_metrics,omitempty"`
// MeanFaithfulness is the mean faithfulness score (when evaluated).
MeanFaithfulness float64 `json:"mean_faithfulness,omitempty"`
// MeanAnswerRelevance is the mean answer-relevance score (when evaluated).
MeanAnswerRelevance float64 `json:"mean_answer_relevance,omitempty"`
// MeanCorrectness is the mean correctness score (when evaluated).
MeanCorrectness float64 `json:"mean_correctness,omitempty"`
// PerQuery holds the per-query breakdown.
PerQuery []PerQueryResult `json:"per_query"`
// GeneratedAt is when the report was produced.
GeneratedAt time.Time `json:"generated_at"`
}
Report aggregates evaluation results across a dataset.
func LoadReport ¶
LoadReport reads a JSON report from path. It is the counterpart to SaveJSON and enables golden-file regression testing in CI.
func NewReportFromResults ¶
func NewReportFromResults(dataset string, k int, results []PerQueryResult) *Report
NewReportFromResults builds a Report from per-query results, computing the aggregate means.
type RetrievalMetrics ¶
type RetrievalMetrics struct {
// Query is the evaluated query text.
Query string
// K is the cutoff used for the @K metrics.
K int
// Precision is Precision@K (fraction of top-K results that are relevant).
Precision float64
// Recall is Recall@K (fraction of relevant items retrieved in top-K).
Recall float64
// MRR is the reciprocal rank of the first relevant result (1/rank, 0 if none).
MRR float64
// NDCG is NDCG@K using graded (or binary) relevance.
NDCG float64
// NumRelevant is the total number of ground-truth relevant items.
NumRelevant int
// NumRetrieved is the number of results actually returned.
NumRetrieved int
}
RetrievalMetrics holds the computed metrics for a single query.
func ComputeRetrievalMetrics ¶
func ComputeRetrievalMetrics(query string, retrieved []string, relevantIDs []string, gradedRelevance map[string]int, k int) RetrievalMetrics
ComputeRetrievalMetrics computes Precision@K, Recall@K, MRR, and NDCG@K for a single query given its ranked results and ground-truth relevance.
If gradedRelevance is non-nil it is used for NDCG (and as the relevance set for the other metrics, treating grade > 0 as relevant). Otherwise relevantIDs define a binary relevance set (grade 1).
type RetrievalSystem ¶
type RetrievalSystem interface {
Retriever
// ContextText returns the text content of a chunk by ID.
ContextText(id string) (string, bool)
}
RetrievalSystem retrieves ranked chunk IDs and can resolve their text. It is satisfied by a thin adapter over store.Store (Retrieve via Search, ContextText via GetChunk).
type Retriever ¶
type Retriever interface {
// Retrieve returns up to k chunk IDs ranked by relevance for the query.
Retrieve(ctx context.Context, query string, k int) ([]string, error)
}
Retriever retrieves ranked chunk IDs for a query. It is intentionally small so it can be satisfied by a store, an index, or a thin adapter in tests.
Source Files