scoring

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package scoring ...

Package scoring provides cost-controlled scoring infrastructure for the evolution system, including strategy hashing, score caching, and tiered scorer pipelines.

Package scoring ...

Package scoring provides memory-aware scoring that extends the tiered scorer with evidence-based bonuses and cost/latency penalties derived from past experiences stored in the experience repository.

Index

Constants

View Source 
const (
	ScorerTypeLLM       = "llm"
	ScorerTypeHeuristic = "heuristic"
)

Scorer type constants for cache entries and tier names.

View Source 
const DefaultTaskType = "default"

DefaultTaskType is the fallback task type when no task type information is found.

Variables

View Source 
var (
	ErrNilCache            = errors.New("cache must not be nil")
	ErrNilUnderlyingScorer = errors.New("underlying scorer must not be nil")
	ErrNilStrategy         = errors.New("strategy must not be nil")
	ErrNilTieredCache      = errors.New("cache must not be nil")
	ErrNilBudget           = errors.New("budget must not be nil")
	ErrNilHeuristicScorer  = errors.New("heuristic scorer must not be nil")
	ErrInvalidBudgetLimit  = errors.New("max LLM calls must be > 0")
)

Functions

func StrategyHash 

func StrategyHash(s *mutation.Strategy) (uint64, error)

StrategyHash computes a stable 64-bit hash for a strategy. Two strategies with the same params, prompt template, tool config, and model config will produce the same hash regardless of creation time or ID.

Hash components (order matters for stability):

  • Sorted params (key-value pairs, values converted to string)
  • PromptTemplate
  • Tools from Params["tools"] (if present)
  • Model from Params["model"] (if present)

Metadata fields that are excluded from the hash:

  • Score, ID, ParentID, Version, CreatedAt, MutationDesc, Name, StrategyMutationType — these are metadata, not identity.

Args: 

s - the strategy to hash (must not be nil).

Returns: 

uint64 - the computed hash value.
error - non-nil if s is nil.

Types

type Budget 

type Budget struct {
	// MaxLLMCalls is the maximum number of LLM scorer calls allowed per generation.
	// Immutable after construction.
	MaxLLMCalls int64

	// UsedLLMCalls is the number of LLM scorer calls made in the current generation.
	UsedLLMCalls atomic.Int64

	// CacheHits is the number of score lookups served from cache.
	CacheHits atomic.Int64

	// FallbackCount is the number of times LLM scoring failed and fell back.
	FallbackCount atomic.Int64
}

Budget holds LLM scoring resource limits and current usage for one evolution generation.

func NewBudget 

func NewBudget(maxLLMCalls int) (*Budget, error)

NewBudget creates a new scoring budget.

Args: 

maxLLMCalls - maximum LLM calls allowed per generation (must be > 0).

Returns: 

*Budget - the budget instance.
error - non-nil if maxLLMCalls <= 0.

func (*Budget) CanCallLLM 

func (b *Budget) CanCallLLM() bool

CanCallLLM checks if an LLM call is still within budget.

Returns: 

bool - true if an LLM call can be made.

func (*Budget) RecordCacheHit 

func (b *Budget) RecordCacheHit()

RecordCacheHit records a cache hit (does not consume budget).

func (*Budget) RecordFallback 

func (b *Budget) RecordFallback()

RecordFallback records a fallback to heuristic scoring.

func (*Budget) Reset 

func (b *Budget) Reset()

Reset resets usage counters for a new generation while keeping limits.

func (*Budget) TryRecordLLMCall added in v0.2.4 

func (b *Budget) TryRecordLLMCall() bool

TryRecordLLMCall atomically checks the budget and records a call if within limit.

Returns true if the call was recorded (budget allowed), false if at capacity.

func (*Budget) Usage 

func (b *Budget) Usage() (used, max, cacheHits, fallbacks int)

Usage returns current budget utilization.

Returns: 

used - LLM calls used.
max - max LLM calls allowed.
cacheHits - cache hit count.
fallbacks - fallback count.

type CacheEntry 

type CacheEntry struct {
	// Hash is the strategy hash this entry corresponds to.
	Hash uint64

	// Score is the cached fitness score.
	Score float64

	// ScorerType identifies which scorer produced this score (e.g., "heuristic", "llm", "arena").
	ScorerType string

	// Timestamp when this entry was created (Unix nanos).
	Timestamp int64

	// SampleCount is how many evaluation samples contributed to this score.
	SampleCount int

	// Confidence is the confidence level (0-1) of this score.
	Confidence float64
}

CacheEntry holds a cached score record for a strategy.

func MakeEntry 

func MakeEntry(hash uint64, score float64, scorerType string, sampleCount int, confidence float64) CacheEntry

MakeEntry constructs a CacheEntry with the current timestamp.

Args: 

hash - the strategy hash.
score - the fitness score.
scorerType - label identifying the scorer (e.g., "llm", "heuristic").
sampleCount - number of evaluation samples that contributed.
confidence - confidence level (0-1).

Returns: 

CacheEntry - the constructed cache entry.

type CachedScorer 

type CachedScorer struct {
	// contains filtered or unexported fields
}

CachedScorer wraps a genome.ScorerFunc with a score cache. Before calling the underlying scorer, it checks if an equivalent strategy has been scored before and returns the cached result if available.

func NewCachedScorer 

func NewCachedScorer(cache *ScoreCache, underlying genome.ScorerFunc, scorerType string) (*CachedScorer, error)

NewCachedScorer creates a cache-backed scorer wrapper.

Args: 

cache - the score cache to use (must not be nil).
underlying - the actual scoring function to call on cache miss.
scorerType - label for cache entries (e.g., "llm", "heuristic").

Returns: 

*CachedScorer - the wrapped scorer.
error - non-nil if cache or underlying is nil.

func (*CachedScorer) Score 

func (cs *CachedScorer) Score(ctx context.Context, s *mutation.Strategy) (float64, bool, error)

Score evaluates a strategy, using cache when possible.

On cache hit: returns the cached score with cached=true and no error. On cache miss: calls the underlying scorer, caches the result, and returns it with cached=false.

Args: 

ctx - operation context.
s - the strategy to score.

Returns: 

float64 - the fitness score.
bool - true if score came from cache.
error - non-nil if scoring fails.

type EvidenceAggregatorProvider added in v0.2.5 

type EvidenceAggregatorProvider struct {
	// contains filtered or unexported fields
}

EvidenceAggregatorProvider implements EvidenceProvider by bridging to an EvidenceAggregator. It returns multi-dimensional aggregated evidence (success_rate, latency_p50, error_rate, etc.) for more nuanced scoring.

func NewEvidenceAggregatorProvider added in v0.2.5 

func NewEvidenceAggregatorProvider(aggregator experience.EvidenceAggregator) *EvidenceAggregatorProvider

NewEvidenceAggregatorProvider creates an EvidenceAggregatorProvider.

Args: 

aggregator - the evidence aggregator to query (must not be nil).

Returns: 

*EvidenceAggregatorProvider - the initialized provider.

func (*EvidenceAggregatorProvider) GetEvidence added in v0.2.5 

func (p *EvidenceAggregatorProvider) GetEvidence(ctx context.Context, strategyID string) (experience.Evidence, error)

GetEvidence returns multi-dimensional evidence for a specific strategy.

func (*EvidenceAggregatorProvider) GetEvidenceByTaskType added in v0.2.5 

func (p *EvidenceAggregatorProvider) GetEvidenceByTaskType(ctx context.Context, taskType string) (experience.Evidence, error)

GetEvidenceByTaskType returns multi-dimensional evidence aggregated across all strategies for a specific task type.

type EvidenceProvider added in v0.2.5 

type EvidenceProvider interface {
	// GetEvidence returns multi-dimensional evidence for a specific strategy.
	//
	// Args:
	//
	//	ctx - operation context.
	//	strategyID - the identifier of the strategy to retrieve evidence for.
	//
	// Returns:
	//
	//	Evidence - multi-dimensional aggregated statistics.
	//	error - non-nil if the query fails.
	GetEvidence(ctx context.Context, strategyID string) (experience.Evidence, error)

	// GetEvidenceByTaskType returns multi-dimensional evidence aggregated
	// across all strategies for a specific task type.
	//
	// Args:
	//
	//	ctx - operation context.
	//	taskType - the type of task to retrieve evidence for.
	//
	// Returns:
	//
	//	Evidence - multi-dimensional aggregated statistics.
	//	error - non-nil if the query fails.
	GetEvidenceByTaskType(ctx context.Context, taskType string) (experience.Evidence, error)
}

EvidenceProvider defines the interface for retrieving multi-dimensional evidence that can inform strategy scoring. This interface provides aggregated statistics (success rate, latency, error rate, etc.) rather than simple count and confidence values.

type ExperienceProvider added in v0.2.4 

type ExperienceProvider interface {
	// FindSimilar returns the count of similar experiences for the given
	// task type along with a confidence factor (0-1) indicating how well
	// the matched experiences align with the current context.
	//
	// Args:
	//
	//	ctx - operation context.
	//	taskType - the type of task being evaluated.
	//	limit - maximum number of similar experiences to consider.
	//
	// Returns:
	//
	//	int - count of similar experiences found.
	//	float64 - confidence factor in [0, 1].
	//	error - non-nil if the query fails.
	FindSimilar(ctx context.Context, taskType string, limit int) (int, float64, error)
}

ExperienceProvider defines the interface for retrieving similar past experiences that can inform strategy scoring. Implementations may query a vector database, keyword index, or other experience store.

type ExperienceStoreProvider added in v0.2.5 

type ExperienceStoreProvider struct {
	// contains filtered or unexported fields
}

ExperienceStoreProvider implements ExperienceProvider by bridging to an ExperienceStore. It queries historical execution data and computes confidence scores from actual strategy performance.

func NewExperienceStoreProvider added in v0.2.5 

func NewExperienceStoreProvider(store experience.ExperienceStore) *ExperienceStoreProvider

NewExperienceStoreProvider creates an ExperienceStoreProvider.

Args: 

store - the experience store to query (must not be nil).

Returns: 

*ExperienceStoreProvider - the initialized provider.

func (*ExperienceStoreProvider) FindSimilar added in v0.2.5 

func (p *ExperienceStoreProvider) FindSimilar(ctx context.Context, taskType string, limit int) (int, float64, error)

FindSimilar queries the experience store for strategies that ran the same task type and returns the match count plus an average-score-based confidence.

Confidence is the mean NormalizedExperience.Score across matching records (range [0, 1] — higher means the store has more consistently good results for this task type).

type MemoryAwareScorer added in v0.2.4 

type MemoryAwareScorer struct {
	// contains filtered or unexported fields
}

MemoryAwareScorer extends a TieredScorer with experience-driven bonuses and cost/latency penalties. It wraps the tiered scorer pipeline and adjusts scores based on evidence from past experiences.

When the ExperienceProvider is nil or the scorer is disabled, it behaves exactly like the underlying tiered scorer with no adjustments.

The scorer supports two modes:

  1. Legacy mode (ExperienceProvider only): uses simple count and confidence for memory bonus calculation.
  2. Evidence mode (EvidenceProvider available): uses multi-dimensional evidence (success_rate, latency_p50, error_rate) for more nuanced scoring adjustments.

func NewMemoryAwareScorer added in v0.2.4 

func NewMemoryAwareScorer(ts *TieredScorer, exp ExperienceProvider, cfg MemoryAwareScoringConfig) (*MemoryAwareScorer, error)

NewMemoryAwareScorer creates a new memory-aware scorer wrapping a tiered scorer.

Args: 

ts - the tiered scorer pipeline to wrap (must not be nil).
exp - the experience provider (may be nil, in which case the scorer
  behaves like a regular tiered scorer).
cfg - the memory-aware scoring configuration (use
  DefaultMemoryAwareScoringConfig() for defaults).

Returns: 

*MemoryAwareScorer - the configured scorer.
error - non-nil if tiered scorer is nil or configuration is invalid.

func (*MemoryAwareScorer) ResetStats added in v0.2.4 

func (ms *MemoryAwareScorer) ResetStats()

ResetStats resets the memory-aware scorer statistics.

func (*MemoryAwareScorer) Score added in v0.2.4 

func (ms *MemoryAwareScorer) Score(ctx context.Context, s *mutation.Strategy) (float64, *ScoreDetail, error)

Score evaluates a strategy through the tiered pipeline and applies memory-aware adjustments. The final fitness is computed as: 

fitness = quality_score + memory_evidence_bonus - cost_penalty
          - latency_penalty - regression_penalty

If the experience provider is nil or the scorer is not enabled, this delegates directly to the underlying tiered scorer.

When EvidenceProvider is available, the scorer uses multi-dimensional evidence (success_rate, latency_p50, error_rate) for more nuanced adjustments. Otherwise, it falls back to legacy ExperienceProvider.

Args: 

ctx - operation context.
s - the strategy to score.

Returns: 

float64 - the adjusted fitness score.
*ScoreDetail - breakdown of score components (nil if not enabled or
  experience provider is nil).
error - non-nil if scoring fails.

func (*MemoryAwareScorer) ScoreAsScorerFunc added in v0.2.4 

func (ms *MemoryAwareScorer) ScoreAsScorerFunc() genome.ScorerFunc

ScoreAsScorerFunc returns a genome.ScorerFunc that wraps the MemoryAwareScorer. This allows the memory-aware scorer to be used wherever a ScorerFunc is expected (e.g., in genome.Population.ScoreAgents).

When the scorer is enabled and has an experience provider, the score detail is logged rather than returned (since ScorerFunc only returns a float64).

Returns: 

genome.ScorerFunc - function that scores strategies with memory awareness.

func (*MemoryAwareScorer) SetEvidenceProvider added in v0.2.5 

func (ms *MemoryAwareScorer) SetEvidenceProvider(ep EvidenceProvider)

SetEvidenceProvider sets the evidence provider for multi-dimensional scoring adjustments. This allows the scorer to use EvidenceProvider in addition to or instead of the legacy ExperienceProvider.

Args: 

ep - the evidence provider (may be nil).

func (*MemoryAwareScorer) Stats added in v0.2.4 

func (ms *MemoryAwareScorer) Stats() map[string]float64

Stats returns scoring statistics since creation or last reset.

Returns: 

map[string]float64 - statistics including adjustments count, avg_bonus,
  avg_penalty, and delegate tiered scorer stats.

type MemoryAwareScoringConfig added in v0.2.4 

type MemoryAwareScoringConfig struct {
	// Enabled enables memory-aware scoring adjustments.
	Enabled bool `json:"enabled"`

	// MemoryWeight controls the contribution of memory evidence bonus to
	// the final score (default 0.2).
	MemoryWeight float64 `json:"memory_weight"`

	// CostWeight controls the penalty multiplier for strategy cost
	// (default 0.1).
	CostWeight float64 `json:"cost_weight"`

	// LatencyWeight controls the penalty multiplier for strategy latency
	// in seconds (default 0.05).
	LatencyWeight float64 `json:"latency_weight"`

	// RegressionWeight controls the penalty multiplier for score regression
	// compared to a known baseline (default 0.1).
	RegressionWeight float64 `json:"regression_weight"`

	// MinEvidenceBonus is the minimum memory evidence bonus (default 0.0).
	MinEvidenceBonus float64 `json:"min_evidence_bonus"`

	// MaxEvidenceBonus is the maximum memory evidence bonus (default 20.0).
	MaxEvidenceBonus float64 `json:"max_evidence_bonus"`

	// ExperienceLookupLimit is the maximum number of similar experiences to
	// retrieve per scoring call (default 10).
	ExperienceLookupLimit int `json:"experience_lookup_limit"`

	// SuccessRateBonusScale controls the bonus multiplier for success rate
	// in evidence-based scoring (default 10.0).
	// Formula: success_rate * confidence * SuccessRateBonusScale.
	SuccessRateBonusScale float64 `json:"success_rate_bonus_scale"`

	// LatencyPenaltyScale controls the penalty multiplier for latency_p50
	// in evidence-based scoring (default 1.0).
	// Formula: (latency_p50 / 10000) * LatencyPenaltyScale.
	LatencyPenaltyScale float64 `json:"latency_penalty_scale"`

	// ErrorRatePenaltyScale controls the penalty multiplier for error rate
	// in evidence-based scoring (default 1.0).
	// Formula: error_rate * confidence * ErrorRatePenaltyScale.
	ErrorRatePenaltyScale float64 `json:"error_rate_penalty_scale"`
}

MemoryAwareScoringConfig holds configuration for memory-aware scoring.

func DefaultMemoryAwareScoringConfig added in v0.2.4 

func DefaultMemoryAwareScoringConfig() MemoryAwareScoringConfig

DefaultMemoryAwareScoringConfig returns sensible defaults for memory-aware scoring configuration.

type ScoreCache 

type ScoreCache struct {
	// contains filtered or unexported fields
}

ScoreCache provides thread-safe score caching for evolved strategies. It avoids redundant LLM calls by caching previously computed scores.

Uses LRU eviction via container/list when at capacity. Zero-value is NOT usable; use NewScoreCache to create an instance.

func NewScoreCache 

func NewScoreCache(maxSize int) *ScoreCache

NewScoreCache creates a new score cache.

Args: 

maxSize - maximum entries (0 = unlimited).

Returns: 

*ScoreCache - the cache instance.

func (*ScoreCache) Clear 

func (c *ScoreCache) Clear()

Clear removes all entries from the cache and resets statistics.

func (*ScoreCache) Get 

func (c *ScoreCache) Get(hash uint64) (CacheEntry, bool)

Get retrieves a cached score for the given strategy hash. This method is thread-safe and promotes the entry as most recently used. Entries older than maxCacheAge generations are treated as misses.

Args: 

hash - the strategy hash to look up.

Returns: 

CacheEntry - the cached entry, zero value if not found.
bool - true if found in cache.

func (*ScoreCache) NewGeneration added in v0.2.5 

func (c *ScoreCache) NewGeneration()

NewGeneration advances the generation counter. Entries older than maxCacheAge will be treated as misses on subsequent Get() calls.

func (*ScoreCache) Put 

func (c *ScoreCache) Put(hash uint64, entry CacheEntry)

Put stores a score in the cache. If the cache is full, evicts the least recently used entry. This method is thread-safe and uses a write lock.

Args: 

hash - the strategy hash.
entry - the cache entry to store.

func (*ScoreCache) SetMaxCacheAge added in v0.2.5 

func (c *ScoreCache) SetMaxCacheAge(generations int)

SetMaxCacheAge sets the maximum number of generations an entry is valid. After this many generations, the entry is treated as a cache miss and re-evaluated on the next lookup. 0 (default) means unlimited — entries never expire by age. Use NewGeneration() to advance the counter.

func (*ScoreCache) Stats 

func (c *ScoreCache) Stats() (hits, misses, size, evictions int64)

Stats returns cache statistics.

Returns: 

hits - number of cache hits since creation (or last ResetStats).
misses - number of cache misses.
size - current entry count.
evictions - number of entries evicted due to capacity.

type ScoreDetail added in v0.2.4 

type ScoreDetail struct {
	// QualityScore is the base score from the underlying scorer pipeline.
	QualityScore float64 `json:"quality_score"`

	// MemoryEvidenceBonus is the bonus from matching past experiences.
	MemoryEvidenceBonus float64 `json:"memory_evidence_bonus"`

	// CostPenalty is the penalty applied for strategy execution cost.
	CostPenalty float64 `json:"cost_penalty"`

	// LatencyPenalty is the penalty applied for strategy latency.
	LatencyPenalty float64 `json:"latency_penalty"`

	// RegressionPenalty is the penalty for score regression.
	RegressionPenalty float64 `json:"regression_penalty"`

	// FinalScore is the sum: quality + memory - cost - latency - regression.
	FinalScore float64 `json:"final_score"`

	// ExperienceCount is the number of similar experiences found.
	ExperienceCount int `json:"experience_count"`

	// Confidence is the confidence factor in [0, 1].
	Confidence float64 `json:"confidence"`

	// SuccessRateEvidence is the success rate from multi-dimensional evidence.
	// Range: [0.0, 1.0]. Higher values indicate better historical performance.
	SuccessRateEvidence float64 `json:"success_rate_evidence"`

	// LatencyEvidence is the P50 latency in milliseconds from multi-dimensional evidence.
	// Lower values indicate better (faster) historical performance.
	LatencyEvidence int64 `json:"latency_evidence"`

	// ErrorRateEvidence is the error rate from multi-dimensional evidence.
	// Range: [0.0, 1.0]. Lower values indicate better historical performance.
	ErrorRateEvidence float64 `json:"error_rate_evidence"`

	// SampleCountEvidence is the number of samples aggregated in the evidence.
	// Higher values indicate more reliable statistics.
	SampleCountEvidence int64 `json:"sample_count_evidence"`
}

ScoreDetail provides a breakdown of the individual components that contributed to a strategy's final fitness score.

type Tier 

type Tier int

Tier defines a scoring tier in the pipeline. 

const (
	// TierCache checks the score cache first.
	TierCache Tier = iota + 1
	// TierHeuristic applies fast, cheap scoring.
	TierHeuristic
	// TierLLM uses expensive LLM-based scoring (budget-gated).
	TierLLM
)

func (Tier) String 

func (t Tier) String() string

String returns a human-readable name for the tier.

type TieredScorer 

type TieredScorer struct {
	// contains filtered or unexported fields
}

TieredScorer implements multi-tier scoring with budget control and caching. It routes each strategy through the most cost-effective tier that can provide a valid score.

func NewTieredScorer 

func NewTieredScorer(cfg TieredScorerConfig) (*TieredScorer, error)

NewTieredScorer creates a new tiered scorer pipeline.

Args: 

cfg - configuration (Cache, Budget, and HeuristicScorer must not be nil).

Returns: 

*TieredScorer - the configured tiered scorer.
error - non-nil if required dependencies are nil.

func (*TieredScorer) ResetForGeneration 

func (ts *TieredScorer) ResetForGeneration()

ResetForGeneration resets budget and per-generation stats. Call this at the start of each evolution generation.

func (*TieredScorer) Score 

func (ts *TieredScorer) Score(ctx context.Context, s *mutation.Strategy) (float64, Tier, error)

Score evaluates a strategy through the tiered pipeline. The flow is:

  1. Check cache → return if hit (TierCache)
  2. If LLM scorer exists and budget allows → try LLM (TierLLM)
  3. Otherwise → use heuristic scorer (TierHeuristic)

On LLM failure, automatically falls back to heuristic and records it.

Args: 

ctx - operation context.
s - the strategy to score.

Returns: 

float64 - the fitness score.
Tier - which tier produced this score.
error - non-nil only if all tiers fail.

func (*TieredScorer) Stats 

func (ts *TieredScorer) Stats() map[string]int64

Stats returns scoring statistics since creation or last ResetStats.

Returns: 

map[string]int64 - statistics including cache_hits, llm_calls, heuristic_calls,
                  fallbacks, total_scored.

type TieredScorerConfig 

type TieredScorerConfig struct {
	// Cache is the shared score cache (required).
	Cache *ScoreCache

	// Budget is the LLM call budget tracker (required).
	Budget *Budget

	// HeuristicScorer is the fast, always-available scoring function (required).
	HeuristicScorer genome.ScorerFunc

	// LLMScorer is the optional expensive LLM-based scoring function.
	// When nil, all strategies are scored by the heuristic tier after cache miss.
	LLMScorer genome.ScorerFunc
}

TieredScorerConfig holds configuration for creating a tiered scorer.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.