evolution

package
v1.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 21, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package evolution — Gap 4: Evolution Control Protocol (ECP).

ECP is the protocol for cross-instance knowledge sharing. It defines the message types, serialization, and aggregation primitives that enable multiple agent instances to share learned skills and patterns. Without ECP, evolution is siloed to a single device. With ECP, the agent's knowledge compounds across every user and every session.

Protocol layers: 

ECP/1.0 — Core types and serialization (this file)
ECP/1.1 — Federation and trust model
ECP/1.2 — Peer discovery and synchronization

This file implements ECP/1.0: the type system and transport-neutral message definitions. The actual transport (HTTP, WebSocket, gRPC) is implemented by the caller.

Package evolution automatically extracts experiences from agent turns and generates skill candidates. This is OK's self-evolution engine.

Architecture: 

OnTurnComplete hook (called from agent.Run after each turn)
  → saveEpisodicMemory() — save turn summary as memory
  → detectPatterns()     — scan recent memory for repeatable patterns
  → saveSkillCandidate()  — when pattern repeats, propose skill

Efficiency: episodic memories are buffered in memory and flushed to disk only every 3 turns, reducing I/O by ~66%.

Package evolution — P2: Automated forgetting mechanism.

Low-value memories decay over time. The forgetter periodically:

  1. Ages episodic memories: deletes entries older than maxAge
  2. Ages candidates: marks old pending-review candidates as "stale"
  3. Limits total episodic count: keeps only the most recent N entries

Package evolution — kernel.Learn interface implementation.

This file makes evolution.Engine implement kernel.Learn, unifying the two previously separate evolution paths:

  1. learnAdapter (adapters.go) — called by LLM via the "learn" tool
  2. evolution.Engine — called by OnTurnComplete hook automatically

After this unification, learnAdapter delegates to evolution.Engine, so both manual (LLM) and automatic (hook) evolution share one engine.

Package evolution — Gap 3: Sandbox execution validation.

Replaces the weak validatePattern (episodic memory keyword check) with layered skill validation: 

Layer 0 — Structural validation (zero-LLM, always available):
  Checks skill body completeness: frontmatter, required sections.
  Stricter than skill.ValidateStructure — requires evolution-specific
  sections ("## When to Use", "## Steps", "## Verification").

Layer 1 — Safety pattern scanning:
  Delegates to skill.ValidateSafety — the canonical implementation
  shared with Store.parse() so hand-written skills are also checked.

Layer 2 — LLM safety review (Learn tool path, provider available).

This file is referenced by learn_interface.go as the P1 smart closure component of the self-evolution engine.

Package evolution — P2: Active forgetting via usage-based scoring.

Replaces the pure time-decay model with a weighted scoring system. Candidates are scored on two axes using file metadata and content heuristics: 

recency  — how recently was the candidate modified?  (weight 0.5)
activity — how many times has the pattern recurred?   (weight 0.5)

Items scoring below the retention threshold are candidates for forgetting.

Package evolution — Gap 1: Semantic pattern recognition.

Replaces the pure keyword-frequency approach (findPatterns) with a layered detection system: 

Layer 0 — Workflow signatures (zero-LLM, always available):
  Pre-defined workflow patterns matched against tool sequences + context
  hints. Catches "TDD workflow", "search-then-edit", "audit-before-deploy"
  etc. without any LLM call.

Layer 1 — Semantic analysis (LLM-driven, via Learn tool path):
  When a provider is available, the episodic corpus is analyzed by the
  LLM to extract abstract patterns ("user prefers incremental commits",
  "security review always precedes deployment"). This is the Learn
  interface's Extract path.

Layer 2 — Cross-turn narrative patterns (zero-LLM):
  Aggregates tool frequency + sequence + temporal proximity into a
  "workflow fingerprint" that captures more signal than isolated counts.

Package evolution — Gap 2: Intelligent skill synthesis.

Replaces the template-based generateSkillBody with a layered approach: 

Layer 0 (automatic hook path, zero-LLM):
  generateSkillBody produces structured playbooks from detected patterns.
  This is the existing function, now enhanced with workflow-aware
  descriptions.

Layer 1 (Learn tool path, LLM-driven):
  SynthesizeSkill sends patterns + episodic context to the LLM and
  receives a high-quality skill prompt that captures:
    - When to invoke the skill
    - Step-by-step workflow
    - Expected outcomes and verification steps
    - Common pitfalls and edge cases

The resulting skill is usable immediately — not just a skeleton, but a real instruction set the agent can follow.

Package evolution — ECP/1.1: Federation transport layer.

ECP/1.0 defined types and serialization. ECP/1.1 adds the ability for agent instances to discover each other, exchange knowledge updates, and merge learned skills — turning single-device evolution into a federated learning network.

This file implements:

  • ECPTransport: transport-agnostic interface for push/pull
  • HTTPPeer: HTTP-based peer client (the reference implementation)
  • Federator: orchestrates peer sync on a schedule

Package evolution — P1: Skill auto-validation and auto-merging.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildSafetyReviewPrompt 

func BuildSafetyReviewPrompt(req SafetyReviewRequest) string

BuildSafetyReviewPrompt creates the LLM prompt for safety review.

func DefaultAcceptPolicy 

func DefaultAcceptPolicy(p ECPSkillPacket) bool

DefaultAcceptPolicy accepts skills with confidence >= 0.6 and no dangerous patterns in the body.

func KnownTools 

func KnownTools() []string

KnownTools returns the canonical set of tool names a skill may reference. Delegates to skill.KnownTools.

func MarshalPatternsForLLM 

func MarshalPatternsForLLM(patterns []string) string

MarshalPatternsForLLM serializes patterns to a JSON string suitable for the LLM context. Used by the Learn tool path.

func ServeECP deprecated 

func ServeECP(mux *http.ServeMux, eng *Engine)

ServeECP registers ECP endpoints on an HTTP mux. The evolution engine provides the knowledge to serve; the skill store receives pushed skills.

Usage: 

mux := http.NewServeMux()
ecp.ServeECP(mux, engine, skillStore)

Deprecated: use ServeECPWithSecret to enable peer authentication.

func ServeECPWithSecret 

func ServeECPWithSecret(mux *http.ServeMux, eng *Engine, sharedSecret string)

ServeECPWithSecret registers ECP endpoints on an HTTP mux with peer authentication via HMAC shared secret. When sharedSecret is empty, HMAC verification is disabled (open federation).

func SynthesizeBody 

func SynthesizeBody(req SynthesisRequest) string

SynthesizeBody generates a skill body from patterns using the LLM. The provider argument is optional — when nil, falls back to enhanced template generation (Layer 0).

This is the Layer 1 path, called via the Learn tool when a provider is available.

func SynthesizePrompt 

func SynthesizePrompt(req SynthesisRequest) string

SynthesizePrompt returns the prompt to send to the LLM. The response should be the full skill body.

func ValidateSkill 

func ValidateSkill(body string, knownTools []string) error

ValidateSkill runs all validation layers against a skill.

func ValidateSkillReferences 

func ValidateSkillReferences(body string, knownTools []string) error

ValidateSkillReferences checks that the skill body only references known tool names. Delegates to skill.ValidateReferences.

func ValidateSkillSafety 

func ValidateSkillSafety(body string) error

ValidateSkillSafety scans the skill body for dangerous shell patterns. Delegates to skill.ValidateSafety.

func ValidateSkillStructure 

func ValidateSkillStructure(body string) error

ValidateSkillStructure checks a skill body for structural completeness. Stricter than the skill-package version: requires evolution-generated sections ("## When to Use", "## Steps", "## Verification").

Types

type AnalyzeRequest 

type AnalyzeRequest struct {
	Entries []string `json:"entries"` // recent episodic memory entries
	MaxN    int      `json:"maxN"`    // max insights to return
}

AnalyzeRequest is the input to LLM-driven semantic analysis.

type ECPKnowledgeUpdate 

type ECPKnowledgeUpdate struct {
	ID           string           `json:"id"`
	PeerInstance string           `json:"peerInstance"`
	GeneratedAt  time.Time        `json:"generatedAt"`
	Skills       []ECPSkillPacket `json:"skills"`
	Stats        ECPPeerStats     `json:"stats"`
	SequenceNum  int64            `json:"sequenceNum"` // monotonic update counter
}

ECPKnowledgeUpdate is an aggregated knowledge snapshot from a peer instance. It carries multiple skill packets and usage statistics.

func UnmarshalKnowledgeUpdate 

func UnmarshalKnowledgeUpdate(data []byte) (ECPKnowledgeUpdate, error)

UnmarshalKnowledgeUpdate deserializes a knowledge update.

func (ECPKnowledgeUpdate) Marshal 

func (u ECPKnowledgeUpdate) Marshal() ([]byte, error)

MarshalKnowledgeUpdate serializes a knowledge update.

type ECPManifest 

type ECPManifest struct {
	Instance    string    `json:"instance"`
	Version     string    `json:"version"`
	GeneratedAt time.Time `json:"generatedAt"`
	SequenceNum int64     `json:"sequenceNum"`
	SkillCount  int       `json:"skillCount"`
	SkillNames  []string  `json:"skillNames"` // for lightweight discovery
	TopTags     []string  `json:"topTags"`    // for topic-based filtering
	LastUpdate  time.Time `json:"lastUpdate"`
}

ECPManifest describes what knowledge a peer has available for sharing. Used during peer discovery to decide what to sync.

func (ECPManifest) Marshal 

func (m ECPManifest) Marshal() ([]byte, error)

MarshalManifest serializes a manifest.

type ECPMergeResult 

type ECPMergeResult struct {
	NewSkills      int      `json:"newSkills"`      // skills installed for the first time
	UpdatedSkills  int      `json:"updatedSkills"`  // skills that were refreshed
	RejectedSkills int      `json:"rejectedSkills"` // skills rejected (unsafe, duplicate, etc.)
	Conflicts      []string `json:"conflicts"`      // skills that conflicted with local versions
}

ECPMergeResult describes the outcome of merging knowledge from a peer.

func MergeKnowledge 

func MergeKnowledge(
	update ECPKnowledgeUpdate,
	existingNames map[string]bool,
	acceptFn func(ECPSkillPacket) bool,
	installFn func(ECPSkillPacket) error,
) ECPMergeResult

MergeKnowledge aggregates skills from a peer update into a local skill set. Returns the merge result describing what was installed/updated/rejected.

acceptFn is called for each skill before installation. If it returns false, the skill is rejected. This allows the caller to enforce acceptance policies (e.g., only accept skills with confidence >= 0.7, only from trusted peers).

type ECPPeerStats 

type ECPPeerStats struct {
	TotalSkills      int      `json:"totalSkills"`
	TotalExtractions int64    `json:"totalExtractions"`
	AvgConfidence    float64  `json:"avgConfidence"`
	UptimeHours      float64  `json:"uptimeHours"`
	TopWorkflows     []string `json:"topWorkflows"` // most frequent workflow names
}

ECPPeerStats carries aggregate statistics about a peer's evolution.

type ECPSkillPacket 

type ECPSkillPacket struct {
	// Header — routing and integrity.
	ID        string    `json:"id"`        // unique packet ID (SHA-256 of body)
	Version   string    `json:"version"`   // ECP protocol version ("1.0")
	CreatedAt time.Time `json:"createdAt"` // when this packet was created

	// Origin — where the skill came from.
	OriginInstance string `json:"originInstance"` // agent instance identifier
	OriginUserHash string `json:"originUserHash"` // SHA-256 of user ID (privacy-preserving)
	OriginOS       string `json:"originOS"`       // "windows/amd64", "linux/arm64"
	OriginVersion  string `json:"originVersion"`  // agent version that generated this

	// Payload — the skill itself.
	SkillName        string   `json:"skillName"`
	SkillDescription string   `json:"skillDescription"`
	SkillBody        string   `json:"skillBody"`
	Patterns         []string `json:"patterns"`   // detected patterns that triggered generation
	Confidence       float64  `json:"confidence"` // 0.0–1.0, how confident the generator was

	// Integrity.
	IntegrityHash string `json:"integrityHash"` // SHA-256 of the skill body

	// Privacy flags.
	Shareable  bool     `json:"shareable"`  // can be federated to other instances
	Tags       []string `json:"tags"`       // categorization tags
	Deprecated bool     `json:"deprecated"` // true when superseded by a newer version
}

ECPSkillPacket is a skill packaged for transmission between agent instances. It carries the skill body, metadata, and integrity proof.

func NewECPSkillPacket 

func NewECPSkillPacket(
	instance, userID, osArch, agentVersion string,
	name, description, body string,
	patterns []string, confidence float64,
) ECPSkillPacket

NewECPSkillPacket creates a new skill packet ready for transmission. The originUserHash is a SHA-256 of the user ID — the raw ID never leaves the local instance.

func UnmarshalECPPacket 

func UnmarshalECPPacket(data []byte) (ECPSkillPacket, error)

UnmarshalECPPacket deserializes a skill packet from JSON.

func (ECPSkillPacket) Marshal 

func (p ECPSkillPacket) Marshal() ([]byte, error)

Marshal serializes a skill packet to JSON.

func (ECPSkillPacket) Verify 

func (p ECPSkillPacket) Verify() error

Verify checks the packet's integrity hash against its skill body.

type ECPTransport 

type ECPTransport interface {
	// Push sends a knowledge update to a peer. The peer may accept, reject,
	// or merge the update at its discretion. Returns the peer's response
	// describing what was done.
	Push(ctx context.Context, peerURL string, update ECPKnowledgeUpdate) (ECPMergeResult, error)

	// Pull fetches a peer's knowledge manifest and (optionally) the full
	// update. When manifestOnly is true, only the lightweight manifest is
	// returned — useful for discovery. Otherwise, the full update is fetched.
	Pull(ctx context.Context, peerURL string) (ECPKnowledgeUpdate, error)

	// PullManifest fetches only the lightweight manifest from a peer.
	// Used during discovery to decide whether a full pull is worthwhile.
	PullManifest(ctx context.Context, peerURL string) (ECPManifest, error)
}

ECPTransport abstracts how knowledge updates move between agent instances. Implementations can use HTTP, gRPC, WebSocket, or even file-based sync.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the self-evolution engine. Zero value is safe (no-op).

func New 

func New(mem *memory.Set, skStore *skill.Store, dir string) *Engine

New creates an evolution engine.

func (*Engine) Extract 

func (e *Engine) Extract(_ context.Context, task kernel.TaskRecord) ([]kernel.Pattern, error)

Extract analyzes a completed task and returns patterns. Satisfies kernel.Learn.Extract.

func (*Engine) Generate 

func (e *Engine) Generate(_ context.Context, patterns []kernel.Pattern) (kernel.Skill, error)

Generate creates a candidate skill from successful patterns using the enhanced synthesis pipeline (Gap 2). When pattern context is available, it produces a rich playbook; otherwise falls back to template generation. Satisfies kernel.Learn.Generate.

func (*Engine) OnTurnComplete 

func (e *Engine) OnTurnComplete(ctx context.Context, input, output string)

OnTurnComplete is called by the agent after each completed turn. Episodic data is buffered in memory and flushed to disk every 3 turns.

func (*Engine) Publish 

func (e *Engine) Publish(_ context.Context, skill kernel.Skill) error

Publish makes a validated skill available to the agent's skill store. Satisfies kernel.Learn.Publish.

func (*Engine) Stats 

func (e *Engine) Stats(_ context.Context) kernel.LearnStats

Stats returns learning metrics. Satisfies kernel.Learn.Stats.

func (*Engine) Validate 

func (e *Engine) Validate(_ context.Context, skill kernel.Skill) error

Validate runs the full sandbox-validation pipeline on a skill: 

Layer 0 — structural completeness
Layer 1 — safety pattern scan + tool reference check
Layer 2 — duplicate name check

Satisfies kernel.Learn.Validate.

type Federator 

type Federator struct {
	// contains filtered or unexported fields
}

Federator orchestrates knowledge exchange with a set of peer instances. It periodically pulls from configured peers, merges their knowledge, and makes the local skill set available for others to pull.

The zero value is safe — Start is a no-op when no peers are configured.

func NewFederator 

func NewFederator(cfg FederatorConfig) *Federator

NewFederator creates a federator. Pass nil transport to use the default HTTP transport with 30s timeout.

func (*Federator) Start 

func (f *Federator) Start()

Start begins periodic federation. Runs the first sync immediately, then every interval. Call Stop to shut down.

func (*Federator) Stop 

func (f *Federator) Stop()

Stop shuts down the federation loop. Safe to call multiple times.

func (*Federator) SyncNow 

func (f *Federator) SyncNow(ctx context.Context) ECPMergeResult

SyncNow triggers an immediate sync with all configured peers. Returns the aggregate merge result.

type FederatorConfig 

type FederatorConfig struct {
	Transport      ECPTransport
	Peers          []string
	Interval       time.Duration // sync interval; default 1h
	InstanceID     string
	SkillStore     SkillInstaller
	ExistingSkills func() []string
}

FederatorConfig configures the federation loop.

type HTTPPeer 

type HTTPPeer struct {
	SharedSecret string
	// contains filtered or unexported fields
}

HTTPPeer implements ECPTransport over plain HTTP. It is the reference transport for ECP/1.1.

Endpoints (on the peer server): 

GET  /ecp/manifest           → ECPManifest
GET  /ecp/skills             → ECPKnowledgeUpdate
POST /ecp/skills             → accept ECPKnowledgeUpdate, return ECPMergeResult

When SharedSecret is non-empty, requests include an X-ECP-HMAC header computed as HMAC-SHA256 of the request body (or URL path for GET requests) so the peer can authenticate the caller.

func NewHTTPPeer 

func NewHTTPPeer(timeout time.Duration) *HTTPPeer

NewHTTPPeer creates an HTTP transport with the given timeout.

func (*HTTPPeer) Pull 

func (h *HTTPPeer) Pull(ctx context.Context, peerURL string) (ECPKnowledgeUpdate, error)

func (*HTTPPeer) PullManifest 

func (h *HTTPPeer) PullManifest(ctx context.Context, peerURL string) (ECPManifest, error)

func (*HTTPPeer) Push 

func (h *HTTPPeer) Push(ctx context.Context, peerURL string, update ECPKnowledgeUpdate) (ECPMergeResult, error)

type SafetyReviewRequest 

type SafetyReviewRequest struct {
	SkillName string `json:"skillName"`
	SkillBody string `json:"skillBody"`
}

SafetyReviewRequest is the input for LLM-based skill safety review.

type SafetyReviewResult 

type SafetyReviewResult struct {
	Safe    bool     `json:"safe"`
	Issues  []string `json:"issues,omitempty"`
	Summary string   `json:"summary"`
}

SafetyReviewResult is the LLM's verdict on skill safety.

type SemanticInsight 

type SemanticInsight struct {
	Pattern     string  `json:"pattern"`     // e.g. "incremental-commit"
	Description string  `json:"description"` // e.g. "User commits after every file edit"
	Confidence  float64 `json:"confidence"`  // 0.0–1.0
	Evidence    string  `json:"evidence"`    // supporting episodic entries
}

SemanticInsight is a high-level pattern extracted by LLM analysis of episodic memory. Unlike raw tool counts, these carry human-readable descriptions of abstract workflows.

type SkillInfo 

type SkillInfo struct {
	Name        string
	Description string
}

SkillInfo is a minimal skill descriptor for federation.

type SkillInstaller 

type SkillInstaller interface {
	Install(name, scope, body string) error
	List() []SkillInfo
}

SkillInstaller abstracts skill installation so the federator doesn't depend on the concrete skill.Store.

type SynthesisRequest 

type SynthesisRequest struct {
	Name      string   `json:"name"`      // proposed skill name
	Patterns  []string `json:"patterns"`  // detected patterns
	Context   []string `json:"context"`   // recent episodic entries for context
	Workflows []string `json:"workflows"` // detected workflow names
	ToolNames []string `json:"toolNames"` // tools involved
}

SynthesisRequest is the input to LLM-driven skill synthesis.

type SynthesisResult 

type SynthesisResult struct {
	Name        string `json:"name"`
	Description string `json:"description"`
	Body        string `json:"body"` // full skill markdown body
}

SynthesisResult is the output of LLM-driven skill synthesis.

func DecodeSynthesisResult 

func DecodeSynthesisResult(response string) SynthesisResult

DecodeSynthesisResult parses an LLM response into a SynthesisResult. The response is expected to be a markdown skill body; we extract the frontmatter to populate Name and Description.

type WorkflowSignature 

type WorkflowSignature struct {
	Name        string   // e.g. "tdd", "search-then-edit"
	Description string   // human-readable explanation
	Tools       []string // ordered tools that must appear in sequence
	MinHits     int      // minimum occurrences to trigger (default 2)
}

WorkflowSignature describes a named, repeatable workflow pattern. Signatures are matched against tool call sequences across episodic entries. This is Layer 0 — zero LLM cost, always active.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.