eval

package
v0.4.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: MIT Imports: 41 Imported by: 0

Documentation

Overview

Package eval provides deterministic contracts and orchestration for Atlas live-agent evaluations.

Index

Constants

View Source 
const (
	// GuidanceProfileNone omits Project-native framework instructions for the control treatment.
	GuidanceProfileNone = "none"
	// GuidanceProfileAgents installs only the canonical Project AGENTS.md treatment.
	GuidanceProfileAgents = "agents"
	// GuidanceProfileAgentsSkills installs canonical Project instructions and recommended native skills.
	GuidanceProfileAgentsSkills = "agents-skills"
	// GuidanceProfileAtlas installs the complete recommended Atlas surface: instructions, skills, and MCP.
	GuidanceProfileAtlas = "atlas"
)
View Source 
const (
	// EventFieldCommandID correlates one trusted command start with its completion.
	EventFieldCommandID = "command_id"
	// EventFieldExecutableDigest identifies the resolved executable observed by the supervisor.
	EventFieldExecutableDigest = "executable_digest"
	// EventFieldArguments carries a JSON array of arguments excluding the executable.
	EventFieldArguments = "arguments"
	// EventFieldExitCode records the decimal process exit code for a completed command.
	EventFieldExitCode = "exit_code"
	// EventFieldPath records one Project-relative observed file path.
	EventFieldPath = "path"
)

Variables

This section is empty.

Functions

func EvaluateWorkflow 

func EvaluateWorkflow(workflow WorkflowExpectation, events []Event, forjDigest string, capabilities []Capability) (EndpointResult, []EndpointResult)

EvaluateWorkflow checks promoted workflow actions against supervisor-observed command events.

func IsResourceExhaustion 

func IsResourceExhaustion(err error) bool

IsResourceExhaustion reports whether an operation failed because its shared storage resource is full.

func PromotedEvaluationIDs 

func PromotedEvaluationIDs(suite string) ([]string, error)

PromotedEvaluationIDs returns promoted evaluation IDs in stable order, optionally limited to one suite.

func PromotedEvaluationIDsMatching 

func PromotedEvaluationIDsMatching(filter EvaluationFilter) ([]string, error)

PromotedEvaluationIDsMatching returns promoted evaluation IDs matching every non-empty dimension.

func SupportedGuidanceProfile 

func SupportedGuidanceProfile(profile string) bool

SupportedGuidanceProfile reports whether profile names one closed evaluation treatment.

Types

type AddHTTPControllerVerifier 

type AddHTTPControllerVerifier struct {
	// contains filtered or unexported fields
}

AddHTTPControllerVerifier verifies invoice HTTP behavior independently from GoForj's golden scenario steps.

func NewAddHTTPControllerVerifier 

func NewAddHTTPControllerVerifier(runner CommandRunner) *AddHTTPControllerVerifier

NewAddHTTPControllerVerifier creates the promoted verifier with a trusted isolated command runner.

func (*AddHTTPControllerVerifier) Capabilities 

func (*AddHTTPControllerVerifier) Capabilities() []Capability

Capabilities returns no agent-observation requirements because outcome checks inspect a sealed tree through supervisor-owned boundaries.

func (*AddHTTPControllerVerifier) ID 

func (*AddHTTPControllerVerifier) ID() string

ID returns the promoted verifier contract identity.

func (*AddHTTPControllerVerifier) Verify 

func (verifier *AddHTTPControllerVerifier) Verify(ctx context.Context, input VerificationInput) (VerificationResult, error)

Verify checks structure, compilation, registration, and route visibility without comparing candidate source to the golden recipe.

type AgentFailure 

type AgentFailure struct {
	Outcome AgentOutcome
	Err     error
}

AgentFailure classifies a failed adapter operation without conflating provider availability with adapter defects.

func (*AgentFailure) Error 

func (failure *AgentFailure) Error() string

Error returns the wrapped operation failure.

func (*AgentFailure) Unwrap 

func (failure *AgentFailure) Unwrap() error

Unwrap exposes the original operation failure.

type AgentOutcome 

type AgentOutcome string

AgentOutcome classifies what happened to the provider-side attempt. 

const (
	// AgentNotStarted indicates preflight ended before a provider session began.
	AgentNotStarted AgentOutcome = "not_started"
	// AgentCompleted indicates the agent reached a terminal response or process exit.
	AgentCompleted AgentOutcome = "completed"
	// AgentAbstained indicates an accepted safe clarification ended the attempt.
	AgentAbstained AgentOutcome = "abstained"
	// AgentProviderError indicates the provider failed independently from the adapter.
	AgentProviderError AgentOutcome = "provider_error"
	// AgentAdapterError indicates the adapter failed independently from the provider.
	AgentAdapterError AgentOutcome = "adapter_error"
	// AgentTimeout indicates the attempt exceeded its wall-time budget.
	AgentTimeout AgentOutcome = "timeout"
	// AgentCancelled indicates an operator cancelled the attempt.
	AgentCancelled AgentOutcome = "cancelled"
)

type AgentPreparation 

type AgentPreparation interface {
	Agent() PreparedAgent
	Close(context.Context) error
}

AgentPreparation owns private adapter resources acquired before a session starts.

type AgentProperties 

type AgentProperties struct {
	Properties []Capability
}

AgentProperties declares provider-neutral safety properties enforced by an adapter.

type AgentResult 

type AgentResult struct {
	Outcome   AgentOutcome
	Events    []Event
	Message   string
	Telemetry *ProviderTelemetry
}

AgentResult records provider completion independently from evaluator validity.

type AgentSessionIdentity 

type AgentSessionIdentity struct {
	Version         string
	Model           string
	ModelProvider   string
	AuthorityDigest string
	SessionDigest   string
}

AgentSessionIdentity records effective provider identity established only after a fresh session starts.

type AgentTurn 

type AgentTurn struct {
	Prompt string
	Limits Limits
}

AgentTurn is one supervisor-selected natural-language interaction.

type AgentTurnResult 

type AgentTurnResult struct {
	Accepted bool
	Events   []Event
}

AgentTurnResult records prompt acceptance and any events emitted before it returned.

type ArtifactFile 

type ArtifactFile struct {
	Path           string `json:"path"`
	Digest         string `json:"digest"`
	Size           int64  `json:"size"`
	Classification string `json:"classification"`
}

ArtifactFile records one retained file's exact identity and classification.

type ArtifactManifest 

type ArtifactManifest struct {
	SchemaVersion int            `json:"schema_version"`
	AttemptID     string         `json:"attempt_id"`
	PlanDigest    string         `json:"plan_digest"`
	BaselineTree  string         `json:"baseline_tree"`
	FinalTree     string         `json:"final_tree"`
	Files         []ArtifactFile `json:"files"`
	Signature     string         `json:"signature"`
}

ArtifactManifest integrity-checks the complete retained evidence set against later accidental changes.

func ReadVerifiedAttemptSummary 

func ReadVerifiedAttemptSummary(directory string, key []byte) (string, ArtifactManifest, error)

ReadVerifiedAttemptSummary authenticates the complete attempt before returning its inert human report.

func VerifyArtifactManifest 

func VerifyArtifactManifest(directory string, key []byte) (ArtifactManifest, error)

VerifyArtifactManifest authenticates metadata and every retained file without executing artifact content.

type ArtifactStore 

type ArtifactStore struct {
	// contains filtered or unexported fields
}

ArtifactStore creates private supervisor-owned attempt directories and manifests with post-run tamper evidence.

func NewArtifactStore 

func NewArtifactStore(root string, key []byte, redactor Redactor) (*ArtifactStore, error)

NewArtifactStore requires an integrity key rather than writing unsigned evidence.

func (*ArtifactStore) Begin 

func (store *ArtifactStore) Begin(attemptID string) (*AttemptArtifacts, error)

Begin creates one private attempt directory outside any agent-provided path.

type AttemptArtifacts 

type AttemptArtifacts struct {
	// contains filtered or unexported fields
}

AttemptArtifacts owns append-only evidence for one attempt until finalization.

func (*AttemptArtifacts) AppendEvent 

func (artifacts *AttemptArtifacts) AppendEvent(event Event) error

AppendEvent redacts and persists one event before accepting the next sequence item.

func (*AttemptArtifacts) Finalize 

func (artifacts *AttemptArtifacts) Finalize(planDigest, baselineTree, finalTree string) (ArtifactManifest, error)

Finalize closes streaming evidence and writes the authenticated manifest last.

func (*AttemptArtifacts) WriteJSON 

func (artifacts *AttemptArtifacts) WriteJSON(name string, value any) error

WriteJSON writes one typed artifact after recursively redacting its serialized representation.

func (*AttemptArtifacts) WriteText 

func (artifacts *AttemptArtifacts) WriteText(name, content string) error

WriteText writes one bounded inert text artifact selected by the supervisor.

type AttemptEnvironment 

type AttemptEnvironment struct {
	Intent                RunIntent       `json:"intent"`
	Backend               string          `json:"backend"`
	RequestedShellNetwork string          `json:"requested_shell_network"`
	EnvironmentKeys       []string        `json:"environment_keys"`
	AgentProperties       []Capability    `json:"agent_properties,omitempty"`
	BackendCapabilities   []Capability    `json:"backend_capabilities,omitempty"`
	UnavailableEvidence   []Capability    `json:"unavailable_evidence,omitempty"`
	Runtime               RuntimeIdentity `json:"runtime"`
}

AttemptEnvironment records names and negotiated capabilities without persisting environment values.

type AttemptRequest 

type AttemptRequest struct {
	AttemptID       string
	LogicalTrialID  string
	Intent          RunIntent
	Definition      EvaluationDefinition
	Preparation     PreparationRequest
	GuidanceProfile string
	Runtime         RuntimeIdentity
}

AttemptRequest contains invocation policy that does not belong in evaluation YAML.

type AttemptResult 

type AttemptResult struct {
	AttemptID               string                 `json:"attempt_id"`
	LogicalTrialID          string                 `json:"logical_trial_id"`
	EvaluationID            string                 `json:"evaluation_id"`
	PromptDigest            string                 `json:"prompt_digest"`
	GuidanceProfile         string                 `json:"guidance_profile"`
	GuidanceDigest          string                 `json:"guidance_digest,omitempty"`
	GuidanceFiles           []GuidanceFileIdentity `json:"guidance_files,omitempty"`
	ScenarioID              string                 `json:"scenario_id"`
	ScenarioSchema          int                    `json:"scenario_schema,omitempty"`
	PlanDigest              string                 `json:"plan_digest,omitempty"`
	ScenarioPlanDigest      string                 `json:"scenario_plan_digest,omitempty"`
	CatalogDigest           string                 `json:"catalog_digest,omitempty"`
	DependencyDigests       map[string]string      `json:"dependency_digests,omitempty"`
	ProjectConfigDigest     string                 `json:"project_config_digest,omitempty"`
	EnvironmentDigest       string                 `json:"environment_digest,omitempty"`
	PreparedTree            string                 `json:"prepared_tree,omitempty"`
	BaselineTree            string                 `json:"baseline_tree,omitempty"`
	FinalTree               string                 `json:"final_tree,omitempty"`
	ForjExecutable          string                 `json:"forj_executable,omitempty"`
	ForjDigest              string                 `json:"forj_digest,omitempty"`
	Backend                 string                 `json:"backend,omitempty"`
	Agent                   string                 `json:"agent,omitempty"`
	AgentExecutable         string                 `json:"agent_executable,omitempty"`
	AgentDigest             string                 `json:"agent_digest,omitempty"`
	ProviderAuthorityDigest string                 `json:"provider_authority_digest,omitempty"`
	ProviderSessionDigest   string                 `json:"provider_session_digest,omitempty"`
	ProviderTelemetry       *ProviderTelemetry     `json:"provider_telemetry,omitempty"`
	AgentVersion            string                 `json:"agent_version,omitempty"`
	Model                   string                 `json:"model,omitempty"`
	ModelProvider           string                 `json:"model_provider,omitempty"`
	Runtime                 RuntimeIdentity        `json:"runtime"`
	AgentOutcome            AgentOutcome           `json:"agent_outcome"`
	EvaluationStatus        EvaluationStatus       `json:"evaluation_status"`
	Milestones              []Milestone            `json:"milestones"`
	Verification            *VerificationResult    `json:"verification,omitempty"`
	SecondaryFailures       []SecondaryFailure     `json:"secondary_failures,omitempty"`
	UnavailableEvidence     []Capability           `json:"unavailable_evidence,omitempty"`
	StartedAt               time.Time              `json:"started_at"`
	FinishedAt              time.Time              `json:"finished_at"`
}

AttemptResult is the complete lifecycle result for one stochastic attempt.

type AttemptScorecard 

type AttemptScorecard struct {
	EvaluationStatus    EvaluationStatus `json:"evaluation_status"`
	AgentOutcome        AgentOutcome     `json:"agent_outcome"`
	FrameworkOutcome    *EndpointResult  `json:"framework_outcome,omitempty"`
	WorkflowConformance *EndpointResult  `json:"workflow_conformance,omitempty"`
	UnavailableEvidence []Capability     `json:"unavailable_evidence,omitempty"`
}

AttemptScorecard is the compact machine-readable outcome layer retained beside full run metadata.

type BackendEnvironment 

type BackendEnvironment interface {
	Environment() RunEnvironment
	Baseline(context.Context) (BaselineSnapshot, error)
	ObservedEvents(context.Context) ([]Event, error)
	Seal(context.Context) (SealedProject, error)
	Close(context.Context) error
}

BackendEnvironment owns the isolation resources that enclose one agent attempt.

type BackendRequest 

type BackendRequest struct {
	Project      PreparedProject
	ShellNetwork string
	Environment  []string
	CommandLimit int
}

BackendRequest binds one prepared Project to explicit execution policy.

type BaselineSnapshot 

type BaselineSnapshot struct {
	TreeDigest string
	Complete   bool
}

BaselineSnapshot is the backend/supervisor-owned baseline captured after treatment setup and before the session starts.

type Capability 

type Capability string

Capability identifies one observation or isolation property an evaluation component can prove. 

const (
	// CapabilityFileReads proves agent file reads through a trusted observation boundary.
	CapabilityFileReads Capability = "file_reads"
	// CapabilityFileWrites proves agent file mutations through a trusted observation boundary.
	CapabilityFileWrites Capability = "file_writes"
	// CapabilityCommands proves process execution with trusted executable identity and arguments.
	CapabilityCommands Capability = "commands"
	// CapabilityMCPToolCalls proves MCP calls through a trusted supervisor-owned interposer.
	CapabilityMCPToolCalls Capability = "mcp_tool_calls"
	// CapabilityProcessCleanup proves complete descendant-job termination.
	CapabilityProcessCleanup Capability = "process_cleanup"
	// CapabilityCredentialIsolation proves candidate processes cannot reach reusable provider authority.
	CapabilityCredentialIsolation Capability = "credential_isolation"
	// CapabilityHostFilesystemIsolation proves candidate processes cannot read or mutate undeclared host paths.
	CapabilityHostFilesystemIsolation Capability = "host_filesystem_isolation"
	// CapabilityNetworkEnforcement proves the backend enforces the requested shell network policy.
	CapabilityNetworkEnforcement Capability = "network_enforcement"
	// CapabilityVerifierIsolation proves candidate execution cannot mutate verifier code, state, or later phases.
	CapabilityVerifierIsolation Capability = "verifier_isolation"
	// CapabilityArtifactIsolation proves candidate processes cannot read signing authority or mutate retained evidence.
	CapabilityArtifactIsolation Capability = "artifact_isolation"
	// CapabilityFinalResponseCapture proves the adapter returns the provider's exact terminal response to the verifier.
	CapabilityFinalResponseCapture Capability = "final_response_capture"
)

type CapabilityCoverage 

type CapabilityCoverage struct {
	ID          string       `json:"id"`
	Area        string       `json:"area"`
	Summary     string       `json:"summary"`
	Tier        CoverageTier `json:"tier"`
	Evaluations []string     `json:"evaluations"`
}

CapabilityCoverage maps one durable framework capability to the evaluations that currently measure it.

func (CapabilityCoverage) Covered 

func (coverage CapabilityCoverage) Covered() bool

Covered reports whether at least one promoted evaluation currently measures the capability.

type CodexAgent 

type CodexAgent struct {
	// contains filtered or unexported fields
}

CodexAgent starts fresh Codex sessions and keeps preparation state private from generic runner contracts.

func NewCodexAgent 

func NewCodexAgent(options CodexOptions) (*CodexAgent, error)

NewCodexAgent creates a diagnostic Codex agent without claiming authoritative observation capabilities.

func (*CodexAgent) Name 

func (*CodexAgent) Name() string

Name returns the adapter identity recorded by evaluation reports.

func (*CodexAgent) Prepare 

func (adapter *CodexAgent) Prepare(_ context.Context, environment RunEnvironment, guidance Guidance) (AgentPreparation, error)

Prepare fingerprints Codex and copies only the credential needed by the private app-server home.

func (*CodexAgent) Properties 

func (*CodexAgent) Properties(context.Context) (AgentProperties, error)

Properties reports exact final-response capture while withholding isolation guarantees from the unconfined diagnostic adapter.

func (*CodexAgent) Start 

func (adapter *CodexAgent) Start(ctx context.Context, agent PreparedAgent) (EvaluationSession, error)

Start requires the app-server boundary to recheck the prepared launcher path and digest immediately before process startup.

This pre-exec launcher digest check does not make the launcher, its interpreter, or loaded modules immutable, and it leaves a residual check-to-exec race.

type CodexCredential 

type CodexCredential struct {
	// contains filtered or unexported fields
}

CodexCredential is an immutable credential snapshot shared by every treatment in one logical trial.

func LoadCodexCredential 

func LoadCodexCredential(source string) (CodexCredential, error)

LoadCodexCredential freezes one auth document before any treatment can begin.

func NewCodexCredential 

func NewCodexCredential(body []byte) (CodexCredential, error)

NewCodexCredential freezes credential bytes supplied by a host-owned read boundary.

func (CodexCredential) Redactor 

func (credential CodexCredential) Redactor(redactor Redactor) Redactor

Redactor returns a copy that covers the exact frozen authority used by Codex sessions.

type CodexOptions 

type CodexOptions struct {
	Executable       string
	Arguments        []string
	Model            string
	ModelProvider    string
	Credential       CodexCredential
	CredentialSource string
	Environment      []string
}

CodexOptions pins the Codex executable, model, provider state, and process environment used by diagnostic sessions.

type CommandRunner 

type CommandRunner interface {
	// Open creates one disposable verifier session from sealed candidate evidence.
	Open(context.Context, VerifierProject) (CommandSession, error)
}

CommandRunner opens verifier-owned black-box checks in an isolated clone of the sealed candidate tree.

type CommandSession 

type CommandSession interface {
	// WriteFile installs supervisor-owned verifier input inside the disposable Project.
	WriteFile(string, []byte) error
	// Run executes one allowlisted verifier command.
	Run(context.Context, []string) (string, error)
	// Close destroys the disposable Project and any supervisor-owned input.
	Close(context.Context) error
}

CommandSession executes a bounded sequence against one disposable candidate clone.

type CoverageCatalog 

type CoverageCatalog struct {
	Capabilities []CapabilityCoverage `json:"capabilities"`
}

CoverageCatalog is the validated framework capability inventory used to expose measured and planned coverage.

func LoadCoverageCatalog 

func LoadCoverageCatalog() (CoverageCatalog, error)

LoadCoverageCatalog returns a validated copy of the promoted framework capability inventory.

func (CoverageCatalog) EvaluationIDs 

func (catalog CoverageCatalog) EvaluationIDs(tier CoverageTier) ([]string, error)

EvaluationIDs returns the promoted evaluations needed to exercise capabilities through the requested tier.

type CoverageTier 

type CoverageTier string

CoverageTier identifies how frequently a capability should participate in live evaluation. 

const (
	// CoverageTierSmoke identifies release-critical behavior suitable for every evaluation smoke run.
	CoverageTierSmoke CoverageTier = "smoke"
	// CoverageTierCore identifies representative framework behavior for the complete core benchmark.
	CoverageTierCore CoverageTier = "core"
	// CoverageTierExtended identifies expensive or specialized behavior for scheduled qualification.
	CoverageTierExtended CoverageTier = "extended"
)

type EndpointResult 

type EndpointResult struct {
	ID      string          `json:"id"`
	Status  EndpointStatus  `json:"status"`
	Details string          `json:"details,omitempty"`
	Kind    RequirementKind `json:"kind,omitempty"`
}

EndpointResult describes one framework, conformance, or contract endpoint.

type EndpointStatus 

type EndpointStatus string

EndpointStatus records one independently meaningful verification outcome. 

const (
	// EndpointPassed indicates that trusted evidence satisfied the endpoint.
	EndpointPassed EndpointStatus = "passed"
	// EndpointFailed indicates that complete trusted evidence violated the endpoint.
	EndpointFailed EndpointStatus = "failed"
	// EndpointIneligible indicates that required trusted evidence was unavailable.
	EndpointIneligible EndpointStatus = "ineligible"
)

type EvaluationAgent 

type EvaluationAgent interface {
	Name() string
	Properties(context.Context) (AgentProperties, error)
	Prepare(context.Context, RunEnvironment, Guidance) (AgentPreparation, error)
	Start(context.Context, PreparedAgent) (EvaluationSession, error)
}

EvaluationAgent prepares and starts fresh non-interactive evaluation sessions.

type EvaluationDefinition 

type EvaluationDefinition struct {
	SchemaVersion   int
	ID              string
	Summary         string
	Suite           string
	TaskKind        EvaluationTaskKind
	ProjectScenario string
	WorkflowID      string
	VerifierID      string
	Limits          Limits
	Prompt          string
	PromptDigest    string
}

EvaluationDefinition is one resolved manifest and its adjacent natural-language prompt.

func LoadDefinition 

func LoadDefinition(directory string) (EvaluationDefinition, error)

LoadDefinition reads one strict evaluation manifest and its adjacent prompt.

func LoadPromotedDefinition 

func LoadPromotedDefinition(id string) (EvaluationDefinition, error)

LoadPromotedDefinition reads one reviewed evaluation bundled with Atlas.

type EvaluationFilter 

type EvaluationFilter struct {
	Suite    string
	TaskKind EvaluationTaskKind
}

EvaluationFilter selects promoted evaluations by stable measurement dimensions.

type EvaluationSession 

type EvaluationSession interface {
	Identity() AgentSessionIdentity
	Turn(context.Context, AgentTurn) (AgentTurnResult, error)
	Wait(context.Context) (AgentResult, error)
	Close(context.Context) error
}

EvaluationSession owns one fresh provider session and its complete descendant job.

type EvaluationStatus 

type EvaluationStatus string

EvaluationStatus classifies whether trusted verification produced a valid result. 

const (
	// EvaluationValid indicates complete evidence produced ordinary endpoints.
	EvaluationValid EvaluationStatus = "valid"
	// EvaluationValidAbstention indicates the scenario accepted a safe abstention.
	EvaluationValidAbstention EvaluationStatus = "valid_abstention"
	// EvaluationNotEvaluated indicates no evidence-valid logical trial began.
	EvaluationNotEvaluated EvaluationStatus = "not_evaluated"
	// EvaluationIneligible indicates a required capability was unavailable at preflight.
	EvaluationIneligible EvaluationStatus = "ineligible"
	// EvaluationDiagnostic indicates diagnostic artifacts were collected without the evidence required for a valid evaluation.
	EvaluationDiagnostic EvaluationStatus = "diagnostic"
	// EvaluationFixtureError indicates Project preparation or starting-state verification failed.
	EvaluationFixtureError EvaluationStatus = "fixture_error"
	// EvaluationEvaluatorError indicates capture, verification, or cleanup became unreliable.
	EvaluationEvaluatorError EvaluationStatus = "evaluator_error"
)

type EvaluationTaskKind 

type EvaluationTaskKind string

EvaluationTaskKind separates workflow discovery from broader implementation and repair measurements. 

const (
	// TaskScaffold evaluates whether an agent discovers and uses a framework-owned creation workflow.
	TaskScaffold EvaluationTaskKind = "scaffold"
	// TaskFeature evaluates a complete application behavior across one or more framework surfaces.
	TaskFeature EvaluationTaskKind = "feature"
	// TaskRepair evaluates diagnosis and correction of an existing broken integration.
	TaskRepair EvaluationTaskKind = "repair"
	// TaskAbstention evaluates safe behavior when the available Project evidence is insufficient.
	TaskAbstention EvaluationTaskKind = "abstention"
)

type Event 

type Event struct {
	Sequence uint64            `json:"sequence"`
	Kind     EventKind         `json:"kind"`
	Source   EventSource       `json:"source"`
	Time     time.Time         `json:"time"`
	Fields   map[string]string `json:"fields,omitempty"`
}

Event is one ordered provider-neutral observation.

type EventKind 

type EventKind string

EventKind is the provider-neutral class of one observed agent action. 

const (
	// EventFileRead records one observed Project read.
	EventFileRead EventKind = "file_read"
	// EventFileWrite records one observed Project mutation.
	EventFileWrite EventKind = "file_write"
	// EventCommandStarted records trusted process execution before completion.
	EventCommandStarted EventKind = "command_started"
	// EventCommandFinished records trusted process completion.
	EventCommandFinished EventKind = "command_finished"
	// EventMCPToolCalled records one trusted MCP request and result classification.
	EventMCPToolCalled EventKind = "mcp_tool_called"
	// EventMessage records inert provider text after redaction.
	EventMessage EventKind = "message"
	// EventRunFinished records agent session completion.
	EventRunFinished EventKind = "run_finished"
)

type EventSource 

type EventSource string

EventSource identifies the observation boundary that produced an event. 

const (
	// EventSourceSupervisor identifies evidence observed by the execution backend or supervisor.
	EventSourceSupervisor EventSource = "supervisor"
	// EventSourceAdapter identifies diagnostic-only telemetry reported by an agent adapter.
	EventSourceAdapter EventSource = "adapter"
)

type ExecutionBackend 

type ExecutionBackend interface {
	Name() string
	Capabilities(context.Context) ([]Capability, error)
	Open(context.Context, BackendRequest) (BackendEnvironment, error)
}

ExecutionBackend creates one private execution boundary and reports what it can prove.

type GeneratorRequirement 

type GeneratorRequirement struct {
	ID        string
	Arguments []string
}

GeneratorRequirement defines one exact successful GoForj generator action without coupling it to a shell spelling.

type Guidance 

type Guidance struct {
	Profile string
	Files   map[string][]byte
	Skills  []string
	MCP     []string
}

Guidance is the exact native instructions, skills, and MCP selection installed for one treatment.

func ResolveProjectGuidance 

func ResolveProjectGuidance(profile string, facts project.Project) (Guidance, error)

Resolve creates one immutable treatment from the canonical Atlas guidance composer.

type GuidanceDiagnosticAttempt 

type GuidanceDiagnosticAttempt struct {
	Profile string        `json:"profile"`
	Result  AttemptResult `json:"result"`
	Error   string        `json:"error,omitempty"`
	Cause   error         `json:"-"`
}

GuidanceDiagnosticAttempt retains one treatment's result and any operational error independently.

type GuidanceDiagnosticRequest 

type GuidanceDiagnosticRequest struct {
	LogicalTrialID    string
	Definition        EvaluationDefinition
	DestinationRoot   string
	ForjExecutable    string
	Environments      map[string][]string
	Runtime           RuntimeIdentity
	Profiles          []string
	TreatmentBoundary func(context.Context) error
}

GuidanceDiagnosticRequest identifies one ordered pair of guidance treatments.

type GuidanceDiagnosticResult 

type GuidanceDiagnosticResult struct {
	LogicalTrialID string                      `json:"logical_trial_id"`
	Attempts       []GuidanceDiagnosticAttempt `json:"attempts"`
}

GuidanceDiagnosticResult contains both treatments even when one attempt fails operationally.

type GuidanceFileIdentity 

type GuidanceFileIdentity struct {
	Path   string `json:"path"`
	Digest string `json:"digest"`
}

GuidanceFileIdentity records one native instruction projection without retaining its content in run metadata.

type GuidanceResolver 

type GuidanceResolver interface {
	Resolve(context.Context, string, PreparationResult) (Guidance, error)
}

GuidanceResolver projects one named treatment from the prepared Project's trusted identity.

type Limits 

type Limits struct {
	WallTime     time.Duration
	Commands     int
	ShellNetwork string
}

Limits bounds one logical evaluation trial independently from provider defaults.

type LocalDiagnosticTreatmentRequest 

type LocalDiagnosticTreatmentRequest struct {
	EvaluationID    string
	GuidanceProfile string
	DestinationRoot string
	Environment     []string
	LogicalTrialID  string
}

LocalDiagnosticTreatmentRequest identifies one guidance treatment without requiring a paired comparison.

type LocalGuidanceDiagnostic 

type LocalGuidanceDiagnostic struct {
	// contains filtered or unexported fields
}

LocalGuidanceDiagnostic owns Atlas's standard paired local diagnostic wiring. Hosts retain their Project preparation, private environments, runtime identity, and filesystem lifecycle.

func NewLocalGuidanceDiagnostic 

func NewLocalGuidanceDiagnostic(options LocalGuidanceDiagnosticOptions) (*LocalGuidanceDiagnostic, error)

NewLocalGuidanceDiagnostic creates the Atlas-owned registry, promoted verifier, Codex adapter, unconfined backend, artifact store, and runner.

func (*LocalGuidanceDiagnostic) Run 

func (diagnostic *LocalGuidanceDiagnostic) Run(ctx context.Context, request LocalGuidanceDiagnosticRequest) (GuidanceDiagnosticResult, error)

Run evaluates the promoted definition with fresh no-guidance and AGENTS.md treatments.

func (*LocalGuidanceDiagnostic) RunTreatment 

func (diagnostic *LocalGuidanceDiagnostic) RunTreatment(ctx context.Context, request LocalDiagnosticTreatmentRequest) (GuidanceDiagnosticAttempt, error)

RunTreatment evaluates one promoted definition and guidance profile through the same diagnostic runner used by comparisons.

type LocalGuidanceDiagnosticOptions 

type LocalGuidanceDiagnosticOptions struct {
	WorkRoot            string
	ArtifactRoot        string
	ArtifactKey         []byte
	Redactor            Redactor
	Preparer            ProjectPreparer
	Codex               CodexOptions
	GoExecutable        string
	ForjExecutable      string
	VerifierEnvironment []string
	// VerifierModuleProxy is the host-owned read-only Go module proxy for verifier commands.
	VerifierModuleProxy string
	Runtime             RuntimeIdentity
}

LocalGuidanceDiagnosticOptions supplies host-owned boundaries to the standard local diagnostic service.

type LocalGuidanceDiagnosticRequest 

type LocalGuidanceDiagnosticRequest struct {
	EvaluationID      string
	DestinationRoot   string
	Environments      map[string][]string
	LogicalTrialID    string
	Profiles          []string
	TreatmentBoundary func(context.Context) error
}

LocalGuidanceDiagnosticRequest identifies one paired treatment while keeping host-private environments outside Atlas policy wiring.

type Milestone 

type Milestone string

Milestone is one monotonic trusted lifecycle boundary. 

const (
	// MilestonePreflight records successful contract and capability resolution.
	MilestonePreflight Milestone = "preflight"
	// MilestoneProviderSessionStarted records acquisition of a provider session.
	MilestoneProviderSessionStarted Milestone = "provider_session_started"
	// MilestonePromptDelivered records successful prompt submission.
	MilestonePromptDelivered Milestone = "prompt_delivered"
	// MilestoneFirstAgentAction records the first trusted action or terminal response.
	MilestoneFirstAgentAction Milestone = "first_agent_action"
	// MilestoneAgentTerminal records provider-side terminal completion.
	MilestoneAgentTerminal Milestone = "agent_terminal"
	// MilestoneEvaluationTerminal records verifier and cleanup completion.
	MilestoneEvaluationTerminal Milestone = "evaluation_terminal"
)

type PreparationCapabilities 

type PreparationCapabilities struct {
	ScenarioSchemaVersions []int
}

PreparationCapabilities describe the schemas and controls a Project preparer supports.

type PreparationRequest 

type PreparationRequest struct {
	ScenarioID      string
	DestinationRoot string
	ForjExecutable  string
	OrchestrationID string
	Environment     []string
}

PreparationRequest identifies trusted scenario inputs without carrying reusable authority.

type PreparationResult 

type PreparationResult struct {
	ResolutionID   string
	ProjectRoot    string
	ScenarioID     string
	ScenarioSchema int
	PlanDigest     string
	CatalogDigest  string
	BaselineTree   string
	ForjExecutable string
	ForjDigest     string
	OwnedPaths     []string
}

PreparationResult records the exact Project and tool identities returned by preparation.

type PreparedAgent 

type PreparedAgent struct {
	Name             string
	Executable       string
	ExecutableDigest string
	AuthorityDigest  string
	Model            string
	Environment      RunEnvironment
}

PreparedAgent records attributable agent identity and private configuration.

type PreparedProject 

type PreparedProject interface {
	Result() PreparationResult
	Close(context.Context) error
}

PreparedProject owns one prepared Project until the supervisor closes it.

type ProjectChange 

type ProjectChange struct {
	Path   string           `json:"path"`
	Before ProjectPathState `json:"before"`
	After  ProjectPathState `json:"after"`
}

ProjectChange records one supervisor-computed path change between the treatment baseline and sealed Project.

type ProjectGuidanceResolver 

type ProjectGuidanceResolver struct{}

ProjectGuidanceResolver composes treatments from the exact prepared Project rather than caller-supplied facts.

func (ProjectGuidanceResolver) Resolve 

func (ProjectGuidanceResolver) Resolve(ctx context.Context, profileName string, preparation PreparationResult) (Guidance, error)

Resolve discovers the prepared Project and delegates to the canonical profile composer.

type ProjectPathState 

type ProjectPathState struct {
	Kind   string `json:"kind,omitempty"`
	Digest string `json:"digest,omitempty"`
	Mode   uint32 `json:"mode,omitempty"`
}

ProjectPathState identifies one Project path at a sealed snapshot. A zero value means the path was absent.

type ProjectPreparer 

type ProjectPreparer interface {
	Capabilities(context.Context) (PreparationCapabilities, error)
	Resolve(context.Context, PreparationRequest) (ResolvedPreparationPlan, error)
	Prepare(context.Context, PreparationRequest, ResolvedPreparationPlan) (PreparedProject, error)
	MaterializeGuidance(context.Context, PreparedProject, Guidance) (Guidance, error)
}

ProjectPreparer is the Atlas-owned boundary implemented by GoForj.

type ProviderTelemetry 

type ProviderTelemetry struct {
	EventsObserved           uint64 `json:"events_observed"`
	BytesObserved            uint64 `json:"bytes_observed"`
	EventsDropped            uint64 `json:"events_dropped,omitempty"`
	BytesDropped             uint64 `json:"bytes_dropped,omitempty"`
	NotificationsDropped     uint64 `json:"notifications_dropped,omitempty"`
	NotificationBytesDropped uint64 `json:"notification_bytes_dropped,omitempty"`
	CommandsObserved         uint64 `json:"commands_observed"`
}

ProviderTelemetry describes bounded diagnostic retention without promoting adapter observations to trusted evidence.

func (ProviderTelemetry) Truncated 

func (telemetry ProviderTelemetry) Truncated() bool

Truncated reports whether any provider-originated evidence was omitted before normalization.

type Redactor 

type Redactor struct {
	// contains filtered or unexported fields
}

Redactor removes known credentials and common secret-bearing values before persistence.

func NewRedactor 

func NewRedactor(secrets []string) Redactor

NewRedactor snapshots non-empty secret values in longest-first order.

func (Redactor) Event 

func (redactor Redactor) Event(event Event) Event

Event redacts a copy so callers cannot mutate or later reveal the persisted evidence.

func (Redactor) JSONValue 

func (redactor Redactor) JSONValue(value any) any

JSONValue recursively redacts string keys and values while preserving typed JSON structure.

func (Redactor) Text 

func (redactor Redactor) Text(value string) string

Text redacts secrets and removes terminal or directional controls so human renderers receive inert content.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry owns the promoted workflow and verifier contracts available to manifest resolution.

func NewRegistry 

func NewRegistry(workflows []WorkflowExpectation, verifiers []Verifier) (*Registry, error)

NewRegistry validates and snapshots promoted contracts so later caller mutation cannot change resolution.

func (*Registry) Resolve 

func (registry *Registry) Resolve(definition EvaluationDefinition) (ResolvedEvaluation, error)

Resolve binds exact references and computes capabilities without allowing manifest overrides.

type RequirementKind 

type RequirementKind string

RequirementKind separates framework workflow gates from optional quality signals. 

const (
	// RequirementWorkflow is a declared framework action required for conformance.
	RequirementWorkflow RequirementKind = "workflow"
	// RequirementQuality is recorded for calibration but does not fail conformance.
	RequirementQuality RequirementKind = "quality"
)

type ResolvedEvaluation 

type ResolvedEvaluation struct {
	Definition   EvaluationDefinition
	Workflow     WorkflowExpectation
	Verifier     Verifier
	Capabilities []Capability
}

ResolvedEvaluation binds one manifest to exact promoted contracts and their capability union.

type ResolvedPreparationPlan 

type ResolvedPreparationPlan struct {
	ResolutionID         string
	ScenarioID           string
	ScenarioSchema       int
	PlanDigest           string
	ScenarioPlanDigest   string
	CatalogDigest        string
	ForjDigest           string
	EnvironmentDigest    string
	DependencyDigests    map[string]string
	ProjectConfiguration []byte
	TargetOmitted        bool
}

ResolvedPreparationPlan is an immutable data contract authenticated by the trusted caller.

type RunEnvironment 

type RunEnvironment struct {
	ProjectRoot string
	HomeRoot    string
	Environment []string
}

RunEnvironment is the backend-owned namespace presented to an agent adapter.

type RunIntent 

type RunIntent string

RunIntent determines whether unavailable evidence blocks execution or remains an explicit diagnostic limitation. 

const (
	// IntentAuthoritative requires every imported observation capability before Project mutation.
	IntentAuthoritative RunIntent = "authoritative"
	// IntentDiagnostic permits useful outcome evaluation while marking unsupported evidence endpoints ineligible.
	IntentDiagnostic RunIntent = "diagnostic"
)

type Runner 

type Runner struct {
	Registry  *Registry
	Preparer  ProjectPreparer
	Backend   ExecutionBackend
	Agent     EvaluationAgent
	Guidance  GuidanceResolver
	Artifacts *ArtifactStore
	Now       func() time.Time
}

Runner coordinates trusted preparation, one agent session, verification, and cleanup.

func (Runner) Run 

func (runner Runner) Run(ctx context.Context, request AttemptRequest) (result AttemptResult, runErr error)

Run executes one logical attempt while preserving agent outcome and evaluator failures separately.

func (Runner) RunGuidanceDiagnostic 

func (runner Runner) RunGuidanceDiagnostic(ctx context.Context, request GuidanceDiagnosticRequest) (GuidanceDiagnosticResult, error)

RunGuidanceDiagnostic runs two isolated treatments against the same promoted definition.

type RuntimeIdentity 

type RuntimeIdentity struct {
	Supervisor         SoftwareIdentity `json:"supervisor"`
	Framework          SoftwareIdentity `json:"framework"`
	GoVersion          string           `json:"go_version"`
	GoExecutableDigest string           `json:"go_executable_digest,omitempty"`
	GoRootDigest       string           `json:"go_root_digest,omitempty"`
	GOOS               string           `json:"goos"`
	GOARCH             string           `json:"goarch"`
}

RuntimeIdentity records the supervisor, framework, and selected Go launcher provenance needed to reconstruct a diagnostic environment.

type SafeAbstentionVerifier 

type SafeAbstentionVerifier struct{}

SafeAbstentionVerifier accepts a precise execution-mode clarification without allowing speculative Project changes.

func NewSafeAbstentionVerifier 

func NewSafeAbstentionVerifier() *SafeAbstentionVerifier

NewSafeAbstentionVerifier creates the promoted unknown-framework-shape verifier.

func (*SafeAbstentionVerifier) Capabilities 

func (*SafeAbstentionVerifier) Capabilities() []Capability

Capabilities requires an adapter that captures the provider's exact terminal response.

func (*SafeAbstentionVerifier) ID 

func (*SafeAbstentionVerifier) ID() string

ID returns the immutable verifier contract identifier.

func (*SafeAbstentionVerifier) Verify 

func (*SafeAbstentionVerifier) Verify(_ context.Context, input VerificationInput) (VerificationResult, error)

Verify accepts only a closed, decision-relevant clarification without authored Project changes.

type SealedProject 

type SealedProject struct {
	Root       string
	TreeDigest string
}

SealedProject is the immutable verifier input captured after every agent descendant has stopped.

type SecondaryFailure 

type SecondaryFailure struct {
	Phase   string `json:"phase"`
	Message string `json:"message"`
	// Cause retains operational identity for callers without serializing host error details.
	Cause error `json:"-"`
}

SecondaryFailure preserves evaluator failures without overwriting the agent outcome.

type SoftwareIdentity 

type SoftwareIdentity struct {
	Module  string `json:"module"`
	Version string `json:"version"`
	Commit  string `json:"commit,omitempty"`
	Dirty   bool   `json:"dirty,omitempty"`
}

SoftwareIdentity records a retrievable release or development revision without relying on a temporary executable path.

type TriageDisposition 

type TriageDisposition string

TriageDisposition is a human-confirmed product cause kept outside immutable attempt artifacts.

type TriageRecord 

type TriageRecord struct {
	State          TriageState `json:"state"`
	SuspectedCause string      `json:"suspected_cause,omitempty"`
	Confidence     string      `json:"confidence,omitempty"`
	EvidenceNeeded []string    `json:"evidence_needed,omitempty"`
}

TriageRecord keeps automated suspicion distinct from a confirmed product cause.

type TriageReview 

type TriageReview struct {
	AttemptID   string            `json:"attempt_id"`
	Disposition TriageDisposition `json:"disposition"`
	Reviewer    string            `json:"reviewer"`
	ReviewedAt  time.Time         `json:"reviewed_at"`
}

TriageReview associates a later human disposition with an authenticated attempt without rewriting its artifacts.

func NewTriageReview 

func NewTriageReview(attemptID string, disposition TriageDisposition, reviewer string, reviewedAt time.Time) (TriageReview, error)

NewTriageReview creates an external review record; callers must store it in their review system rather than mutating signed attempt artifacts.

type TriageState 

type TriageState string

TriageState records whether a human has established the cause of a failed attempt. 

const (
	// TriageUnreviewed indicates that no product disposition has been confirmed.
	TriageUnreviewed TriageState = "unreviewed"
	// TriageNeedsEvidence indicates that retained evidence cannot yet support a disposition.
	TriageNeedsEvidence TriageState = "needs-evidence"
)

type TrustedTestFile 

type TrustedTestFile struct {
	Path string
	Body []byte
	Mode uint32
}

TrustedTestFile retains one immutable pre-agent test for isolated verifier clones.

type UnconfinedLocal 

type UnconfinedLocal struct {
	WorkRoot string
}

UnconfinedLocal provides disposable process state without claiming host, command, filesystem, or network isolation.

func (UnconfinedLocal) Capabilities 

func (UnconfinedLocal) Capabilities(context.Context) ([]Capability, error)

Capabilities intentionally returns none because host-local process execution is not authoritative evidence.

func (UnconfinedLocal) Name 

func (UnconfinedLocal) Name() string

Name returns the backend identity recorded in diagnostic artifacts.

func (UnconfinedLocal) Open 

func (backend UnconfinedLocal) Open(_ context.Context, request BackendRequest) (BackendEnvironment, error)

Open creates private agent state outside the candidate-writable Project tree.

type VerificationInput 

type VerificationInput struct {
	ProjectRoot  string
	BaselineTree string
	FinalTree    string
	// BaselineTests contains supervisor-captured tests from before the agent received the Project.
	BaselineTests []TrustedTestFile
	// Changes is the supervisor-computed, path-level projection of the sealed Project delta.
	Changes []ProjectChange
	Events  []Event
	// FinalResponse is the exact terminal provider response captured by the adapter.
	FinalResponse string
}

VerificationInput contains sealed Project and evidence identities, never candidate verifier code.

type VerificationResult 

type VerificationResult struct {
	FrameworkOutcome    EndpointResult   `json:"framework_outcome"`
	WorkflowConformance EndpointResult   `json:"workflow_conformance"`
	Contract            *EndpointResult  `json:"contract,omitempty"`
	Abstention          *EndpointResult  `json:"abstention,omitempty"`
	Checks              []EndpointResult `json:"checks,omitempty"`
}

VerificationResult separates framework behavior from workflow conformance.

type Verifier 

type Verifier interface {
	ID() string
	Capabilities() []Capability
	Verify(context.Context, VerificationInput) (VerificationResult, error)
}

Verifier is one promoted deterministic outcome contract.

func PromotedVerifiers 

func PromotedVerifiers(runner CommandRunner) []Verifier

PromotedVerifiers returns every live verifier that can run through the selected isolated command boundary.

type VerifierCommands 

type VerifierCommands struct {
	WorkRoot       string
	GoExecutable   string
	ForjExecutable string
	Environment    []string
	// ModuleProxy is the host-owned read-only Go module proxy exposed to verifier commands.
	ModuleProxy string
}

VerifierCommands executes an allowlisted toolchain in a disposable copy of the candidate Project.

func (VerifierCommands) Open 

func (runner VerifierCommands) Open(ctx context.Context, project VerifierProject) (CommandSession, error)

Open clones the sealed Project for exactly one verifier phase, so no phase can observe another phase's candidate-controlled mutation.

type VerifierProject 

type VerifierProject struct {
	Root                   string
	BaselineTests          []TrustedTestFile
	BaselineTestExclusions []string
}

VerifierProject combines the sealed candidate with immutable pre-agent tests restored into its private clone.

type WorkflowExpectation 

type WorkflowExpectation struct {
	ID           string
	Requirements []WorkflowRequirement
	Generators   []GeneratorRequirement
}

WorkflowExpectation is a promoted and versioned framework workflow contract.

func PromotedWorkflows 

func PromotedWorkflows() []WorkflowExpectation

PromotedWorkflows returns the versioned workflow contracts available to live evaluations.

type WorkflowRequirement 

type WorkflowRequirement struct {
	ID          string
	Kind        RequirementKind
	Capability  Capability
	Description string
	Paths       []string
}

WorkflowRequirement defines one typed, observation-backed workflow expectation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.