execution

package
v0.0.0-...-5a20295 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2026 License: Apache-2.0 Imports: 58 Imported by: 0

Documentation

Overview

Package execution's orchestrator is the single response kernel: it drives a run through the canonical state machine and a live engine channel, committing every provider and tool result before it reaches the engine. It writes no second agent loop — the engine owns the loop; the orchestrator only correlates requests, commits state, and dispatches (spec §24.7, §25.10).

Index

Constants

View Source 
const DefaultAbandonedLeaseGrace = 2 * time.Minute

DefaultAbandonedLeaseGrace is how long after a run reaches terminal the sweep waits before it will touch that run's writer lease. It exists for ONE window and it is a short one: UpdateRunState commits the terminal transition before ExecuteAttempt's deferred releaseWorkspace has run, so for the length of an attempt's unwind the run reads terminal while the process that owns the lease is still finishing with the directory. Two minutes is far longer than that unwind and far shorter than the interval over which a held machine matters.

It is NOT the idle TTL and must not be confused with it. The idle TTL asks "has this thread gone quiet long enough to take its machine away", a question about a person; this asks "has the process that held this lease finished exiting", a question about a goroutine. Making the second one long would leave a machine held for no reason; making the first one short would take a workspace away from someone still reading their diff.

View Source 
const DefaultIdleWorkspaceTTL = 5 * time.Minute

DefaultIdleWorkspaceTTL is how long a session's workspace may hold a machine with nothing happening on it before the sweep archives it and hands the machine back. Five minutes is the operator-facing number: long enough that a human reading a diff and typing a reply keeps their workspace warm, short enough that a Mac is not held overnight by a conversation somebody walked away from.

A knob (PALAI_WORKSPACE_IDLE_TTL), not a switch. There is no "off": a deployment that wants a workspace held longer says how much longer.

Variables

View Source 
var ErrEmptyCheckpoint = errors.New("empty checkpoint state")

ErrEmptyCheckpoint reports an offer whose decoded state is empty (absent or non-string): there is nothing to restore, so it is rejected rather than stored as a 0-byte object + size-0 row.

View Source 
var ErrExactStandDown = fmt.Errorf("exact stand-down: %w", coordinator.ErrSoftRequeue)

ErrExactStandDown reports that this attempt took the exact rung and stood down because a sibling response.run job holds a live lease (spec §26.3 rung 1). It wraps coordinator.ErrSoftRequeue, so the worker SOFT-requeues the job (no attempt-budget consumed) rather than completing or failing it: completing would hang the run when two jobs mutually stand down, and failing would dead-letter a run whose live sibling outlasts MaxAttempts (MUST-FIX #2). The standby politely requeues (jitter breaks the mutual-standoff symmetry) until the sibling reaches terminal, then a claim finds the run terminal and exits cleanly — never dead-lettering on a stand-down alone.

View Source 
var ErrHostQuarantined = errors.New("host quarantined")

ErrHostQuarantined reports that a placement was refused because the host was quarantined by a prior allocation-destroy failure (spec §29 SAN-008). A run already executing on the host is untouched; only a NEW allocation is denied, so a poisoned host stops accepting tenants until an operator clears it.

View Source 
var ErrMachineUnreachable = errors.New("no reachable session for this machine")

ErrMachineUnreachable is what a caller gets when the named machine has no live session on this gateway, or has one this control plane is not allowed to speak on. It is deliberately ONE error for both, because a caller's correct response is the same for either: report `lost` — the process may still be running and this control plane cannot prove it is ours — and never signal.

View Source 
var ErrPoolHasNoRunner = errors.New("runner pool has no machine for this tenant")

ErrPoolHasNoRunner is returned by Dial when the attempt's pool held NO machine of the attempt's tenant for the whole dial budget. It is the difference between "everything is busy" and "there is nothing here", and that difference is why it exists as its own error: the orchestrator PARKS the run on it (§3.6 D12) instead of failing the attempt, so a Mac that takes six to twenty minutes to boot still finds its run waiting when it arrives. A pool whose machines are merely all leased returns the context error exactly as before and rides the existing retry ladder.

View Source 
var ErrRecoveryImpossible = errors.New("workspace recovery impossible")

ErrRecoveryImpossible reports that a host-lost workspace cannot be recovered: there is no byte-archived snapshot to restore from, or the archive is missing/corrupt (spec §26.3 rung 4, §29.7). The workspace is driven recovering→failed with a typed reason rather than left dangling or resumed on an empty tree.

View Source 
var ErrRunnerCordoned = errors.New("runner gateway cordoned: no new leases")

ErrRunnerCordoned is returned by Dial when the gateway is cordoned: no NEW lease is offered so an attempt requeues rather than dispatching onto a runner that is draining for an upgrade (§48.4). An in-flight lease is untouched — cordon stops new work, drain waits for the current work to finish.

View Source 
var ErrRunnerRevoked = errors.New("runner gateway revoked: leases and session frames refused")

ErrRunnerRevoked is returned by Dial (and closes an incoming connect) when the gateway is revoked: a decommissioned/compromised runner's new leases AND stale session frames are refused (SAN-011). Revoke is the hard stop cordon is not — a cordoned runner still completes its lease, a revoked one does not.

View Source 
var ErrSnapshotArchiveMissing = errors.New("snapshot archive missing")

ErrSnapshotArchiveMissing reports that a snapshot row exists but its byte-archive is absent from the object store (a manifest-only E09 snapshot, or lost bytes): the restore has nothing to reconstruct from, so a recovering workspace fails EXPLICITLY (recovering→failed, spec §26.3 rung 4) rather than resuming on an empty tree.

View Source 
var MaxSnapshotArchiveBytes int64 = 64 * 1024 * 1024

MaxSnapshotArchiveBytes bounds a workspace snapshot archive. It matches the object store's read bound (artifacts.maxReadBytes) so a snapshot that could never be GET back for a restore is rejected at capture rather than written and later un-restorable. A var, not a const, so a deployment can tune it (and a test can lower it to drive the overflow path without a 64 MiB fixture). ponytail: 64 MiB covers the local-tier repos; a larger repo needs streaming archive/restore — this buffers the whole archive.

Functions

func AdvanceRun 

func AdvanceRun(advancer RunAdvancer) coordinator.Handler

AdvanceRun is the coordinator Handler that turns a claimed response.run job into durable run assignment: it drives the referenced run queued -> provisioning -> running through ApplyRunTransition, which emits the journal events the SSE layer already serves. It is idempotent under redelivery — a step already applied by an earlier attempt (for example after the previous worker was killed mid-assign) is skipped rather than errored, so a reclaimed job resumes instead of failing.

"Already applied" is narrowed to a non-terminal run that has moved past this step: a run that reached a terminal state by another path (cancellation before dispatch) is not silently reported as assigned. The job stops and records that the run was terminal, so the queue is cleared without asserting an assignment that never happened. The result hash records the outcome for the authoritative completion.

func BackgroundDisabled 

func BackgroundDisabled() bool

BackgroundDisabled reports the deployment kill switch, and it FAILS CLOSED on a value it cannot read: an operator who wrote `PALAI_BACKGROUND_DISABLED=yes` meant to switch the feature off, and answering "that is not a boolean, so the feature stays on" would be the least useful reading available.

It is read here, per call, rather than captured once at wiring time — the same reason backgroundRunnerFor is a named function in main.go: a test that builds its own configuration never exercises the configuration production builds, and this way the switch is reached through the production path or not at all.

func BackgroundLogTTL 

func BackgroundLogTTL() time.Duration

BackgroundLogTTL bounds how long a finished task's output file stays on the allocation. Unset and unparseable both mean the 24h default: there is no opt-out spelling here, because the opt-out is an unbounded `.palai-session` that Snapshot skips (§3.6 D7) and that therefore nobody would ever see fill a disk. An operator who wants to keep logs longer writes a longer duration.

func BackgroundMaxPerHost 

func BackgroundMaxPerHost() int

func BackgroundMaxPerRun 

func BackgroundMaxPerRun() int

BackgroundMaxPerRun and BackgroundMaxPerHost are §0.3's two ceilings. A non-positive value is an explicit opt-out and means no ceiling — the same shape as the wall time's zero, and the same rule: the unbounded reading requires a written value.

func BackgroundMaxWallTime 

func BackgroundMaxWallTime() (limit time.Duration, bounded bool)

BackgroundMaxWallTime is the deployment's ceiling on a background task's life, and its `bounded` half carries the one thing a duration cannot say: whether the operator ASKED for no ceiling.

UNSET IS BOUNDED. `PALAI_BACKGROUND_MAX_WALL_TIME=0` is unbounded and has to be written, so the silent case is the safe one — the exact inverse of PALAI_FLEET_PARK_TTL, and the difference is that here the unbounded case is the failure the epic is named after.

A VALUE THIS CANNOT PARSE FALLS BACK TO THE CEILING RATHER THAN TO INFINITY. An operator who typed `60min` meant to bound something, and reading a typo as "run forever" is the least useful answer available.

func CompileChangeset 

func CompileChangeset(ctx context.Context, ledger ChangesetLedger, aw ArtifactWriter, in ChangesetInput) (coordinator.ChangesetRecord, bool, error)

CompileChangeset compiles a first-class, immutable changeset from the run's file-tool write ledger — NOT from the model's prose (spec §30.6, REP-005): the changed-file set + provenance come from the tool_calls the run actually issued, the patch is the real working-tree diff against the preparation base, and any likely-committed-secret is a finding. It writes the patch + test-log artifacts to the object store, records the changeset, and returns it. compiled is false when the run prepared no repository (no base to diff against) — the caller then has no changeset to record.

It is a COMPOSED step (like PrepareRepository) the orchestrator's finalize auto-invokes for a run that bound a workspace (E09 Task 10): the changeset is compiled while the workspace is still on disk and the writer lease still held, so a terminated coding run records its changeset without any caller driving it. The live smoke + coding journey drive the same step against a real provider.

func EngineWallTime 

func EngineWallTime() time.Duration

EngineWallTime reads the operator's bound, falling back to the measured default.

‼️ IT IS A KNOB BECAUSE THE RIGHT VALUE IS A PROPERTY OF THE DEPLOYMENT, not of this code: model latency, repository size and machine speed all move it, and a fixed number that kills legitimate work is the defect this replaced. PALAI_SANDBOX_WALL_TIME is the same decision one layer down for a single shell call, with the same grammar.

AN UNPARSEABLE VALUE IS THE DEFAULT AND NOT A ZERO, deliberately: `runner.Limits` refuses a non-positive wall time, so coercing "5min" to 0 would take the deployment from "runs are slow" to "nothing runs at all" — the shell posture's own note records that exact trap.

func ExecuteRun 

func ExecuteRun(_ *coordinator.Store, _ *store.Store, orch *Orchestrator) coordinator.Handler

ExecuteRun is the production worker handler: it drives a claimed response.run job to a terminal outcome through the orchestrator. Where AdvanceRun (the listener-off wiring) only assigns the run, ExecuteRun opens the engine and runs the full kernel: ExecuteAttempt applies the same idempotent provision -> start transitions before dialing, so a redelivered or reclaimed job resumes at the claim's fence under a fresh attempt id, and any error falls to the coordinator's retry/dead-letter policy — no hidden retry. The engine image the control plane pins into every lease comes from PALAI_ENGINE_IMAGE (compose deploy config).

The spine and store are the coordinator's durable dependencies the handler interface carries; this implementation routes entirely through the orchestrator, which already owns both, so they are unused here.

func MergeChildBranch 

func MergeChildBranch(ctx context.Context, store MergeStore, tenant coordinator.Tenant, in MergeChildBranchInput) (repositories.MergeResult, error)

MergeChildBranch performs an EXPLICIT conflict-aware merge of a child's branch into the parent worktree and records the outcome, naming the source child run (spec §30.5, REP-011). A conflict is a recorded merged=false result — the parent worktree is left consistent (the merge aborted), never silently overwritten. The merge is a local Git operation; no credential is involved.

func MigrateCheckpoint 

func MigrateCheckpoint(ctx context.Context, sink *CheckpointSink, spine *coordinator.Store, tenant coordinator.Tenant, sessionID, responseID, fromCheckpointID, fromFormat string, meta CheckpointMeta, v2OfferData map[string]any) (toCheckpointID string, err error)

MigrateCheckpoint persists a migrated (v2) checkpoint as a NEW immutable row alongside the original and journals the provenance link (spec §26.2, ENG-011). The v1->v2 transform is engine-owned (checkpoint.migrate, proven in the reference kernel) and the control plane treats the bytes opaquely (§26.2), so this reuses the ordinary offer/persist path for the migrated bytes and records checkpoint.migrated.v1 {from_id, to_id, from_format, to_format}. The original checkpoint is never touched (append-only) — it stays integrity-valid and restore-selectable, which IS the rollback.

Ceiling: the MECHANISM is proven with the reference-kernel v2 SHAPE. The production engine stays v1 (engine.ready.checkpoint_formats == ["reference-kernel/1"], schema-pin unchanged) — no live run migrates yet; this is the reversible-migration seam a future format bump would call. It is a free function (not a CheckpointSink method) so the sink stays pure persistence, uncoupled from the journal.

func NewMCPProgressSink 

func NewMCPProgressSink(spine *coordinator.Store) mcp.ProgressSink

NewMCPProgressSink wires the durable progress sink the MCP manager tags each tools/call progress notification through.

func SplitTenantSecretRef 

func SplitTenantSecretRef(ref modelbroker.SecretRef) (project, name string, ok bool)

SplitTenantSecretRef reverses TenantSecretRef. ok=false means the ref is an unqualified deployment handle (the env route's), which redeems through the env bridge exactly as before.

A ref carrying the prefix but NO owner segment is ALSO ok=false, and that is deliberate rather than lenient: it is the pre-000006 form, and the only safe reading of it is "this handle names no project". Treating it as a bare name would send it to the store under whatever tenant happened to be at hand, which is the failure this whole change exists to remove — so it falls through to the env bridge, and if no fallback is wired it is refused.

func TenantSecretRef 

func TenantSecretRef(project, name string) modelbroker.SecretRef

TenantSecretRef marks a connection's secret-ref handle as one a TENANT ROUTE minted, so Redeem sends it to the DB-backed store instead of the deployment env bridge, AND names the project it must be redeemed for. It stays a HANDLE: no credential value is ever encoded here.

THE OWNER SEGMENT IS BACK, AND THE ROUND TRIP IS WHY THIS PARAGRAPH IS LONG. It carried the owning ORGANIZATION originally; A.2 Task 6 removed it rather than swapping in the project, and that was the right call at the time for a reason worth keeping: identity's secret store had stopped enforcing any boundary, so a project segment would have "read as an isolation this deployment does not have". The segment was deleted because it had become decoration. 000006 gives secret_refs a project_id, so the isolation exists again and the segment is load-bearing again — this time it is not decoration, because RouteSecretResolver.Redeem SPENDS it on the lookup rather than only carrying it.

WHY IN THE REF AT ALL, rather than in the broker's signature: modelbroker.SecretResolver is `Redeem(ref) (string, error)` with no tenant and no context, and it is implemented four times over (StaticResolver, EnvResolver, ChainResolver, this) plus the conformance corpus. The ref is the only channel that already reaches every implementation, and nothing persists this form except the config snapshot's provenance field — it is built per dispatch from the stored bare name.

The separator is a single colon and the SPLIT IS ON THE FIRST ONE ONLY, because a secret name may legitimately contain colons: an environment value is stored under `env:<environment_id>:<key>`. A project id carries none (`prj_` + hex), so `tenant:<project>:<name>` parses back exactly.

func ToolErrorBudget 

func ToolErrorBudget() int

ToolErrorBudget exposes the reader the dispatcher actually calls, and it exists for exactly one caller: the composition root's TestEveryDesiredValueThisBinaryAcceptsIsParsedByItsOwnReader, which drives every value the write surface ACCEPTS through the REAL reader. That guard is the one that stops the panel showing one number while the process runs another, and it can only do its job if the reader is reachable from where it lives. A test-local copy of the parse would be the defect it fences.

Types

type ArtifactDeleter 

type ArtifactDeleter interface {
	Delete(ctx context.Context, objectKey string) error
}

ArtifactDeleter erases an artifact's bytes from the object store by object key. It is the seam the retention sweep uses to close the store:false purge, satisfied by the control-plane's S3-backed artifacts.Store (spec §24 — the deleter, and thus the S3 credential, lives only in the control plane).

type ArtifactWriter 

type ArtifactWriter interface {
	WriteArtifact(ctx context.Context, project, runID string, content []byte, mediaType, logicalType string, provenance map[string]any) (string, error)
}

ArtifactWriter is the object-store write-path the compiler persists the patch + test-log artifacts through (spec §22.6, T2), returning the artifact id. *artifacts.Writer implements it; a fake records the writes in the unit test. Primitive params keep this seam free of the artifacts package, so execution does not depend on the S3 write-path's types (the retention ArtifactDeleter decoupling, and it breaks the artifacts↔execution test import cycle).

type AttemptDescriptor 

type AttemptDescriptor struct {
	RunID     contracts.RunID
	AttemptID contracts.AttemptID
	Fence     uint64
	// Tenant is the org/project this attempt runs for, and it is here because the runner plane had NO
	// tenant on it at all (§3.6 D8): enrolment carried none, this descriptor carried none, the lease
	// offer carried none, and Dial checked none — so ANY enrolled machine could take ANY tenant's
	// attempt. In a single-runner topology that is a definition; the moment two customers have Macs it
	// is a hole. The orchestrator fills it from the run's own scope, which it has already resolved.
	//
	// IT IS DELIBERATELY NOT ON THE LEASE OFFER, and that is a correction to how the plan described the
	// fix: the refusal is a control-plane decision, and adding a field to the offer would tell a machine
	// something it cannot act on while breaking the byte-identical offer §2 makes non-negotiable.
	// Empty is the pre-E24 attempt (every wire proof, the conformance tier) and matches a machine whose
	// tenant is equally unknown — it is not a wildcard.
	Tenant      coordinator.Tenant
	ImageDigest string
	Limits      runner.Limits
	// WorkspaceHostPath is the host allocation directory this attempt's workspace tools confine to
	// and the runner bind-mounts to /workspace (spec §29.9). Empty means no workspace bound — the
	// pre-E09 behaviour. WorkspaceReadOnly binds a child's read-only snapshot (spec §29.8, T6).
	WorkspaceHostPath string
	WorkspaceReadOnly bool
	// WorkspaceID and WorkspaceAllocationID name the workspace this attempt holds and the allocation it
	// holds it THROUGH. They are here for the workspace store's publish path, and the allocation is the
	// load-bearing one: it is the fence.
	//
	// ‼️ IT MUST COME FROM THE LEASE, NOT FROM A LOOKUP AT PUBLISH TIME. The fence exists to refuse a
	// machine whose allocation has been superseded — the idle releaser handed the machine back and the
	// session came up somewhere else. A handler that resolved "the workspace's current allocation" itself
	// would compare that value against itself and the fence could never fire, which is the same defect as
	// a caller-supplied fence and looks even more correct. What this attempt was GIVEN at provisioning is
	// the only value that can disagree with what is true now.
	//
	// Empty is every tier that binds no workspace, and publishes nothing.
	WorkspaceID           string
	WorkspaceAllocationID string
	// WorkspaceUnsafe marks a §30.13 unsafe-local-bind (REP-012): the host path is deliberately
	// outside the runner's managed allocation root, so the runner skips its containment check. It
	// defaults false — a normal allocation must sit under the runner's root.
	WorkspaceUnsafe bool
	// JobID is the durable response.run job this attempt was claimed under, threaded so the recovery
	// ladder's "exact" rung can exclude the attempt's OWN live lease when it asks whether the ORIGINAL
	// process is still driving the run (spec §26.3 rung 1, E10 T4). Empty for a direct-drive attempt
	// with no claimed job (tests) — then any live sibling job wins exact.
	JobID string
	// PoolID is the runner pool this attempt must run in (E24 T2). The gateway offers it to a machine
	// enrolled in THAT pool and to no other — a pool is a posture, so "the nearest runner" is not a
	// weaker answer, it is the wrong machine. Empty means fleet.DefaultPoolID, which is where a
	// deployment that has configured no pool places every run, so its behaviour is unchanged.
	PoolID string
	// QueuedAt is the moment this attempt's RUN entered the queue — the run's created_at, not this
	// attempt's. It is what orders a pool's waiting attempts (E24 T2: created_at FIFO), and the
	// distinction is the whole point: a run that was requeued (a cordon, a retry, a resume) re-dials
	// with its ORIGINAL timestamp and keeps its place, instead of going to the back of the line every
	// time it is bounced. Zero means "order me by the moment I reached the gateway".
	QueuedAt time.Time
}

AttemptDescriptor is one run attempt the orchestrator executes: the fenced run/attempt identity, the pinned engine image, the execution bounds, and — since E24 T4 — the tenant the attempt belongs to.

type BlobWriter 

type BlobWriter interface {
	PutIfAbsent(ctx context.Context, project, digest string, body []byte) error
}

BlobWriter uploads content-addressed bytes.

type CapacityWaker 

type CapacityWaker interface {
	// WakeRunAwaitingCapacity drives that run waiting->running and enqueues its response.run job in ONE
	// transaction, and reports the run it woke (empty when there was none). A run parked for any OTHER
	// reason — a human's pause, an approval, a detached child — is NOT a candidate: those have their own
	// wakers and waking them here would override a decision somebody else owns.
	WakeRunAwaitingCapacity(ctx context.Context, tenant coordinator.Tenant, poolID string) (string, error)
}

CapacityWaker re-enters the oldest run parked on a pool for want of a machine. *coordinator.Store is the only implementation; the interface exists so the gateway does not depend on the database, exactly as it does not depend on it for the registry.

type CertIssuer 

type CertIssuer interface {
	SignRunnerCertificate(publicKeyDER []byte, runnerDNS string) (certificateDER []byte, err error)
}

CertIssuer signs an enrolling runner's public key into a short-lived client certificate with the local control-plane CA. The CA the gateway binds is injected (an in-test CA in the conformance proof); binding it to the .palai layout is Task 12.

type ChangesetInput 

type ChangesetInput struct {
	Tenant         coordinator.Tenant
	SessionID      string
	ResponseID     string
	RunID          string
	AllocationRoot string
}

ChangesetInput is the infrastructure-owned input to a changeset compile. AllocationRoot is the workspace allocation dir; the repo the changeset diffs lives at AllocationRoot/repo (spec §29.9).

type ChangesetLedger 

type ChangesetLedger interface {
	RunToolCalls(ctx context.Context, tenant coordinator.Tenant, runID string) ([]coordinator.ToolCallRow, error)
	RunBaseCommit(ctx context.Context, tenant coordinator.Tenant, runID string) (string, bool, error)
	RecordChangeset(ctx context.Context, tenant coordinator.Tenant, sessionID, responseID string, rec coordinator.ChangesetRecord) error
}

ChangesetLedger is the coordinator seam the compiler reads the run's tool ledger + base commit from and records the changeset through. *coordinator.Store implements it; a fake implements it in the unit test (the RepositoryStore idiom), so the projection is provable without a database.

type CheckpointMeta 

type CheckpointMeta struct {
	Project             string
	RunID               string
	AttemptID           string
	OfferSequence       int64
	EngineDigest        string
	EngineVersion       string
	ProtocolVersion     string
	ConfigSnapshotHash  string
	TranscriptSequence  int64
	WorkspaceSnapshotID string
	// PendingOperations is the run's unresolved (uncertain/manual_resolution) tool operations at the
	// boundary as a JSON array (spec §26.2, §26.4, E10 T7), CP-resolved from the tool ledger at persist
	// time — the engine's opaque offer never carries it. Empty is normalised to '[]' by the recovery layer.
	PendingOperations []byte
}

CheckpointMeta is the control-plane-resolved provenance the engine's OPAQUE offer does not carry (spec §26.2): tenant/run/attempt identity, the engine handshake identity, the effective config snapshot hash, and the journal boundary. The offer frame itself carries only format + the opaque bytes. OfferSequence is the engine frame sequence of this offer; it makes the checkpoint id STABLE so a retransmitted offer re-derives the same id and is rejected by the immutable-row guard rather than duplicated.

type CheckpointObjectStore 

type CheckpointObjectStore interface {
	Put(ctx context.Context, key string, body []byte) (checksum string, size int64, err error)
	// Get reads the opaque bytes back for a restore (spec §26.3 rung 2). found is false — with a nil
	// error — when the object is absent, so the ladder distinguishes a miss from a transport failure.
	Get(ctx context.Context, key string) (body []byte, found bool, err error)
}

CheckpointObjectStore is the object-store PUT the opaque checkpoint bytes are written through. The engine never sees the S3 credential (spec §24) — the control plane holds it — so this seam is a structural interface satisfied by the control-plane-only artifacts.Store and injected at composition. Declaring the interface here (not importing artifacts) keeps the same decoupling the changeset ArtifactWriter uses.

type CheckpointSink 

type CheckpointSink struct {
	// contains filtered or unexported fields
}

CheckpointSink persists an engine checkpoint.offer (spec §26.1-26.2): it decodes the opaque bytes, SIZE-BOUNDS them before the store PUT (so an oversize offer leaves no orphan object), writes them under a tenant-scoped key, and records the immutable metadata row. The bytes stay opaque — the control plane stores and checksums them, never interpreting them (§26.2, the engine boundary).

func NewCheckpointSink 

func NewCheckpointSink(store CheckpointObjectStore, objects *recovery.Objects) *CheckpointSink

NewCheckpointSink binds the object store and the recovery persistence layer.

func (*CheckpointSink) Persist 

func (s *CheckpointSink) Persist(ctx context.Context, meta CheckpointMeta, offerData map[string]any) error

Persist writes one checkpoint.offer's bytes + metadata. offerData is the engine frame's data map {format, format_version, boundary_kind, state}; state is base64 of the opaque canonical bytes. A retransmitted offer (same run/attempt/offer-sequence) re-derives the same id and is rejected by the immutable-row guard (recovery.ErrCheckpointExists), never written twice.

func (*CheckpointSink) Retrieve 

func (s *CheckpointSink) Retrieve(ctx context.Context, key string) (body []byte, computedChecksum string, found bool, err error)

Retrieve fetches an opaque checkpoint's bytes for a restore and returns their sha256 as "sha256:<hex>" (spec §26.3-26.4). The COMPUTED checksum is handed to the pure ladder, which compares it to the recorded one — a mismatch (a tampered or partial object) is the §26.4 checksum condition failing, so the checkpoint is rejected rather than restored as garbage. found is false when the object is absent (the ladder treats an absent checkpoint as no candidate).

type CloneToDir 

type CloneToDir func(ctx context.Context, tenant coordinator.Tenant, bindingID, ref, dir string) error

CloneToDir materialises a binding's repository into a local directory. It is a function rather than an interface because there is exactly one thing it does, and the two implementations that matter — the real preparation and a test's fixture writer — differ in nothing else.

type ConfigSnapshot 

type ConfigSnapshot struct {
	Hash       string            `json:"hash"`
	Model      string            `json:"model"`
	Tools      []string          `json:"tools"`
	SecretRef  string            `json:"secret_ref"`
	Provenance map[string]string `json:"provenance"`
	// Skills is the run's frozen skill set (E12 Task 7, spec §28.16), empty for a skill-less run. It rides
	// the snapshot as a progressive-loading rider (metadata only — the body reads from Path via the file
	// tool), and folds into Hash only when non-empty (bit-compat for a skill-less run).
	Skills []SkillRef `json:"skills,omitempty"`
}

ConfigSnapshot is the resolved, redacted, content-addressed effective configuration with per-value provenance (spec §14). Hash is SHA-256 over the canonical JSON of the effective values (the LP Task 11 content_hash pattern), so identical inputs address identically. SecretRef stays a reference — the credential value never enters the snapshot (redaction).

func Resolve 

func Resolve(in ResolveInput) ConfigSnapshot

Resolve layers deployment → project → session → request into the effective ConfigSnapshot (spec §14). The session model, when set, wins over the deployment default; a REQUEST model wins over the session, because a caller that named a model for one call named it for that call. The session tools, when set, win over the project baseline. Each effective value records the layer that set it, so the journal and API can explain why a model/tool was selected. Pure: no I/O, so the same input always yields the same hash.

type DestroyInput 

type DestroyInput struct {
	WorkspaceID string
	SessionID   string
	ResponseID  string
	HostPath    string
}

SessionID is what keys the account release, and it is CARRIED rather than derived from anything: SessionAccounts is defined on the session, Acquire was called with the session, and an identity recovered by parsing a path is the shape this tree keeps finding defeated. An AllocationID field lived here to key that release and was the wrong key for it; with the release corrected to SessionID it had no reader and no writer, so it is gone rather than left as a field a future caller would fill in.

type EngineChannel 

type EngineChannel interface {
	Send(ctx context.Context, frame contracts.EngineFrame) error
	Receive(ctx context.Context) (contracts.EngineFrame, error)
	Close() error
}

EngineChannel is a handshake-complete, single-attempt frame transport. The first frame Receive yields is engine.ready; a clean close returns io.EOF. Implementations own the engine lifecycle: the deterministic e2e drives a bare subprocess, and the hardened production path (Task 11c) drives the runner gateway — the orchestrator is written once against this seam and never learns which.

type EngineDialer 

type EngineDialer interface {
	Dial(ctx context.Context, attempt AttemptDescriptor) (EngineChannel, error)
}

EngineDialer opens a live channel for one attempt. Dial completes the handshake, so the channel it returns is ready for run.start. ctx bounds ONLY the dial and handshake — the orchestrator gives it an attempt-scoped deadline and stops honoring it once the handshake completes — so the returned channel must not tie its lifetime to ctx.

type EnrollmentTokens 

type EnrollmentTokens interface {
	Consume(token string) error
}

EnrollmentTokens redeems the bootstrap credential a machine presents to enrol. Consume returns an error for a credential this control plane does not recognise, or recognises and refuses.

IT IS NOT ONE-USE, AND THIS COMMENT USED TO SAY IT WAS (§3.6 D4, corrected in E24 T3). The claim was written when the only implementation spent a token permanently, and it outlived that by a long way: the shipped FileEnrollmentTokens re-reads its file on every call and admits a redemption once per issued-certificate lifetime, deliberately, because renewal runs over the certificate that is expiring and a machine that missed its window would otherwise have NO way back — see the threat model in local_credentials.go, which explains at length why one-use was replaced. Presenting a credential twice is therefore an ordinary event on this path; what a refusal means is "not live", not "already used".

type ExecAnswer 

type ExecAnswer struct {
	Result toolbroker.ShellResult
	Err    error
}

ExecAnswer is one exec.result, decoded. The two fields mirror the wire's two answer shapes and exactly one of them is set: data.result for a command that RAN (including one that exited non-zero), data.error for one that did not.

type ExecConn 

type ExecConn interface {
	// StartExec registers execID as awaiting an answer and then sends the request, in that order.
	// The order is the correctness argument: the connection's reader can deliver an answer the
	// instant after the write returns, and a registration made afterwards would race it.
	//
	// It returns the channel the answer arrives on and the release that unregisters the wait, which
	// the caller must call once it has stopped waiting.
	StartExec(ctx context.Context, execID string, cmd toolbroker.ShellCommand) (<-chan ExecAnswer, func(), error)
}

ExecConn is the lease connection RemoteShell asks a machine to run commands over. It is a narrow interface rather than *gatewayChannel so this file states what it needs — and so that the exec surface is something a caller ASSERTS for, which is the honest shape: Dial returns an EngineChannel, and whether that channel can also reach the machine is a fact about the transport, not about every engine channel.

type FileCertIssuer 

type FileCertIssuer struct {
	// contains filtered or unexported fields
}

FileCertIssuer implements CertIssuer with the local control-plane CA `palai init` writes into the .palai layout. It signs an enrolling (or renewing) runner's public key into a short-lived client certificate — the file-backed counterpart of the in-test CA the gateway conformance proof drives.

func NewFileCertIssuer 

func NewFileCertIssuer(certPath, keyPath string, ttl time.Duration) (*FileCertIssuer, error)

NewFileCertIssuer loads the PEM CA certificate and its EC private key (PKCS#8) from the .palai CA files. A non-positive ttl uses runnerCertTTL; the fault-live renewal proof injects a short TTL so a long-lived runner's rollover is provable in seconds.

func (*FileCertIssuer) SignRunnerCertificate 

func (i *FileCertIssuer) SignRunnerCertificate(publicKeyDER []byte, runnerDNS string) ([]byte, error)

SignRunnerCertificate signs the runner's public key into a short-lived client certificate under runnerDNS, usable only for client authentication.

func (*FileCertIssuer) TTL 

func (i *FileCertIssuer) TTL() time.Duration

TTL is the effective issued-certificate lifetime (the configured value, or runnerCertTTL when none was configured) — the interval FileEnrollmentTokens rate-limits a token's redemption by.

type FileEnrollmentTokens 

type FileEnrollmentTokens struct {
	// contains filtered or unexported fields
}

FileEnrollmentTokens implements EnrollmentTokens against the one-line token file `palai local up` mints fresh on every boot. Consume re-reads the file each call so a re-up that rotated the token is honored.

WHY THIS IS NOT ONE-USE, AND WHAT REPLACED THAT. Renewal runs over mTLS with the certificate that is expiring, so it serves only a runner still inside its validity window. A runner that missed the window — a sleeping laptop, a stalled Docker Desktop, a loaded host — holds a dead identity that can neither renew nor connect, and a strictly one-use token left it no way back: the stack bricked itself until an operator ran `palai local down && palai local up`, which nothing told them. So the file token is a BOOTSTRAP CREDENTIAL the runner may re-present when it holds no valid identity, not a one-shot.

THREAT MODEL. The token grants exactly one capability: mint one runner client certificate. It lives in .palai/runner-token on the host, mounted read-only into the control-plane and the runner. Anyone who can read it therefore reads either (a) the host's .palai directory, which also holds ca/ca.key — the CA private key that signs runner certificates outright, a strictly greater capability than the token's — or (b) the runner container's filesystem, which mounts /var/run/docker.sock and is therefore already host root. Against both adversaries a replayable token grants nothing that was not already theirs, which is the bar: an attacker who cannot read the runner's own filesystem gains nothing. What one-use WAS load-bearing against is a token that leaks WITHOUT that access — copied into a shell history, a support bundle, a `docker inspect` — and minInterval is what stands in its place: at most one certificate is minted per token per issued-certificate lifetime.

Be honest about what minInterval is: a RATE LIMIT, not an exclusion. A healthy runner rolls its identity forward over /v1/runner/renew, which never touches the token, so the interval elapses while that runner is alive and a token holder could mint a second, concurrent identity. It bounds replay VOLUME (no fleet from one leaked token) and it makes a runner restart wait out one certificate lifetime rather than re-minting at will.

AND IT LIVES IN MEMORY, which E24 T3 measured rather than assumed (§3.6 D6): lastIssued is process state, so a control-plane restart resets the counter, and `restart: always` plus a host reboot does that on a schedule. The registry this comment used to name as the missing upgrade path now EXISTS (E24 T1, internal/fleet) and the durable counterpart of this timestamp is runner_pool_keys.last_used_at — but it is deliberately NOT a rate limit there, because a pool key is a fleet credential presented by many machines and one-redemption-per-lifetime would make a ten-machine pool unenrollable. So the honest statement is: nothing anywhere caps the number of concurrent identities one credential can mint. See storage/queries/runners.sql (TouchRunnerPoolKey) and internal/fleet/keys.go.

func NewFileEnrollmentTokens 

func NewFileEnrollmentTokens(path string, minInterval time.Duration) *FileEnrollmentTokens

NewFileEnrollmentTokens binds the token file path; the file is read at Consume time. minInterval is the shortest gap between two redemptions of the same token — pass the issued certificate TTL, so a token mints at most one certificate per certificate lifetime. Zero or negative disables the rate limit (every redemption is admitted).

func (*FileEnrollmentTokens) Consume 

func (t *FileEnrollmentTokens) Consume(token string) error

Consume redeems the current file token. It returns an error for an empty token, one that does not match the current file, or one redeemed less than minInterval ago.

func (*FileEnrollmentTokens) Path 

func (t *FileEnrollmentTokens) Path() string

Path is the bound token file, so a caller can name it in an operator-facing message.

type GenesisSpine 

type GenesisSpine interface {
	PublishGenesis(ctx context.Context, tenant coordinator.Tenant, workspaceID string, in coordinator.ManifestInput) error
}

GenesisSpine is the durable write a genesis publish makes.

type HookFirer 

type HookFirer interface {
	Fire(ctx context.Context, ev extensions.HookEvent) (extensions.HookOutcome, error)
}

HookFirer runs a run's registered hooks at a dispatch point and returns the verdict (spec §28.17, E12 T8). *extensions.Store satisfies it; a test fakes it. The orchestrator depends only on this narrow seam so it stays free of the registry's DB + transport mechanics.

type IdleReleaser 

type IdleReleaser struct {
	// contains filtered or unexported fields
}

IdleReleaser hands a machine back when a session stops using it, WITHOUT ending the session.

THE SESSION IS THE THREAD'S IDENTITY AND IT DOES NOT CLOSE. What closes is the physical claim on a machine: the allocation. A workspace idle for the TTL is archived (bytes and .git, so uncommitted work survives — SAN-005) and its host directory reclaimed; the logical workspace id, the session, and the conversation are untouched. The next message in that thread reaches planRootWorkspace, which finds the workspace `paused` and restores it onto a NEW allocation — possibly on a different machine, because the bytes live in the object store and nothing about the restore names the machine that captured them.

WHY NOT close_session, WHICH IS THE OTHER THING THAT COULD HAVE BEEN WIRED HERE. Measured 2026-08-05: 281 of 292 sessions on this stack were `active`, the oldest for days, and nothing in the tree sends close_session at all. But closing them would have been the wrong repair twice over. First, it frees almost nothing: applyCloseSessionTx touches sessions, commands, events and session_sequences and not one workspace table — the directory and the uid would still leak. Second, `closed` is a lifecycle exit, so the next message in that thread would be refused rather than resumed, which is the opposite of what an idle thread needs. The occupancy is the allocation; the identity is the session; only the first is reclaimable while the conversation is still alive.

It is the retention Reaper's sibling by construction (retention.go): a durable maintenance job on the coordinator running one bounded, tenant-safe pass per tick, whose errors are logged and retried on the next tick rather than being fatal.

WHY THIS IS NOT DestroyAllocation. That path drives ready→destroying→destroyed, and `destroyed` is terminal in the WorkspaceTable — a workspace that reached it can never come back, which is right for a workspace nobody will ask for again and wrong for every one of these. The releaser uses the pause/restore branch instead, which is the branch that was built for exactly this and had no driver.

func NewIdleReleaser 

func NewIdleReleaser(spine *coordinator.Store, snapshots *SnapshotSink, ttl time.Duration) *IdleReleaser

NewIdleReleaser binds the durable spine, the snapshot capture path, and the idle TTL.

THE SNAPSHOT SINK IS NOT OPTIONAL, unlike the Reaper's artifact deleter. Releasing a workspace whose bytes were never archived is not a weaker release, it is data loss: the directory holding a session's uncommitted work would be deleted with nothing to restore from. A composition root with no object store must not construct this at all — main.go's startIdleRelease says so and refuses — and a nil sink here makes every sweep refuse rather than delete.

func (*IdleReleaser) Nudge 

func (r *IdleReleaser) Nudge()

func (*IdleReleaser) Run 

func (r *IdleReleaser) Run(ctx context.Context, interval time.Duration) error

Run sweeps every interval until ctx is cancelled, exactly as the retention Reaper does. A sweep error is logged and non-fatal: a transient database blip must not stop machines being handed back, and the next tick retries from the durable state rather than from anything this goroutine remembers.

func (*IdleReleaser) SetTeardown 

func (r *IdleReleaser) SetTeardown(remove func(string) error)

SetTeardown overrides how an allocation's host directory is reclaimed (default os.RemoveAll). Same seam as WorkspaceRecovery.SetTeardown: a remote tier tears down over the wire, and a test drives the failure path through it.

func (*IdleReleaser) Sweep 

func (r *IdleReleaser) Sweep(ctx context.Context) (released int, err error)

Sweep runs one idle pass and returns the number of workspaces released. A failure on one candidate is logged and the pass continues to the next: one workspace whose archive will not upload must not stop every other machine from being handed back. The first error is returned so a caller can surface that the pass was not clean; a repeat offender (e.g. an allocation over MaxSnapshotArchiveBytes) logs and is retried every tick, so a SWEEP-LEVEL summary is what tells an operator "2 stuck" apart from "2 of 32 stuck" — the per-candidate lines above name them but never say how many candidates there were.

type ImageReader 

type ImageReader interface {
	ReadImageArtifact(ctx context.Context, project, artifactID string) (mediaType string, content []byte, found bool, err error)
}

ImageReader is the object-store read path an image content item resolves through. It is a separate one-method seam from ArtifactWriter on purpose: the write side has its own implementors (including test doubles), and widening it would have made every one of them carry a read they do not do.

type InitInput 

type InitInput struct {
	Tenant      coordinator.Tenant
	WorkspaceID string
	BindingID   string
	Ref         string
}

InitInput is one workspace to bring into existence.

type InstructionLayer 

type InstructionLayer struct {
	Layer string
	Text  string
}

InstructionLayer is one resolved layer of the §25.12 stack: the text, and the layer that supplied it. The provenance is carried rather than concatenated away because §25.12 requires instruction layers to be "delimited and provenance-tagged", and because a caller debugging "why did it answer like that" needs to know whether the agent or their own request said so.

type LeaseReclaimer 

type LeaseReclaimer struct {
	// contains filtered or unexported fields
}

LeaseReclaimer returns a workspace whose holder run reached terminal from `leased` back to `ready`.

WHY IT HAS TO EXIST, WHICH IS NOT OBVIOUS BECAUSE A RECLAIM ALREADY EXISTED. acquireWriterLease has reclaimed dead leases since E09 T10, and it is correct — but it runs from exactly one place, on a lease CONFLICT, which means it fires only when a SECOND attempt arrives on the same allocation. A thread whose last run ended and which nobody writes to again never produces that second attempt. Its lease stays active, its workspace stays `leased`, and IdleWorkspacesForRelease selects `state = 'ready'` — so the idle releaser, the one thing that hands a machine back, can never see it. Nothing is broken and nothing is logged; the directory and the uid slot are simply held until somebody types into that thread again.

Measured on the running stack 2026-08-05, after the release defer was repaired: 19 active leases and 19 `leased` workspaces, ZERO of them with a live run — 16 completed, 2 canceled, 1 failed — holding 197MB across 18 allocation directories, the oldest abandoned for 9h43m.

THIS IS STAGE ONE OF TWO AND THAT SPLIT IS THE SAFETY ARGUMENT. The obvious repair is to widen the idle sweep to accept `leased`, and it is the wrong one: that sweep ARCHIVES AND DELETES a directory, so pointing it at a live lease would destroy work a running attempt is in the middle of. This stage does the cheap, reversible half — release the lease, leased→ready, not one byte touched — and then the EXISTING idle sweep picks the workspace up on its own terms (ready, no active lease, idle for the TTL) and does the half that deletes. workspaces.sql already says the split out loud: the idle candidate query documents skipping a dangling lease as "the stuck-lease reclaim's business, not this sweep's". That reclaim is this.

It is the retention Reaper's and the IdleReleaser's sibling by construction: a durable maintenance job running one bounded, tenant-safe pass per tick, whose errors are logged and retried on the next tick rather than being fatal.

func NewLeaseReclaimer 

func NewLeaseReclaimer(spine *coordinator.Store, grace time.Duration) *LeaseReclaimer

NewLeaseReclaimer binds the durable spine and the post-terminal grace.

func (*LeaseReclaimer) Run 

func (r *LeaseReclaimer) Run(ctx context.Context, interval time.Duration) error

Run drives one sweep per interval until ctx ends, in the shape the Supervisor restarts.

func (*LeaseReclaimer) Sweep 

func (r *LeaseReclaimer) Sweep(ctx context.Context) (reclaimed int, err error)

Sweep runs one reclaim pass and returns the number of workspaces returned to ready. A failure on one candidate is logged and the pass continues: one workspace whose transition loses a race must not keep every other machine held. The first error is returned so a caller can surface that the pass was not clean.

type MCPSamplingRouter 

type MCPSamplingRouter struct {
	// contains filtered or unexported fields
}

MCPSamplingRouter is the control-plane sampling seam (E12 T6, TOL-010). When an MCP connection enables sampling, a server sampling/createMessage is routed as a SEPARATE brokered, budgeted model step: a Route through packages/model-broker under its OWN Reservation, journalled as model_step.created/completed.v1 events tagged source:"mcp_sampling" (no new engine frame or event kind — §61; the sampling call is entirely control-plane-side, never crossing the engine wire). The provider credential is the platform's OWN model credential (route.Secret), redeemed control-plane-side by the broker — the MCP server never sees it. The call is METERED and CUT OFF at Admit when it runs past the connection's budget, so an enabled server can never drive an unbounded model spend from one tools/call.

func NewMCPSamplingRouter 

func NewMCPSamplingRouter(broker *modelbroker.Broker, route ModelRoute, emit func(ctx context.Context, scope mcp.CallScope, eventType string, payload []byte) error) *MCPSamplingRouter

NewMCPSamplingRouter wires the router. emit journals one model_step event (best-effort — a journal failure must never fail the sampling call; the budget is enforced by the broker regardless). A nil emit is inert.

func (*MCPSamplingRouter) RouteSampling 

func (r *MCPSamplingRouter) RouteSampling(ctx context.Context, scope mcp.CallScope, conn mcp.ConnConfig, params json.RawMessage) (json.RawMessage, error)

RouteSampling implements mcp.SamplingRouter: decode → Route (separate Reservation) → journal + return the MCP sampling result, or a denial (the gate turns the returned error into a JSON-RPC error to the server).

type MachineCaller 

type MachineCaller interface {
	CallMachine(ctx context.Context, machineID, verb string, payload map[string]any) (map[string]any, error)
}

MachineCaller reaches a named machine with no lease in hand. *RunnerGateway satisfies it; a deployment without one leaves it nil and every background call is refused.

type MachineIdentified 

type MachineIdentified interface {
	MachineID() string
}

MachineIdentified is the sibling of ExecConn: a channel that can say WHICH machine it reached. An attempt learns its machine the same way it learns its executor — from the connection it holds — so a background task started during that attempt records a machine the reconciler can address later.

type MachineLoadView 

type MachineLoadView interface {
	// PoolMachineLoads reports every machine in the pool with its open holds and the ceiling it declared.
	// A machine it does not answer for is one the caller knows nothing about — see emptiestFirst for what
	// the ranking does with that, which is deliberately NOT "assume the worst".
	PoolMachineLoads(ctx context.Context, tenant coordinator.Tenant, poolID string) ([]coordinator.PoolMachineLoad, error)
}

MachineLoadView weighs one pool's machines for a placement. *coordinator.Store is the only implementation; the interface exists so the gateway does not depend on the database, exactly as it does not depend on it for the registry or for the capacity wake.

type MaterializeReport 

type MaterializeReport struct {
	Found        bool   `json:"found"`
	Fetched      int    `json:"fetched"`
	Skipped      int    `json:"skipped"`
	Entries      int    `json:"entries"`
	TreeChecksum string `json:"tree_checksum"`
}

MaterializeReport is what a machine did when asked to write the head into its allocation.

type MergeChildBranchInput 

type MergeChildBranchInput struct {
	MergeID          string
	ParentRunID      string
	SourceChildRunID string
	RepoDir          string
	ChildBranch      string
}

MergeChildBranchInput is the infrastructure-owned input to an explicit child-branch merge. RepoDir is the parent worktree; ChildBranch is the child's agent/<session>/<run> branch (spec §30.5).

type MergeStore 

type MergeStore interface {
	RecordMerge(ctx context.Context, tenant coordinator.Tenant, in coordinator.MergeRecordInput) error
}

MergeStore records merge outcomes (the coordinator seam; *coordinator.Store in prod, a fake in the unit test — the ReconcileStore idiom).

type ModelRoute 

type ModelRoute struct {
	Provider string
	Model    string
	Secret   modelbroker.SecretRef
	// BaseURL is the CONNECTION's endpoint (E29, migration 000049), empty meaning the family's own. It is
	// the seam that makes a custom OpenAI-compatible provider a PER-PROJECT property: before it the only
	// way to move that endpoint was PALAI_OPENAI_COMPATIBLE_BASE_URL, read once at boot into a single
	// adapter value, so one deployment had exactly one custom endpoint no matter how many projects it ran.
	BaseURL string
	// RevisionID / Revision are set ONLY when the route was resolved from the project's DB route. An
	// empty RevisionID means this is the deployment default, which pins no revision.
	RevisionID string
	Revision   int
	// Thinking is whether runs on this route ask the provider for the model's reasoning. Empty on the
	// deployment default and on every revision that did not pin it, which is what keeps such a run's
	// provider request byte-identical to the pre-reasoning one. See coordinator.ModelRouteTarget.Thinking
	// for why the decision belongs to the revision and not to a deployment env var or a per-request flag.
	Thinking modelbroker.ThinkingMode
}

ModelRoute is the broker coordinates this kernel routes a model.request to: the adapter name, the model id put on the provider wire, and the SecretRef the executor redeems at call time.

It has two provenances. As the Orchestrator's `route` field it is the DEPLOYMENT DEFAULT the composition root selects from the environment (PALAI_MODEL_PROVIDER/PALAI_MODEL), defaulting to the deterministic fake provider every existing suite registers under. As the value effectiveRoute returns it may instead be the project's DB-backed route (E13 T8) — in which case RevisionID/Revision name the model_route_revision that selected it and Secret is a tenant-qualified handle.

type Orchestrator 

type Orchestrator struct {

	// DialHandshakeDeadline bounds the dial + engine.ready handshake per attempt. Zero uses
	// dialHandshakeDeadline; NewOrchestrator sets the default. Tests shorten it.
	DialHandshakeDeadline time.Duration
	// contains filtered or unexported fields
}

Orchestrator executes response run attempts through the common kernel.

func NewOrchestrator 

func NewOrchestrator(st *store.Store, dialer EngineDialer, models *modelbroker.Broker, tools *toolbroker.Broker) *Orchestrator

NewOrchestrator binds the durable store, the engine dialer, and the model and tool brokers into one kernel. The model route defaults to the deterministic fake provider; main.go overrides it for a live provider via SetModelRoute.

func (*Orchestrator) BackgroundKiller 

func (o *Orchestrator) BackgroundKiller() coordinator.BackgroundKiller

BackgroundKiller is what this orchestrator hands the coordinator so that CANCELLING A RUN ENDS ITS LIVE WORK (E26 T5, §3.6 D10). It is the kill half of the same seam BackgroundObserver is the probe half of: the coordinator owns the transaction and the state vocabulary, this package owns the signal.

It reports the STATE the kill leaves the row in rather than an error to be classified, because a handle that cannot be proven to be ours is not a failure — it is the answer `lost`, and the rule built on it is absolute: it receives no signal, on a deadline, on a cancellation or on a sweep.

func (*Orchestrator) BackgroundObserver 

func (o *Orchestrator) BackgroundObserver() coordinator.BackgroundObserver

BackgroundObserver is what this orchestrator hands the reconciler: the function that answers, for one background_tasks row, what the OPERATING SYSTEM says about its handle and what its output ends with (E26 T4). Nil when no background runner is wired, which makes the sweep a no-op — the honest reading of a deployment that cannot start a background task in the first place.

It is a method value rather than a captured closure so that a composition-root test can ask an orchestrator what production would hand over, the same reason backgroundRunnerFor is a named function in main.go.

func (*Orchestrator) ExecuteAttempt 

func (o *Orchestrator) ExecuteAttempt(ctx context.Context, attempt AttemptDescriptor) error

ExecuteAttempt drives one run attempt to a terminal outcome. It provisions and starts the run through canonical transitions, opens the engine channel, and runs the frame-intake loop: every frame is validated and deduped before any dispatch, and every provider/tool result is committed before it is delivered to the engine.

func (*Orchestrator) SetBackgroundMachine 

func (o *Orchestrator) SetBackgroundMachine(id string)

SetBackgroundMachine names the machine this process's background runner reaches (A.3 T6). It is recorded on every task started here and compared before any later probe or signal.

IT IS SEPARATE FROM SetBackgroundRunner BECAUSE THE TWO ANSWER DIFFERENT QUESTIONS, and folding them would make the identity look like a property of the executor. It is a property of the HOST: a control plane that restarts, or is replaced by a second process on the same box, reaches the same kernel and must still settle what it started — which is the crash-restart property the wake suite proves. Left unset, every task this plane starts is written with an unknown machine and is therefore never probed locally again, which is the same conservative reading rows predating the column get.

func (*Orchestrator) SetBackgroundRunner 

func (o *Orchestrator) SetBackgroundRunner(b toolbroker.BackgroundRunner)

SetBackgroundRunner injects the detached shell runner a background task is started through (E26 T1). Left unset — which is every deployment before E26 and every posture that cannot detach — the dispatch is bit-unchanged: the same SetHookFirer/SetPublisher discipline, where an unwired seam means the feature is simply absent rather than half-present.

IT ALSO WIRES THE CANCELLATION KILLER, IN THE SAME CALL AND DELIBERATELY (E26 T5). A deployment that can START a background task and cannot END one when its run is cancelled is precisely the orphan this epic exists to prevent — and E26 T2 already found one instance of that shape by omission, where every deployment granting the shell tool could begin a build and none could stop it because the kill tool sat on a different conditional. Two setters would be two chances to wire one and forget the other; one setter makes the broken state unrepresentable.

func (*Orchestrator) SetChangesetWriter 

func (o *Orchestrator) SetChangesetWriter(aw ArtifactWriter)

SetChangesetWriter injects the object-store write-path the finalize changeset compile persists the patch + test-log through (spec §30.6). Left unset, a terminated coding run compiles no changeset — the same discipline as SetPublisher.

func (*Orchestrator) SetCheckpointSink 

func (o *Orchestrator) SetCheckpointSink(cs *CheckpointSink)

SetCheckpointSink injects the checkpoint persistence path (spec §26.1-26.2). Left unset, a checkpoint.offer is dropped (no durable boundary) — the same discipline as SetChangesetWriter.

func (*Orchestrator) SetConnectionSecrets 

func (o *Orchestrator) SetConnectionSecrets(secrets SecretResolver)

SetConnectionSecrets wires the resolver a binding's connection_ref is redeemed through (E13 Task 9), so a tenant's own Git credential — provisioned and rotated over the secret-ref API — backs the clone for the bindings that name one. main.go calls it unconditionally next to SetWorkspaceProvisioner: any composition root that provisions workspaces MUST wire it, because a ref-bearing binding fails closed without it rather than quietly borrowing the deployment-global credential.

UPGRADING FROM PRE-T9: connection_ref had no reader, so a ref-bearing binding cloned with the global GitHub App credential. It now has to resolve — the ref must exist under the binding's organization and the deployment must have a secret master key — or the clone fails. A binding that really does want the deployment credential carries an EMPTY connection_ref.

func (*Orchestrator) SetEnvironmentSecrets 

func (o *Orchestrator) SetEnvironmentSecrets(secrets SecretResolver)

SetEnvironmentSecrets wires the resolver an environment VALUE is read through (E25 T3). Nil means no environment value can be resolved, and an attempt whose revision names an environment then FAILS its tool call rather than running without the credential — the fail-closed direction, for the reason repositoryConnectionSecret gives: a `gh` or `curl` that runs without the token it was supposed to have succeeds anonymously often enough to look like a working run.

main.go wires it from the same DB-backed store the four other secret resolvers front.

func (*Orchestrator) SetHookFirer 

func (o *Orchestrator) SetHookFirer(h HookFirer)

SetHookFirer injects the hook dispatcher the five pinned points fire through (spec §28.17, E12 T8). Left unset, no hook fires — the dispatch is bit-unchanged (the same discipline as SetBackgroundRunner/SetPublisher). It also propagates the firer to the publication registry, so the before_repository_publish point fires from inside the publish tool's RequestPublication.

func (*Orchestrator) SetIdleNudge 

func (o *Orchestrator) SetIdleNudge(nudge func())

SetIdleNudge wires the hint the orchestrator gives the idle sweep when an attempt finishes with a workspace. Nil is a deployment with no releaser — the sweep's own ticker is then the only trigger, which is the behaviour every stack had before this existed.

func (*Orchestrator) SetImageReader 

func (o *Orchestrator) SetImageReader(ir ImageReader)

SetImageReader injects the read side of the object store, which is what lets a run SEE an image: an `image_ref` content item in the run's input names an artifact, and the bytes are joined to the provider request here in the control plane (spec §24 — the object-store credential never reaches the engine, and the 1 MiB engine frame could not carry a screenshot regardless).

Left unset, every image_ref resolves as a miss and the turn carries the "no longer available" marker — which is the truth for a stack with no object store, and leaves a text-only run bit-unchanged.

func (*Orchestrator) SetMachineCaller 

func (o *Orchestrator) SetMachineCaller(m MachineCaller)

SetMachineCaller wires the surface background execution travels on (A.3 T7). It is separate from the EngineDialer even though the gateway satisfies both, because the two answer different questions: a dialer hands out WHICHEVER machine a pool has free for an attempt, and this one addresses the machine a task ALREADY runs on — by name, long after that attempt ended.

func (*Orchestrator) SetModelRoute 

func (o *Orchestrator) SetModelRoute(r ModelRoute)

SetModelRoute sets the DEPLOYMENT-DEFAULT provider/model/secret the composition root (main.go) selects from the environment. Since E13 T8 it is the FALLBACK layer: a project with a published model route dispatches through that route instead (effectiveRoute), and a project without one runs on this.

func (*Orchestrator) SetPublisher 

func (o *Orchestrator) SetPublisher(p Publisher)

SetPublisher injects the repository publisher the approval pump publishes through. Left unset, an approved publication simply waits (the pump is a no-op) — no push happens without a wired publisher.

func (*Orchestrator) SetQueueDeadline 

func (o *Orchestrator) SetQueueDeadline(d time.Duration)

SetQueueDeadline sets the §20.12 admission queue deadline (see the field doc). Left unset (zero), a run is never timed out for queue age — the pre-E13-T7 behaviour, so every existing tier is bit-unchanged. main.go injects it from PALAI_QUEUE_DEADLINE.

func (*Orchestrator) SetReconstructionForbidden 

func (o *Orchestrator) SetReconstructionForbidden(forbidden bool)

SetReconstructionForbidden sets the §26.3 policy: when true, an incompatible checkpoint fails the run explicitly rather than reconstructing from the transcript (spec §26.3 rung 4).

func (*Orchestrator) SetRemoteChildren 

func (o *Orchestrator) SetRemoteChildren(agents RemoteAgents, runner RemoteChildRunner)

SetRemoteChildren wires remote child-run dispatch (E19 T5, §38.5): the registered-agent lookup and the A2A client that dials it. Left unset (either nil), a delegation naming a remote agent is DENIED rather than dispatched — the same fail-closed discipline as SetPublisher, except the fallback here would be a SECURITY substitution (our engine, our credentials) rather than a no-op, so it is refused explicitly.

func (*Orchestrator) SetSnapshotSink 

func (o *Orchestrator) SetSnapshotSink(ss *SnapshotSink)

SetSnapshotSink injects the workspace snapshot capture/restore path (spec §29.10, SES-009). Left unset, no boundary snapshot is cut at a pause — the checkpoint declares no workspace dependency, the T4 behaviour. Wired alongside SetCheckpointSink where an object store is configured.

func (*Orchestrator) SetStoreGenesis 

func (o *Orchestrator) SetStoreGenesis(p *StoreProvisioner)

SetStoreGenesis binds the genesis publish: a fresh workspace's first tree goes into the object store BEFORE any machine holds it, so the workspace exists independently of a host and "give me the current tree" is answerable at any moment, including while nobody holds it.

‼️ IT IS ADDITIVE TODAY AND THE ON-MACHINE CLONE STILL RUNS. The inversion's other half — the machine materialising the head INSTEAD of cloning, which is what removes the second clone — is not taken here, because this tree's rule is to verify the new path before destroying the old one. So for now the same repository is fetched twice for a fresh allocation, and that cost is real and is not being hidden: it buys a head that provably exists before a machine does, which is the thing the next step needs.

Nil leaves provisioning exactly as it was.

func (*Orchestrator) SetWorkspaceProvisioner 

func (o *Orchestrator) SetWorkspaceProvisioner(root string, broker repositories.Broker)

SetWorkspaceProvisioner wires the root run's workspace auto-provisioning (spec §29.7-30.3, E09 Task 10): root is the host directory allocations are minted under (PALAI_WORKSPACE_ROOT), and broker mints the short-lived read credential the clone runs behind. Left unset, a run with an attached binding simply gets no workspace — its coding tools then fail cleanly (no host path), the SetBackgroundRunner discipline. main.go wires it env-gated where a sandbox is configured.

type PoolEnrollment 

type PoolEnrollment interface {
	EnrollmentTokens
	// RedeemPoolKey resolves a presented value into the grant it authorises. runnerID is the
	// server-minted id a refusal is journalled against; declaredPool is what the MACHINE said it
	// belongs to and never widens the grant. fleet.ErrUnknownPoolKey — and only that error — means "not
	// mine", so the gateway may fall through to the file token; every other error is a recognised
	// credential that was refused.
	RedeemPoolKey(ctx context.Context, presented, runnerID, declaredPool string) (fleet.PoolGrant, error)
}

PoolEnrollment is the pool-scoped enrolment credential (E24 T3): a credential that names the ONE pool it admits into, can expire, can be revoked, and is RECORDED against the certificate it minted.

It EMBEDS EnrollmentTokens rather than replacing it, and that is the substitutability requirement enforced by the compiler instead of promised in prose: a pool-key implementation has to be usable anywhere the file token is. The gateway nonetheless calls RedeemPoolKey and not Consume, because the pool binding is the entire point and Consume discards it.

type PoolSettings 

type PoolSettings interface {
	DesiredSettingsForPool(ctx context.Context, poolID string) (map[string]string, error)
	// DesiredSettingsForMachine is the pool's document with this machine's own laid over it, plus the
	// revision that produced the pair — a CHANGE DETECTOR the machine compares against what it is running.
	DesiredSettingsForMachine(ctx context.Context, poolID, runnerID string) (map[string]string, int64, error)
	// RecordRunnerConfigReport stores the machine's verdict per setting. matched=false with a nil error is a
	// real state rather than a fault — a machine that enrolled before the registry existed has no row and
	// never will — so it is a return value rather than an error the caller has to pick apart.
	RecordRunnerConfigReport(ctx context.Context, dns string, revision int64, applied map[string]string, at time.Time) (matched bool, err error)
}

PoolSettings answers one pool's desired configuration — what the admin plane decided this machine should be, which is the half a machine cannot know about itself.

The poolID it is asked about comes from the RESOLVED GRANT and never from the enrolment request, so a machine cannot read another pool's document by declaring that pool. THREE METHODS, TWO MOMENTS. DesiredSettingsForPool serves ENROLMENT, where the machine has no identity yet and the pool comes from the resolved grant. The other two serve the SETTINGS POLL, where the machine has a certificate and is therefore addressable individually — so the poll can overlay a machine document on the pool's, and can record what the machine says it did with the result.

type PrepareRepositoryInput 

type PrepareRepositoryInput struct {
	BindingID    string
	RunID        string
	RequestedRef string // empty falls back to the binding's default branch
	WorkBranch   string // the generated agent/<...> branch; empty = detached read-only
	TargetDir    string
	SecretsDir   string // the snapshot-excluded /secrets area for the credential helper (§29.10)
	AttemptFence uint64 // binds the minted read credential to this attempt (§28.11)
	ToolCall     string
	// ConnectionSecrets resolves a binding's connection_ref to its Git credential (E13 Task 9). It is a
	// DEPLOYMENT capability, not run data, and the composition root ALWAYS wires it alongside the workspace
	// provisioner — nil is a test/no-provisioning stack, and there a ref-bearing binding fails closed
	// rather than borrowing the global credential (see bindingBroker). A ref-less binding never consults it.
	//
	// UPGRADING FROM PRE-T9: a ref-bearing binding used to clone with the deployment-global GitHub App
	// credential, because connection_ref had no reader. It now needs its ref actually provisioned — over
	// POST /v1/secret-refs, under the binding's own organization, on a deployment that configured
	// PALAI_SECRET_MASTER_KEY_FILE. Without that the clone FAILS (fail-closed, deliberate); a binding that
	// genuinely wants the deployment credential must carry an EMPTY connection_ref.
	ConnectionSecrets SecretResolver
}

PrepareRepositoryInput is the infrastructure-owned input to a run's repository-preparation step (spec §30.3). It comes from the resolved binding and the run, never from model output, so the recorded provenance does not depend on model behavior (REP-001). TargetDir is the READY workspace's repo dir — there is no clone before the workspace is ready.

type PreparedRepository 

type PreparedRepository struct {
	Receipt  contracts.PreparationReceipt
	Findings []repositories.Finding
}

PreparedRepository is the outcome of the step: the recorded model-independent receipt plus any untrusted-repo containment findings (§30.4).

func PrepareRepository 

func PrepareRepository(ctx context.Context, store RepositoryStore, broker repositories.Broker, tenant coordinator.Tenant, in PrepareRepositoryInput) (PreparedRepository, error)

PrepareRepository is the run-start repository-preparation step (spec §30.3): once the workspace is READY, it resolves the run's repository binding, runs the infrastructure-owned deterministic preparation under a brokered short-lived read credential, and records the model-independent receipt. The model never sees the credential (§30.2); the broker revokes it after the fetch.

The root run's auto-provisioning (Orchestrator.provisionFreshAllocation, E09 Task 10) is the production caller: it drives the workspace to preparing and calls this to clone @ the attached ref. The coding journey (T9) and the live smoke drive the same composed step; nothing waits on it now.

type PreviewGrant 

type PreviewGrant struct {
	Route     string
	Tenant    coordinator.Tenant
	SessionID string
	RunID     string
	Target    string
	Protocols []string
	ExpiresAt time.Time
}

PreviewGrant authorizes one inbound connection to a sandbox preview/terminal endpoint through the reverse proxy (spec §29.16, SAN-010). Route is a random, non-guessable token — the only handle the caller ever holds; Target is the internal sandbox address the proxy dials and NEVER discloses. The grant binds the tenant, session, and run so a connection is authorized on every request and audited against its lineage. ExpiresAt is a short expiry after which the route is dead. Protocols is the connection allowlist (http today; ws is the terminal path, spec §29.17).

type PreviewProxy 

type PreviewProxy struct {
	// contains filtered or unexported fields
}

PreviewProxy is the authenticated reverse proxy for inbound sandbox connectivity (spec §29.16). It never exposes a direct pod/container address: a caller reaches a sandbox only by a random route, authorized against the grant's tenant and expiry on every connection, and a denial carries no sandbox address. It is the SAN-010 enforcement point.

func NewPreviewProxy 

func NewPreviewProxy(now func() time.Time) *PreviewProxy

NewPreviewProxy builds an empty proxy that stamps expiry checks with now (tests inject a clock).

func (*PreviewProxy) Grant 

func (p *PreviewProxy) Grant(g PreviewGrant) PreviewGrant

Grant registers a preview grant, minting a random non-guessable route when the caller left one empty, and returns the stored grant (its Route is the caller-facing handle). The Target is kept server-side and never returned in any response.

func (*PreviewProxy) ServeHTTP 

func (p *PreviewProxy) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP authorizes the caller against the routed grant and, only on success, reverse-proxies to the sandbox. Every failure path returns a generic denial that names no sandbox address, and a wrong-tenant caller is answered exactly like an unknown route so the proxy discloses no tenant's route existence (spec §29.16 no direct address exposure).

type PublicationPrechecker 

type PublicationPrechecker interface {
	CanPublish(connectionRef string) error
}

PublicationPrechecker is a Publisher that can answer CanPublish. It is a SEPARATE interface from Publisher rather than a second method on it, because the seam it serves is a different one: Publisher is the pump's, and this is the publication REGISTRY's — the tool that records a pending approval, which runs long before any pump. A Publisher that cannot answer (a test fake) leaves the precheck inert, which is the same fail-open-to-today discipline every other optional seam in this file uses.

type PublicationPump 

type PublicationPump interface {
	ApprovedPublicationsForRun(ctx context.Context, tenant coordinator.Tenant, runID string) ([]coordinator.Publication, error)
	MarkPublicationPublished(ctx context.Context, tenant coordinator.Tenant, sessionID, responseID, publicationID, operation string, receipt map[string]any) error
	RecordPublicationWarning(ctx context.Context, tenant coordinator.Tenant, sessionID, responseID, publicationID, detail string) error
	// ExpireApprovalIfElapsed is the consume-time expiry guard (spec §22.4, E10 T7): before a publish, an
	// approved publication whose one-shot approval elapsed is expired + journaled and reported true, so
	// the pump skips it. A live approval reports false and publishes unchanged.
	ExpireApprovalIfElapsed(ctx context.Context, tenant coordinator.Tenant, sessionID, responseID, publicationID string) (bool, error)
}

PublicationPump is the store seam the approval pump reads approved publications from and records receipts / warnings through (the ReconcileStore idiom). *coordinator.Store implements it; a fake in the pump test proves the boundary-pump against a real bare remote without a database.

type PublishTarget 

type PublishTarget struct {
	Publication   coordinator.Publication
	WorkspaceRoot string // the attempt's workspace allocation root (the repo lives at WorkspaceRoot/repo)
	Project       string
	AttemptFence  uint64 // binds the minted push credential to this attempt (§28.11)
	// ConnectionRef is the binding's own credential handle, empty for a binding that takes the
	// deployment-global App. NEVER a token — the publisher resolves it server-side.
	ConnectionRef string
	// Identity is the binding's owner/repo, for the pull-request client.
	Identity string
}

PublishTarget is one approved publication plus the per-attempt context needed to execute it.

type Publisher 

type Publisher interface {
	Publish(ctx context.Context, target PublishTarget) (map[string]any, error)
}

Publisher executes one approved publication against the external repository and returns the receipt to record (spec §30.9-30.10). RepositoryPublisher is the real implementation (push via the broker, PR via the provider client); a fake proves the pump deterministically. A nil publisher disables the pump — a stack with no repository publication wired (every existing orchestrator test) simply skips it, the SetBackgroundRunner discipline.

type Reaper 

type Reaper struct {
	// contains filtered or unexported fields
}

Reaper periodically purges the content of store=false responses whose retention TTL has elapsed, leaving a tombstone (spec §8.3, §20.9). It is the retention sibling of the Reconciler's dead-letter sweep: a durable maintenance job on the coordinator that runs one bounded, tenant-safe pass per tick.

func NewReaper 

func NewReaper(store *store.Store, storeFalseTTL time.Duration) *Reaper

NewReaper binds the store:false retention TTL to the store.

func (*Reaper) Run 

func (r *Reaper) Run(ctx context.Context, interval time.Duration) error

Run sweeps every interval until ctx is cancelled. A sweep error is logged (the reaper's only failure surface) and non-fatal: the next tick retries the DB purge, because a transient database blip must not stop retention. A delete failure is not retried by the reaper — it orphans the scrubbed object's bytes (see Sweep) — but the artifact orphan-GC reclaims those bytes on a later reconcile, so the log is a trace, not a permanent leak.

func (*Reaper) Sweep 

func (r *Reaper) Sweep(ctx context.Context) (purged int, err error)

Sweep runs one retention pass and returns the number of responses purged. The DB scrub commits first with each victim's object_key cleared, so the keys it named are surfaced here and their bytes deleted from the object store afterward. A delete error is returned and logged by Run, but it is NOT retried here: the scrub already committed the key away, so a failed delete orphans that object exactly like the crash case below. That is no longer a lost byte — the artifact orphan-GC (artifacts.Collector, wired beside this reaper) reconciles the bucket against the artifacts rows and reclaims any object no live, non-empty object_key row references, so a failed delete self-heals on a later GC pass. The purge itself is durable — the rows are correctly tombstoned regardless.

func (*Reaper) WithArtifactStore 

func (r *Reaper) WithArtifactStore(a ArtifactDeleter) *Reaper

WithArtifactStore wires the object-store byte-deleter the purge uses to erase an expired run's artifact bytes (LP §7.2). Nil (the default) scrubs only the DB row, which keeps the deployments and tests that run without an object store working unchanged.

type ReconcileStore 

type ReconcileStore interface {
	ReclaimExpired(ctx context.Context, maxAttempts int) (int, error)
	SweepDeadLetteredRuns(ctx context.Context) (int, error)
	// SweepExpiredApprovals expires publications whose one-shot approval passed its expiry with no
	// consume observing it (the idle-expiry half; the consume-time guards catch the rest). Returns the
	// number expired this pass.
	SweepExpiredApprovals(ctx context.Context) (int, error)
	// SweepExpiredToolApprovals is the same sweep for a GATED TOOL CALL, and it carries the half that
	// publications never needed: it WAKES the parked run. A publication whose approval lapses leaves
	// nothing waiting — the run finished long before. A tool call's run is parked on the question, so an
	// expiry that only cancelled the call would leave that run waiting forever (E23 T1).
	SweepExpiredToolApprovals(ctx context.Context) (int, error)
	// SweepExpiredCapacityParks ends the runs parked for want of a MACHINE longer than the operator's TTL
	// (E24 T5). ttl <= 0 is a no-op and is the shipped default: there is no honest default duration for
	// "how long until a rented Mac arrives". The projection is the terminal Response body, passed in because
	// it lives in this package with the other terminal projections.
	SweepExpiredCapacityParks(ctx context.Context, ttl time.Duration, projection []byte) (int, error)
	// SweepFinishedBackgroundTasks turns a finished background PROCESS into the model's next turn
	// (E26 T4): each running task row is probed through the observer, and each one that has finished
	// produces EXACTLY ONE notification — enqueued as a background_notice command, and waking the run
	// only if it parked on the task. A nil observer makes it a no-op, which is the honest reading of a
	// deployment with no background runner wired.
	SweepFinishedBackgroundTasks(ctx context.Context, observe coordinator.BackgroundObserver) (int, error)
	// BackgroundLogRetention is the log garbage sweep's two inputs (E26 T5): the allocation roots that
	// may hold a background log, and the ids of the tasks still writing to one. The deletion itself is
	// this package's, because the coordinator owns transactions and not files.
	BackgroundLogRetention(ctx context.Context) (roots []string, live map[string]bool, err error)
}

ReconcileStore is the coordinator seam the reconciler sweeps through: it dead-letters abandoned jobs, drives dead-lettered response.run jobs to a failed run terminal, and expires approvals whose minutes-scale deadline elapsed while idle (spec §22.4, E10 T7). *coordinator.Store implements it; a fake implements it in unit tests.

type Reconciler 

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler periodically dead-letters jobs whose lease has lapsed and whose attempts are exhausted — the abandoned-work safety net for workers killed every attempt that never self-report a terminal failure (spec §24.4) — and then bridges each dead-lettered response.run to a failed run terminal so its response never hangs. Expired leases still under their attempt ceiling are reclaimed inline by the next claim, so the reconciler only enforces the dead-letter ceiling and the terminality bridge.

func NewReconciler 

func NewReconciler(store ReconcileStore, interval time.Duration, maxAttempts int) *Reconciler

NewReconciler binds a sweep interval and attempt ceiling to the store.

func (*Reconciler) Run 

func (r *Reconciler) Run(ctx context.Context) error

Run sweeps every interval until ctx is cancelled. A sweep error is non-fatal: the next tick retries, because a transient database blip must not stop the safety net.

func (*Reconciler) Sweep 

func (r *Reconciler) Sweep(ctx context.Context) (int, error)

Sweep runs one reconciliation pass: it dead-letters abandoned jobs, then drives every dead-lettered response.run to a failed run terminal so a run whose every attempt violated the protocol reaches terminal rather than hanging in running with an open SSE stream (spec §24.4 -> §22.3). It returns the number of jobs dead-lettered this pass.

func (*Reconciler) WithBackgroundTasks 

func (r *Reconciler) WithBackgroundTasks(observe coordinator.BackgroundObserver) *Reconciler

WithBackgroundTasks opts the deployment into the exit-notification sweep (E26 T4). A setter for the same two reasons WithCapacityParkTTL is one: every existing caller compiles unchanged, and the honest default is off — a control plane that cannot start a background task has none to notice finishing.

func (*Reconciler) WithCapacityParkTTL 

func (r *Reconciler) WithCapacityParkTTL(ttl time.Duration) *Reconciler

WithCapacityParkTTL opts the deployment into expiring capacity parks (PALAI_FLEET_PARK_TTL). A setter rather than a constructor parameter so every existing caller compiles unchanged, and because the honest default is "off" — see the field.

type RecoverInput 

type RecoverInput struct {
	WorkspaceID string
	RunID       string
	SessionID   string
	ResponseID  string
	SnapshotID  string
}

RecoverInput names the host-lost workspace and the run/session recovering it. SnapshotID is optional: empty resolves the workspace's latest byte-archived snapshot (the natural boundary), a set value pins the checkpoint's linked snapshot (spec §26.4).

type RecoverResult 

type RecoverResult struct {
	Allocation coordinator.Allocation
	SnapshotID string
	Manifest   workspace.Manifest
}

RecoverResult is the recovered workspace's new allocation and the restored create-side manifest — the evidence a host move happened (a strictly higher fence) and the restore is checksum-EQUAL (SAN-005).

type RemoteAgents 

type RemoteAgents interface {
	GetRemoteAgent(ctx context.Context, project, id string) (a2a.RemoteAgent, bool, error)
}

RemoteAgents resolves a REGISTERED remote A2A agent row (a2a_remote_agents, §38.5, E19 T5). *a2a.Store satisfies it. The lookup is tenant-scoped by construction — org/project are parameters, not anything the child.request supplies — so a run can only reach an agent its OWN project registered, and a frame naming another tenant's registration id resolves to nothing (no existence oracle).

type RemoteChildRunner 

type RemoteChildRunner interface {
	RemoteChildRun(ctx context.Context, agent a2a.RemoteAgent, req a2a.RemoteChildRequest) (a2a.RemoteChildResult, error)
}

RemoteChildRunner dispatches one delegated child objective to a resolved remote agent. *a2a.Client satisfies it. The signature is the structural half of "no credential inheritance" (A2A-005/SUB-007): it takes the resolved agent and the child's own request and NOTHING else, so there is no parameter through which the parent's credential — or the parent's context — could travel. Do not add one.

type RemoteShell 

type RemoteShell struct {
	// contains filtered or unexported fields
}

RemoteShell runs a run's shell commands on the machine that holds its lease. It satisfies toolbroker.ShellRunner, so tools/shell.go dispatches through it without knowing where the command lands — the substitution IS the feature.

func NewRemoteShell 

func NewRemoteShell(conn ExecConn) *RemoteShell

NewRemoteShell binds one attempt's lease connection. A RemoteShell is therefore attempt-scoped, the same as the lease: it can only reach the machine that took this attempt.

func (*RemoteShell) Run 

func (s *RemoteShell) Run(ctx context.Context, cmd toolbroker.ShellCommand) (toolbroker.ShellResult, error)

Run sends one command to the machine and waits for its answer.

A NON-ZERO EXIT IS A RESULT, NOT AN ERROR, and the distinction decides whether a run continues: tools/shell.go turns an error here into an uncertain abort that ends the attempt, while a result with exit_code 1 is handed to the model. `false` exits 1 and completes cleanly. This tree has already paid for the opposite arrangement once, when every tool Exec error wedged its run forever.

There is no timer here on purpose. The command's wall clock is bounded on the MACHINE, by the executor that runs it, and a second bound on this side would have to be either shorter than that one — killing the `xcodebuild` this epic exists for — or longer, in which case it never fires. What ends an unanswerable wait instead is the connection: when the lease drops, every command still waiting is answered (execPending.closeAll).

type RemoteToolProber 

type RemoteToolProber struct {
	// contains filtered or unexported fields
}

RemoteToolProber is the FIRST real ToolDestinationProber (spec §26.7, E12 T4): the "destination read" for an uncertain remote_http tool_call is the durable remote-operation ledger — a LATE signed callback (after the executor's deadline) wrote late_result there without ever touching the tool ledger, so the prober reads it and reports the effect landed. A tool_call with no resolved operation (no callback ever arrived) is unsupported, so the reconcile loop escalates it to manual_resolution rather than guessing. A non-remote uncertain call (a built-in) also has no operation row, so it too escalates to manual — exactly the pre-T4 behaviour, unchanged.

func NewRemoteToolProber 

func NewRemoteToolProber(ops remoteOperationReader) *RemoteToolProber

NewRemoteToolProber binds the remote-operation ledger the prober reads.

func (*RemoteToolProber) Probe 

func (p *RemoteToolProber) Probe(ctx context.Context, call coordinator.UncertainToolCall) (applied bool, result []byte, supported bool, err error)

Probe reads the uncertain call's remote operation. A completed/late_result row carries the result the callback delivered, so the effect landed (applied=true -> reconciled_completed). No resolved row means the callback never arrived, so the prober is unsupported for this call (-> manual_resolution).

type RemoteWorkspace 

type RemoteWorkspace struct {
	// contains filtered or unexported fields
}

RemoteWorkspace acts on the workspace of the machine that holds one attempt's lease. It satisfies toolbroker.WorkspaceOps, so the file, media, commit, push and pull-request tools reach it without knowing where the bytes are — the substitution IS the feature, exactly as it is for RemoteShell.

func NewRemoteWorkspace 

func NewRemoteWorkspace(conn WorkspaceConn, root string) *RemoteWorkspace

NewRemoteWorkspace binds one attempt's lease connection and the allocation root ON THAT MACHINE. A RemoteWorkspace is therefore attempt-scoped, the same as the lease and the same as RemoteShell.

func (*RemoteWorkspace) Archive 

func (w *RemoteWorkspace) Archive(ctx context.Context) ([]byte, workspace.Manifest, error)

Archive tars this allocation ON THE MACHINE and returns the bytes with the manifest the machine derived from its own tree (Faz A.5 T5). It is the read half of moving a session between Macs: the control plane owns the object store and the snapshot row, and the machine owns the bytes, so the archive is produced there and stored here.

THE MANIFEST IS THE MACHINE'S ANSWER AND NOT THIS SIDE'S GUESS. Restore verifies against it, so a manifest computed here — over a tree this process cannot see — would verify the archive against itself and prove nothing.

func (*RemoteWorkspace) Checksum 

func (w *RemoteWorkspace) Checksum(ctx context.Context, rel string) (string, error)

func (*RemoteWorkspace) Commit 

func (w *RemoteWorkspace) Commit(ctx context.Context, message string) (string, error)

func (*RemoteWorkspace) Glob 

func (w *RemoteWorkspace) Glob(ctx context.Context, pattern string, limit int) ([]string, bool, error)

Glob asks the machine holding this allocation to search its own disk by filename. The pattern travels; the walk does not — a control plane that globbed its own filesystem here would answer for the wrong machine, and would answer at all only by accident of the two happening to be the same.

func (*RemoteWorkspace) Grep 

func (w *RemoteWorkspace) Grep(ctx context.Context, req toolbroker.GrepRequest) (toolbroker.GrepResult, error)

Grep asks the machine holding this allocation to search its own files' contents.

func (*RemoteWorkspace) Head 

func (w *RemoteWorkspace) Head(ctx context.Context) (string, string, error)

func (*RemoteWorkspace) List 

func (w *RemoteWorkspace) List(ctx context.Context, rel string) ([]toolbroker.DirEntry, error)

func (*RemoteWorkspace) Materialize 

func (w *RemoteWorkspace) Materialize(ctx context.Context) (MaterializeReport, error)

Materialize asks the machine to write the workspace store's current head into its allocation.

‼️ IT IS THE CLONE'S REPLACEMENT WHERE THE STORE IS THE ORIGIN. The repository has already been fetched once, centrally, into content-addressed blobs; a machine that cloned as well would fetch the same bytes a second time and would need a repository credential to do it. This fetches only the blobs that machine lacks, through short-lived signed URLs, with no repository credential near it.

‼️ Skipped COUNTS WHAT WAS ALREADY AT THE DESTINATION PATH, AND THAT IS NARROWER THAN IT SOUNDS. This sentence first claimed "a second session on the same Mac pays only for the diff", and the measurement that was supposed to confirm it refuted it instead: two consecutive sessions on one machine both reported `27 fetched, 0 already on the machine`, because each gets a FRESH allocation directory and presence is checked against that directory rather than against anything the machine keeps. So a machine re-fetches every blob for every new allocation today.

The saving that DOES exist is within one allocation: a re-materialise — a resume, a repair, a second pass over the same directory — writes only what changed.

A MACHINE-WIDE BLOB CACHE WOULD WIDEN THAT, AND IT IS A REJECTED OPTION RATHER THAN A GAP. A session does not stay on one machine: the idle releaser hands the machine back and the next message brings the session up wherever there is capacity, so a cache would be paying storage and eviction on every Mac for a hit rate set by how often a session happens to land twice in the same place. The bandwidth is spent instead, deliberately. Recorded here because "no cache" reads like an omission to anyone who finds this field and does the arithmetic — it is a decision, and the reason is the fleet, not the difficulty.

func (*RemoteWorkspace) Open 

func (w *RemoteWorkspace) Open(ctx context.Context) (string, error)

Open creates the §29.9 layout for this allocation on the machine and returns the path the machine gave it back. The returned path may differ from the requested one by symlink resolution — a macOS runner rooted under /var answers /private/var — and the ANSWER is the one to keep, because it is what every later under-root check on that machine compares against.

func (*RemoteWorkspace) Read 

func (w *RemoteWorkspace) Read(ctx context.Context, rel string, maxBytes int64) ([]byte, bool, error)

func (*RemoteWorkspace) Restore 

func (w *RemoteWorkspace) Restore(ctx context.Context, body []byte, want workspace.Manifest) (workspace.Manifest, error)

Restore untars a control-plane-held archive into this allocation on the machine, requiring the restored tree to re-derive `want` (SAN-005). It is the write half of Archive, and the verification happens on the far side for the reason stated there: what must match is the tree a run is about to be given, not the bytes this process is about to send.

The allocation must be FRESH — snapshot.Restore's own requirement — which on this path is guaranteed by the caller opening a newly-named allocation on the machine immediately before.

func (*RemoteWorkspace) Root 

func (w *RemoteWorkspace) Root() string

Root reports the allocation root on the machine. It is the value ShellCommand.WorkspaceRoot carries, so a shell command and a file write name the same directory on the same host.

func (*RemoteWorkspace) Stat 

func (w *RemoteWorkspace) Stat(ctx context.Context, rel string) (toolbroker.FileStat, error)

func (*RemoteWorkspace) Write 

func (w *RemoteWorkspace) Write(ctx context.Context, rel string, content []byte) (toolbroker.WriteReport, error)

type RepositoryPublisher 

type RepositoryPublisher struct {
	// Broker is the DEPLOYMENT-GLOBAL credential — the GitHub App — and it is nil on a deployment that has
	// not configured one. NIL IS A SUPPORTED STATE, not a half-built object: a single-tenant stack whose
	// bindings each name their own connection_ref needs no App at all, and requiring one is what used to
	// make the per-binding credential unreachable. A publication with no ConnectionRef is then REFUSED by
	// CanPublish rather than published under nothing.
	Broker    repositories.Broker
	PRClient  repositories.PullRequestClient // nil disables PR publication (push-only stacks, and App-less ones)
	Protected []string                       // policy-widened protected branches (default main/master always)
	// ConnectionSecrets resolves a binding's OWN credential handle to its bytes, server-side. It is the
	// same function the clone half already uses (main.repositoryConnectionSecret), passed in rather than
	// re-implemented, so exactly one place in this tree turns a connection_ref into a credential.
	//
	// THE TWO NIL CASES ARE DIFFERENT AND ONLY ONE OF THEM IS QUIET. A target carrying no ConnectionRef
	// never reads this field, so the deployment-global path is untouched by its absence. A target that
	// DOES carry one and finds this nil is REFUSED — a binding that deliberately named its own credential
	// must not publish under the deployment App because the resolver went missing.
	ConnectionSecrets func(tenant coordinator.Tenant, ref string) ([]byte, error)
	// PRClientFor builds a pull-request client over an ALREADY-RESOLVED token, for one owner/repo. It is a
	// factory rather than a client because after this change both the credential and the repository are
	// per-binding: one stack serves many bindings, and PRClient above can only ever serve one.
	PRClientFor func(token, owner, repo string) (repositories.PullRequestClient, error)
}

RepositoryPublisher publishes through the adapters/repositories push/PR path (spec §30.9-30.10): a push mints a write-scoped broker credential and pushes the exact approved head; a pull request finds or opens a draft through the provider client. It holds only the broker + PR client + branch policy — everything else comes from the publication row + the attempt's workspace, so the model never supplies a destination.

func (*RepositoryPublisher) CanPublish 

func (p *RepositoryPublisher) CanPublish(connectionRef string) error

CanPublish answers whether a publication carrying this connection ref has a credential PATH on this deployment at all. It is the STRUCTURAL half of the decision credentialFor makes below, factored out so that the answer the publish boundary gives and the answer a human is given BEFORE they approve cannot disagree — they are the same call, not two copies of one rule.

THE SECOND BRANCH IS THE ONE THIS EXISTS FOR. A binding with no connection_ref has always meant "publish under the deployment's GitHub App". On a deployment that has no App there is nothing for that to mean, and both wrong answers are worse than a refusal: publishing anonymously, or publishing under whatever credential happens to be reachable. Before this, the whole publisher was built inside the App gate, so this case was not refused — it was INVISIBLE: the pump had no publisher, an approved row sat at `approved` forever, and a human had been told they authorized a push.

WHAT IT DOES NOT CLAIM, and the qualification is load-bearing: it does not say the credential RESOLVES. A ref naming a secret nobody provisioned passes here and fails at publish, with a warning on the row. That split is deliberate — resolving a tenant's secret to throw the bytes away is a read this question does not need, and the refusal it would add is one the publish boundary already makes.

func (*RepositoryPublisher) Publish 

func (p *RepositoryPublisher) Publish(ctx context.Context, target PublishTarget) (map[string]any, error)

Publish executes one approved publication (spec §30.9-30.10). The credential rides only the broker helper file and binds to the attempt fence; a per-publish temp secrets dir holds it and is removed after (defence in depth on top of the broker's own revoke).

type RepositoryStore 

type RepositoryStore interface {
	GetRepositoryBinding(ctx context.Context, tenant coordinator.Tenant, id string) (contracts.RepositoryBinding, bool, error)
	RecordPreparationReceipt(ctx context.Context, tenant coordinator.Tenant, in coordinator.PreparationReceiptInput) error
}

RepositoryStore is the coordinator seam the repository-preparation step resolves bindings and records receipts through. *coordinator.Store implements it; a fake implements it in the unit test (the ReconcileStore pattern), so the composition is provable without a database.

type ResolveInput 

type ResolveInput struct {
	DeploymentModel  string
	DeploymentSecret string // SecretRef NAME, never a credential value
	// ProjectRouteModel / ProjectRouteSecret are the project's DB-backed model route (E13 Task 8), the
	// layer directly above the deployment default: the model id its published revision selects and the
	// tenant-qualified credential REF of the connection that revision binds. Empty means the project
	// published no route, leaving resolution bit-identical to the pre-T8 deployment-only path.
	ProjectRouteModel  string
	ProjectRouteSecret string
	ProjectTools       []string
	// AgentRevisionID / Model / Tools are the pinned AgentRevision or RunTemplateRevision layer
	// (E11 Task 1, AGT-001). An empty AgentRevisionID means the run pins no revision — the layer is
	// skipped and resolution behaves exactly as before (the profile-free path). AgentRevisionModel,
	// when set, pins the model above the deployment default (a session override still wins).
	// AgentRevisionTools, when non-nil, is a capability CEILING intersected with the resolved tools:
	// a tool the revision does not declare never reaches the effective set (63.4 "capability never
	// expands"), even when the project baseline or a session override carries it.
	AgentRevisionID    string
	AgentRevisionModel string
	AgentRevisionTools []string
	// AgentRevisionToolSetTools is the model-visible short names contributed by the pinned revision's
	// tool_sets (E12 Task 2, EXT-003). They UNION into the resolved baseline (provenance agent_revision)
	// BEFORE the AgentRevisionTools ceiling intersects — so a revision can both grant a registered tool
	// via a set AND narrow the whole set with a ceiling. Empty leaves resolution bit-identical to before.
	AgentRevisionToolSetTools []string
	// RequestModel, POST /v1/responses gövdesinin `model` alanının ÇÖZÜLMÜŞ model id'sidir —
	// bağlantı öneki admission'da ayrılmıştır (modelbroker.SplitTarget), yani buraya asla
	// `ad:model` biçiminde bir değer gelmez. "" her katmanı olduğu gibi bırakır ve config
	// adresini kımıldatmaz (config_test.go: TestAnAbsentRequestModelLeavesTheHashUnmoved).
	RequestModel string
	SessionModel string   // cumulative session model override ("" = never set)
	SessionTools []string // cumulative session tools override (nil = never set)
	// SkillPinsJSON is the run's frozen skill set (E12 Task 7, spec §28.16): the JSON-encoded
	// []{name,description,digest,path} pinned at run-start. nil/empty means the run resolved no skills —
	// the snapshot then carries no Skills and the hash is BIT-IDENTICAL to the pre-skills path (a
	// skill-less run's config address never moves, the T1 advertising regression discipline). A non-empty
	// pin folds into both the Skills rider and the content hash, so a mid-run enable that would change the
	// pin (it cannot — the pin is frozen) would change the hash, keeping the checkpoint coherent.
	SkillPinsJSON []byte
}

ResolveInput is the config resolution input: the deployment default (model + credential ref), the project's model route, the project tools baseline, and the cumulative session override (spec §14). An empty SessionModel / nil SessionTools means the session never overrode that value, so it inherits the lower layer. The PROVIDER is not a layer here: which adapter a call goes to is decided by the route itself (execution.effectiveRoute), not by config resolution — this input resolves the model id, its credential ref, and the tool set.

type RouteSecretResolver 

type RouteSecretResolver struct {
	// Lookup resolves a ref name to the credential value FOR ONE TENANT; ok=false is a clean miss.
	Lookup func(tenant coordinator.Tenant, name string) ([]byte, bool, error)
	// Fallback redeems unqualified deployment refs (the env bridge). Nil rejects every such ref.
	Fallback modelbroker.SecretResolver
}

RouteSecretResolver is the broker's credential redemption for DB-backed routes. A tenant-qualified ref (minted by a project's route) resolves through Lookup — the E13 T3 secret store — and every other ref falls through to Fallback, the deployment-default env bridge.

A tenant-qualified ref the store cannot resolve FAILS CLOSED. Falling back to the deployment credential would silently run — and bill — one tenant's project on the operator's own key.

func (RouteSecretResolver) Redeem 

func (r RouteSecretResolver) Redeem(ref modelbroker.SecretRef) (string, error)

Redeem implements modelbroker.SecretResolver. The value lives only in the returned frame — it is never logged, and the error text names the ref, never the credential.

type RunAdvancer 

type RunAdvancer interface {
	ApplyRunTransition(ctx context.Context, tenant coordinator.Tenant, runID string, cmd statemachines.RunCommand) (coordinator.Transition, error)
}

RunAdvancer is the transactional run-transition seam a job handler drives runs through. *coordinator.Store implements it (its ApplyRunTransition locks the run, applies the pure state machine, and commits state + event + outbox in one tx); a fake implements it in unit tests.

type RunnerGateway 

type RunnerGateway struct {
	// contains filtered or unexported fields
}

RunnerGateway is the control-plane counterpart of the runner's outbound-only model: it serves the enrollment endpoint and the mutually-authenticated session endpoint the runner dials out to, and it is the production EngineDialer — Dial offers a connected runner the waiting attempt's lease and bridges its session frames as an EngineChannel. The orchestrator is written once against that seam and never learns it drives a runner over a WebSocket rather than a subprocess.

func NewRunnerGateway 

func NewRunnerGateway(issuer CertIssuer, tokens EnrollmentTokens) *RunnerGateway

NewRunnerGateway binds the CA issuer and the one-use token store into a gateway.

func (*RunnerGateway) ApproveRunner 

func (g *RunnerGateway) ApproveRunner(runnerID string)

ApproveRunner admits ONE machine out of a strict pool's waiting room (E24 T6): its held-open session leaves awaitApproval, joins its pool's rendezvous, and — because the join is followed by the same wake every connect performs — re-enters a run that parked on that pool for want of a machine.

IT IS CALLED FOR EVERY *found* APPROVAL AND NOT ONLY FOR THE ONE THAT MOVED THE ROW, which is what makes the operator's retry the fix for one narrow race: the API writes the row and then tells this gateway, so an approval landing between a connecting machine's row read and its adoption of that row leaves the row admitted and the session still waiting. A second approve is a 200 (the statement is idempotent) and reaches the session. Safe for a machine that has never connected: the record is created and adopted at its next connect, the same as a cordon written while a Mac is rebooting.

func (*RunnerGateway) CallMachine 

func (g *RunnerGateway) CallMachine(ctx context.Context, runnerID, verb string, payload map[string]any) (map[string]any, error)

CallMachine asks ONE named machine a bg.* question and waits for its answer.

THE THREE LIFECYCLE VERBS ANSWER DIFFERENTLY, AND THE DIFFERENCES ARE THE POINT rather than an accident of where the checks happen to sit:

  • CORDON ANSWERS. A cordon stops new PLACEMENT, not running work; a machine is cordoned precisely so it can be emptied safely. Refusing here would orphan every background task already running on the machine an operator is trying to drain — the opposite of what the verb is for.
  • DRAIN ANSWERS until the connection goes, which is what a graceful close means. After it goes there is no session and the caller gets ErrMachineUnreachable, which it reports as `lost`.
  • REVOKE REFUSES. It is the hard stop (SAN-011): the machine is no longer ours, its frames are refused and its sessions are cut. `lost` is the honest answer for a machine we do not own — the process may still be running and we cannot prove it is ours — and it carries the rule that matters, because a lost handle is never signalled.

func (*RunnerGateway) Connected 

func (g *RunnerGateway) Connected() int64

Connected reports the number of runner sessions currently held open on the gateway — the value behind the palai_runner_sessions gauge. Safe to call from the metrics scrape goroutine.

func (*RunnerGateway) Cordon 

func (g *RunnerGateway) Cordon()

Cordon stops the gateway offering NEW leases: Dial returns ErrRunnerCordoned so a waiting attempt requeues instead of dispatching onto a runner that is about to be replaced (§48.4 drain). An in-flight lease is untouched. Resume clears it. Idempotent.

func (*RunnerGateway) CordonRunner 

func (g *RunnerGateway) CordonRunner(runnerID string)

CordonRunner stops offering NEW leases to ONE machine: it leaves its pool's rendezvous and stays connected, so an in-flight lease finishes and nothing new arrives. ResumeRunner puts it back. Both are idempotent, and both are safe for a machine that has never connected — the record is created and the machine adopts it when it does, which is what makes a cordon written while a Mac is rebooting stick.

func (*RunnerGateway) Cordoned 

func (g *RunnerGateway) Cordoned() bool

Cordoned reports whether new leases are currently refused (cordoned or revoked). Revoked reports the hard-stop state. Both back the drain/revoke drills and a doctor surface.

func (*RunnerGateway) Dial 

func (g *RunnerGateway) Dial(ctx context.Context, attempt AttemptDescriptor) (EngineChannel, error)

Dial offers a machine IN THE ATTEMPT'S POOL the attempt's lease and returns the bridged EngineChannel. It blocks until such a machine is free or ctx is done, then publishes the channel to the connection's readLoop and writes the lease.offer. It is the production EngineDialer the orchestrator drives unchanged.

THE POOL IS A REFUSAL, NOT A PREFERENCE (§2). A machine enrolled in another pool is not offered this lease — not as a fallback, not as the nearest thing — because a pool IS a posture: an attempt that needs an unsandboxed host is not nearly satisfied by a sandboxed container. Structurally the queue is the enforcement: a machine parked in pool A is unreachable from a Dial on pool B, so there is no code path that could relax it later.

An empty pool means the Dial WAITS, which today ends at ctx (~20s, then the retry ladder). T4 turns that wait into a durable park; nothing here needs to change for it.

func (*RunnerGateway) Drain 

func (g *RunnerGateway) Drain(ctx context.Context) error

Drain cordons the gateway and blocks until every in-flight lease has quiesced (active == 0) or ctx is done. It stops new leases and waits for the in-flight lease to finish; if it cannot finish within ctx, the caller (a control-plane shutting down for a swap) exits anyway and the interrupted run is reclaimed and completed by the EXISTING E10 recovery layer (coordinator reconcile + WorkspaceRecovery, §26.3) — drain REUSES that layer, it does not re-implement run migration here. Returns nil on quiesce, ctx.Err() on timeout. THE BODY IS E15 T2'S, UNCHANGED, AND ONLY THE COUNTER MOVED (E24 T5): `g.active.Load()` became the SUM of the per-machine counters. The cordon is still whole-gateway, the tick is still 25ms, the handover to the E10 recovery layer is still by exiting anyway, and the return is still nil-on-quiesce / ctx.Err()-on-timeout. A per-runner drain is deliberately NOT a surface: nothing in production would call one, and this task's own subject is surfaces with no caller.

func (*RunnerGateway) Heartbeat 

func (g *RunnerGateway) Heartbeat(ctx context.Context, timeout time.Duration) (alive, cut int)

Heartbeat pings every live runner session, advances the registry's liveness stamp for the ones that answer, and CUTS the ones that do not. It reports (alive, cut).

A PING RATHER THAN A RUNNER-SENT FRAME, and this is the load-bearing design decision of the reaper. §T5 assumed the runner already sends heartbeats; it does not (see readLoop's default arm). A ping needs NO runner change at all — coder/websocket answers one from inside the peer's own read loop, and every runner is in that loop for its whole session — and it proves strictly more than a timer-driven frame would: that the connection is alive in BOTH directions right now. That is what catches the failure this reaper exists for, which is a session alive to the kernel and dead to the process — a suspended laptop, an unplugged Mac, a wedged runner — where `readLoop` learns nothing because a peer that stopped answering produces no read error.

CUTTING IS ALL IT DOES, AND IT DELIBERATELY DOES NOT WAKE THAT POOL'S PARKED RUNS: capacity is still absent, so waking a run would hand it straight back to a pool with nothing in it. What the cut DOES do is decrement the count that decides whether the next run parks or rides the retry ladder — the pool's membership — so a run placed there parks (honest: nothing here) instead of being handed to a corpse.

IT WRITES NO `unhealthy` STATE, WHICH IS A CORRECTION TO §T5. `runners.state` is CHECK-constrained to ('pending','active','cordoned','revoked') by migration 000045 and E24 owns exactly one migration, which is T1's — so `unhealthy` is not a value this task can write. It is also not a value this task needs: health is DERIVED from `last_seen_at`, which is already durable and already advancing, and a stamped flag would additionally have to be CLEARED when the machine came back. A reaper re-derives after a restart either way.

func (*RunnerGateway) HeartbeatLoop 

func (g *RunnerGateway) HeartbeatLoop(ctx context.Context) error

HeartbeatLoop runs Heartbeat every heartbeatInterval until ctx is done. It is the shape every other supervised loop in this tree has, so a transient error is a logged restart rather than a dead reaper — and it returns ctx.Err() so the supervisor knows the difference between cancelled and crashed.

func (*RunnerGateway) LastRunnerIdentity 

func (g *RunnerGateway) LastRunnerIdentity() (RunnerIdentity, bool)

LastRunnerIdentity reports the client certificate the gateway last saw, or false when no runner has presented one yet.

func (*RunnerGateway) Resume 

func (g *RunnerGateway) Resume()

Resume clears a cordon so the gateway offers leases again — the rollback/abort counterpart to Cordon.

func (*RunnerGateway) ResumeRunner 

func (g *RunnerGateway) ResumeRunner(runnerID string)

ResumeRunner clears a cordon. It does NOT un-revoke: see runnerLifecycle.set.

func (*RunnerGateway) Revoke 

func (g *RunnerGateway) Revoke()

Revoke is the hard stop (SAN-011): new connects are rejected and session frames from any live runner connection are dropped (stale events refused), on top of the cordon's new-lease refusal. A revoked gateway never un-revokes in-process — a revoked runner identity is decommissioned, not paused.

func (*RunnerGateway) RevokeRunner 

func (g *RunnerGateway) RevokeRunner(runnerID string)

RevokeRunner is the hard stop for ONE machine (SAN-011, per-runner): its live sessions are CUT, its in-flight lease dies with them and is reclaimed by the existing E10 recovery layer, its session frames are refused, and it cannot reconnect. Irreversible in this process; the durable half is `runners.state = 'revoked'`, which is what makes it survive a restart.

func (*RunnerGateway) Revoked 

func (g *RunnerGateway) Revoked() bool

func (*RunnerGateway) Routes 

func (g *RunnerGateway) Routes() http.Handler

Routes returns the gateway HTTP surface: the certless enrollment endpoint and the mutually-authenticated session endpoint. It carries no public API auth middleware — the endpoints assert their own token and mTLS identity.

func (*RunnerGateway) RunnerActiveLeases 

func (g *RunnerGateway) RunnerActiveLeases(runnerID string) int64

RunnerActiveLeases reports how many leases ONE machine is currently serving — the answer to the question a cordon exists to let an operator ask, which is "can I take this Mac away yet?". It is the per-runner half of the counter Drain sums.

func (*RunnerGateway) RunnerConnections 

func (g *RunnerGateway) RunnerConnections(runnerID string) int64

RunnerConnections reports how many live sessions ONE machine is holding on this gateway right now.

‼️ IT IS THE AUTHORITATIVE "IS THIS MACHINE CONNECTED", and it exists because `last_seen_at` is not one. A durable timestamp answers "when did this machine last say something", which is a different question from "is it there now" — and on a Mac that was unplugged four minutes ago the two answers disagree in the direction that matters to an operator deciding whether to send it work.

ONE MACHINE IS ONE MACHINE HOWEVER MANY LOOPS IT PARKS. A runner keeps one session per concurrent lease (PALAI_RUNNER_CONCURRENCY), so this counts the sessions belonging to a single runner id rather than the runners belonging to a session set. The panel renders the state from >0 and the count beside it: two connections is one online machine with room for two leases, not two machines.

func (*RunnerGateway) RunnerRefusal 

func (g *RunnerGateway) RunnerRefusal(runnerID string) (string, time.Time)

RunnerRefusal reports why ONE machine's last connect attempt was turned away, or empty when its last attempt succeeded — which is the difference between a Mac somebody unplugged and a Mac that is trying and being rejected. Only one of those is a job for an operator.

func (*RunnerGateway) SetCapacityWaker 

func (g *RunnerGateway) SetCapacityWaker(w CapacityWaker)

SetCapacityWaker wires the durable wake a machine's arrival performs (E24 T4). Unset, handleConnect wakes nothing — the posture every Docker-free wire proof runs in, where there are no durable runs.

func (*RunnerGateway) SetControlPlaneVersion 

func (g *RunnerGateway) SetControlPlaneVersion(v string)

SetControlPlaneVersion overrides the control-plane version stamp the connect handshake checks the runner's advertised version against (§48.2 window). Defaulted to version.Resolve; a test injects a concrete version to exercise the OPS-008 skew rejection deterministically.

func (*RunnerGateway) SetMachineLoadView 

func (g *RunnerGateway) SetMachineLoadView(v MachineLoadView)

SetMachineLoadView wires the placement preference. Unset, Dial takes the machine at the head of its pool's parked list exactly as it did before this file existed — the posture of every Docker-free wire proof and of the conformance tier, where there is no database to weigh anything with.

func (*RunnerGateway) SetPoolKeys 

func (g *RunnerGateway) SetPoolKeys(k PoolEnrollment)

SetPoolKeys wires the pool enrolment keys (E24 T3) ahead of the file bootstrap token in the credential chain. Unset, the gateway admits only the file token and every machine lands in the default pool — the pre-E24 posture, unchanged to the byte.

func (*RunnerGateway) SetPoolSettings 

func (g *RunnerGateway) SetPoolSettings(s PoolSettings)

SetPoolSettings wires the desired-configuration read into enrolment. Unset, every enrolment answers with no settings and every runner falls back to the configuration it was started with — the posture of every deployment built before the runner plane had a reader.

func (*RunnerGateway) SetRegistry 

func (g *RunnerGateway) SetRegistry(r fleet.Registry)

SetRegistry wires the durable runner inventory (E24 T1). Unset, the gateway records nothing and behaves exactly as it did before the registry existed — see the field comment for why that is a supported posture rather than a fallback. A setter rather than a constructor parameter for the reason SetControlPlaneVersion is one: every existing caller compiles unchanged.

func (*RunnerGateway) SetRunnerLogs 

func (g *RunnerGateway) SetRunnerLogs(sink RunnerLogSink)

SetRunnerLogs wires the fleet log sink. A setter rather than a constructor argument for the reason every other optional organ on this gateway is one: a deployment without it behaves exactly as it did.

func (*RunnerGateway) SetWorkspaceStore 

func (g *RunnerGateway) SetWorkspaceStore(store WorkspaceStoreService)

SetWorkspaceStore binds the store the machine-originated workspace frames are served from. Nil — the default — leaves those frames unanswered by design: a deployment without the store must not pretend to accept a publish, so the arms below answer with an explicit refusal rather than silence.

func (*RunnerGateway) Waiting 

func (g *RunnerGateway) Waiting(poolID string) int

Waiting reports how many attempts are currently queued for a pool with no machine free to take them, across every tenant that has a machine or an attempt in it. It is the number behind the one question an operator of a fleet actually asks — "why is nothing running in my Mac pool" — which before E24 had no answer at all: a blocked Dial was a goroutine parked on a channel and nothing counted it.

type RunnerIdentity 

type RunnerIdentity struct {
	RunnerDNS string    `json:"runner_dns"`
	NotAfter  time.Time `json:"not_after"`
	SeenAt    time.Time `json:"seen_at"`
}

RunnerIdentity is the client certificate a runner last presented to the gateway: who it claimed to be, when that certificate stops being valid, and when the gateway saw it.

type RunnerLogLine 

type RunnerLogLine struct {
	At        time.Time `json:"at"`
	Level     string    `json:"level,omitempty"`
	SessionID string    `json:"session_id,omitempty"`
	Message   string    `json:"message"`
}

RunnerLogLine is one line as it crosses the wire. It mirrors the store's shape rather than sharing it, because this one is a WIRE CONTRACT: a field added here is a field every deployed agent may send, and the two should be free to move apart.

type RunnerLogSink 

type RunnerLogSink interface {
	Append(ctx context.Context, project, runnerID string, lines []RunnerLogLine) error
}

RunnerLogSink takes the lines one machine shipped. It is an interface so the gateway does not depend on the store, and so a test can drive the route without a database.

type SecretResolver 

type SecretResolver func(tenant coordinator.Tenant, ref string) ([]byte, error)

SecretResolver bridges a server-minted (tenant, secret ref) to the secret bytes at use time. It named an ORGANIZATION until 000006, and for the phase between A.2 and that migration it named a two-element tuple with one element: the ref was all the redemption used — the same shape the webhook / inbound / remote-tool / MCP resolvers already take in their packages. The composition root satisfies it from the DB-backed secret-ref store (E13 Task 3). The org is never tenant-supplied, so a ref can only ever name a secret provisioned under the caller's OWN organization.

type SkillRef 

type SkillRef struct {
	Name        string `json:"name"`
	Description string `json:"description"`
	Digest      string `json:"digest"`
	Path        string `json:"path"`
}

SkillRef is one entry of a run's progressively-loaded skill set (spec §28.16): the model-visible name + description (the context rider), the exact pinned digest, and the workspace-relative body path the file tool reads on-demand. It grants NO capability — metadata + a file location, never a tool.

type SnapshotCaptureInput 

type SnapshotCaptureInput struct {
	SnapshotID   string
	Project      string
	WorkspaceID  string
	AllocationID string
	HostPath     string
	Reason       string
	// Ops is the filesystem HostPath is a path ON (Faz A.5 T5). Nil means this process's own, which is
	// what every caller meant before a lease could place an allocation on another Mac and is still right
	// for the deterministic tiers and for a control plane serving its own runs. A caller inside an attempt
	// passes the attempt's ops, so the tar is produced by whoever is holding the bytes.
	//
	// IT IS A FIELD RATHER THAN A SECOND FUNCTION because the two differ in ONE step — who reads the tree
	// — and everything after it (the size bound, the object key, the fence-guarded row) must not fork. A
	// captureRemote() would be a second answer to "what is a snapshot", and this file exists to have one.
	Ops toolbroker.WorkspaceOps
}

SnapshotCaptureInput is one allocation to snapshot: its tenant, the logical workspace + physical allocation ids, the on-host allocation directory to archive, and the reason (e.g. a pause boundary).

type SnapshotObjectStore 

type SnapshotObjectStore interface {
	Put(ctx context.Context, key string, body []byte) (checksum string, size int64, err error)
	Get(ctx context.Context, key string) (body []byte, found bool, err error)
}

SnapshotObjectStore is the object-store PUT/GET the opaque snapshot archive bytes ride. It is the same control-plane-only artifacts.Store the checkpoint sink uses (spec §24 — the engine never holds the S3 credential); declaring the interface here keeps the sink decoupled from the artifacts import.

type SnapshotSink 

type SnapshotSink struct {
	// contains filtered or unexported fields
}

SnapshotSink captures + restores a workspace snapshot's BYTE-archive (spec §29.10, E10 Task 6). E09 recorded a manifest-only snapshot; this tars the allocation, PUTs it under a tenant-scoped snapshots/<id> key, and records the row with the object key (guarded by the allocation's fence currency, so a stale host's snapshot is rejected, SAN-006). Restore fetches the bytes and verifies the restored tree re-derives EQUAL create-side checksums (SAN-005).

func NewSnapshotSink 

func NewSnapshotSink(store SnapshotObjectStore, spine *coordinator.Store) *SnapshotSink

NewSnapshotSink binds the object store and the durable spine.

func (*SnapshotSink) Capture 

func (s *SnapshotSink) Capture(ctx context.Context, in SnapshotCaptureInput) (string, error)

Capture archives the allocation at HostPath (INCLUDING .git, secrets excluded — SAN-005), size-bounds the archive BEFORE the PUT (so an oversize snapshot leaves no orphan object), stores the bytes, and records the immutable row. The row insert is fence-guarded: a stale allocation (a host move advanced the fence) affects zero rows and returns coordinator.ErrStaleAllocation — the DB-level SAN-006 reject. It returns the snapshot id the checkpoint boundary links (spec §26.4).

func (*SnapshotSink) RestoreThrough 

func (s *SnapshotSink) RestoreThrough(ctx context.Context, tenant coordinator.Tenant, snapshotID, dest string, ops toolbroker.WorkspaceOps) (workspace.Manifest, error)

RestoreThrough is RestoreTo onto the filesystem `ops` names (Faz A.5 T5). Nil ops is this process's own — RestoreTo, unchanged — and an attempt's ops put the bytes on the machine that took the lease, which is what lets a session resume on a Mac that is not the one it was captured on.

func (*SnapshotSink) RestoreTo 

func (s *SnapshotSink) RestoreTo(ctx context.Context, tenant coordinator.Tenant, snapshotID, dest string) (workspace.Manifest, error)

RestoreTo fetches snapshot snapshotID's archived bytes and restores them into dest (a FRESH allocation dir), verifying the restored tree re-derives EQUAL create-side checksums (SAN-005). An absent archive is ErrSnapshotArchiveMissing and a checksum mismatch is snapshot.ErrRestoreChecksumMismatch — both surfaced so a recovering workspace fails explicitly rather than resuming on a wrong/empty tree.

type StoreProvisioner 

type StoreProvisioner struct {
	// contains filtered or unexported fields
}

StoreProvisioner puts a workspace's first tree into the object store, BEFORE any machine is chosen.

‼️ THIS IS THE INVERSION, IN ONE FUNCTION. Until it, a workspace came into existence by cloning onto a Mac: the repository's first bytes and the machine that would hold them arrived together, and every later question — can another device read this, what happens when the machine is handed back — was answered by "no" or "recover it from a backup". Cloning into the store first makes the workspace a thing that EXISTS independently of any machine, and a machine's copy a materialisation of it.

What it buys, in order of how much it matters:

  • The clone happens ONCE, centrally. Two devices no longer clone the same repository from GitHub, and the repository credential never has to reach either of them.
  • A workspace has contents before it has a host, so "give me the current tree" is answerable at any moment, including while no machine holds it at all.
  • The resume path stops being a special case. Restoring a paused workspace and provisioning a fresh one become the same operation — materialise the head — and one of the two code paths goes away.

func NewStoreProvisioner 

func NewStoreProvisioner(spine GenesisSpine, blobs BlobWriter, prepare CloneToDir) *StoreProvisioner

NewStoreProvisioner binds the three halves: where the repository comes from, where the bytes go, and where the head is recorded.

func (*StoreProvisioner) InitWorkspace 

func (p *StoreProvisioner) InitWorkspace(ctx context.Context, in InitInput) (string, error)

InitWorkspace clones a binding into a temporary directory, publishes the resulting tree to the store, and throws the directory away.

THE TEMPORARY DIRECTORY IS THE POINT, not an implementation detail: the clone has to land on SOME filesystem for git to write it, and this one belongs to nobody. It is removed on every return path, so the control plane accumulates no workspace trees — it is not a machine, and a directory left behind here would be the first step back toward one.

The upload order is the same contract the ordinary publish keeps: every blob first, the manifest and the head last, so a workspace is never visible in a half-written state.

type ToolApprovalDisplay 

type ToolApprovalDisplay = slack.ApprovalDisplay

ToolApprovalDisplay is an alias rather than a wrapper struct: an alias keeps every field, every JSON tag and every existing `display.Arguments` untouched, so nothing downstream had to change to follow the move.

func DeriveToolApprovalDisplay 

func DeriveToolApprovalDisplay(identity, operatorLabel string, arguments []byte) ToolApprovalDisplay

DeriveToolApprovalDisplay computes the screen from a parked call's ledger row. identity and operatorLabel MUST come from the same resolution the executor uses (RequiresApprovalResolved returns both from one lookup, which is why it returns both), and arguments are the row's committed bytes — the ones Execute will run, never the frame's.

type ToolDestinationProber 

type ToolDestinationProber interface {
	Probe(ctx context.Context, call coordinator.UncertainToolCall) (applied bool, result []byte, supported bool, err error)
}

ToolDestinationProber queries a tool's DESTINATION to decide whether an uncertain effect actually landed (spec §26.7, fork 5: the tool's own read surface, e.g. a GET against the same endpoint — NOT a generic prober framework). Returns applied + the result to record when applied. A tool with no probe surface returns supported=false, so the loop escalates to manual_resolution rather than guessing.

type UncertainReconcileStore 

type UncertainReconcileStore interface {
	UncertainToolCalls(ctx context.Context, limit int) ([]coordinator.UncertainToolCall, error)
	ReconcileToolCall(ctx context.Context, tenant coordinator.Tenant, sessionID, responseID, runID, callID, resolution string, result []byte) error
	ReenqueueResponseRun(ctx context.Context, tenant coordinator.Tenant, runID string) error
}

UncertainReconcileStore is the coordinator seam the reconcile loop drives (the ReconcileStore idiom): read the uncertain set, resolve each to an exit, and re-enqueue the run. *coordinator.Store implements it; a fake implements it in the deterministic test.

type UncertainReconciler 

type UncertainReconciler struct {
	// contains filtered or unexported fields
}

UncertainReconciler resolves uncertain tool_calls on a supervised interval. batch bounds one pass.

func NewUncertainReconciler 

func NewUncertainReconciler(store UncertainReconcileStore, prober ToolDestinationProber, interval time.Duration, batch int) *UncertainReconciler

NewUncertainReconciler binds the store, the destination prober, and the sweep interval/batch.

func (*UncertainReconciler) Run 

func (r *UncertainReconciler) Run(ctx context.Context) error

Run sweeps every interval until ctx is cancelled. A sweep error is non-fatal — the next tick retries — so a transient database or probe blip never stops the reconcile safety net (reconciler.go's discipline).

func (*UncertainReconciler) Sweep 

func (r *UncertainReconciler) Sweep(ctx context.Context) (int, error)

Sweep runs one reconciliation pass and returns the number of uncertain tool_calls resolved. Each is driven to its §26.7 exit; a run whose call reached a reconciled_* exit is re-enqueued to continue, while a manual_resolution one stays stopped for a human.

type WorkspaceAnswer 

type WorkspaceAnswer struct {
	Data map[string]any
	Err  error
}

WorkspaceAnswer is one ws.result, decoded. Exactly one side is meaningful: Err for a failure the machine named, Data for an operation that completed.

type WorkspaceBinder 

type WorkspaceBinder interface {
	BindWorkspace(workspaceID, allocationID string)
}

WorkspaceBinder is a channel that can be told which workspace its lease ended up holding.

‼️ IT EXISTS BECAUSE THE DESCRIPTOR THE GATEWAY KEEPS IS A SNAPSHOT TAKEN AT DIAL TIME, and since A.3 T5 provisioning runs AFTER the dial — the control plane must know which machine it got before it can lay an allocation out on that machine's disk. So the attempt gains its WorkspaceID and WorkspaceAllocationID ~80 lines below the line that copied it into the channel, and the copy never sees them. Measured 2026-08-10 on a live stack: the machine asked for its head and this control plane answered "this lease holds no workspace" while the allocation was on disk with a clone in it.

Binding is NOT a lookup, and the distinction is the fence. What is handed over here is what provisioning DECIDED for this lease; a handler that resolved the workspace's current allocation itself could never disagree with the store it is about to be checked against.

type WorkspaceConn 

type WorkspaceConn interface {
	// StartWorkspace registers wsID as awaiting an answer and then sends the request, in that order —
	// the connection's reader can deliver an answer the instant after the write returns, and a
	// registration made afterwards would race it. It returns the channel the answer arrives on and the
	// release that unregisters the wait.
	StartWorkspace(ctx context.Context, wsID, op, root string, payload map[string]any) (<-chan WorkspaceAnswer, func(), error)
}

WorkspaceConn is the lease connection RemoteWorkspace asks a machine to act on its own disk over. It is a narrow interface for the reason ExecConn is one: Dial returns an EngineChannel, and whether that channel can also reach a machine's filesystem is a fact about the transport rather than about every engine channel.

type WorkspaceRecovery 

type WorkspaceRecovery struct {
	// contains filtered or unexported fields
}

WorkspaceRecovery drives a host-lost workspace back to ready on a NEW fenced allocation (spec §29.7-29.8, REC-005/ENG-006). It is the FIRST driver of the leased→host_lost→recovering→ready transitions the workspace SM has carried since E09 (packages/state-machines/workspace.go) with no driver. The logical workspace id is STABLE across the move; only a strictly higher allocation fence appears, which fences out the old host's writes/snapshots at the DB (SAN-006, already proven).

CEILING — DETECTION is not yet wired into the binary. This is the recovery DRIVER (component- and fault-proven); nothing in the control-plane binary yet DETECTS a lost host to invoke it. The intended trigger is the lease-liveness path already in the tree: the durable-job lease-expiry reconcile (coordinator.Supervisor) noticing a workspace whose writer run has a dead response.run job (RunHasLiveResponseJob=false) — the same liveness signal acquireWriterLease reclaims on. Wiring that detect→RecoverWorkspace hook is the remaining E10 integration; until then a host-lost workspace is recovered lazily on the next attempt's provisionRootWorkspace (which reclaims the stale lease) rather than eagerly re-allocated. Named here so this reads as a driver-with-pending-trigger, not as live.

func NewWorkspaceRecovery 

func NewWorkspaceRecovery(spine *coordinator.Store, snapshots *SnapshotSink, root string) *WorkspaceRecovery

NewWorkspaceRecovery binds the durable spine, the snapshot restore path, and the provision root (which doubles as the local-tier host identity for quarantine).

func (*WorkspaceRecovery) DestroyAllocation 

func (r *WorkspaceRecovery) DestroyAllocation(ctx context.Context, tenant coordinator.Tenant, in DestroyInput) error

DestroyAllocation tears an allocation down: it drives the workspace ready/paused/failed→destroying→ destroyed and REMOVES the on-host directory so a later allocation on the same host inherits zero residue — files, credentials, or a dirty writable layer (SAN-007). If the physical teardown FAILS, the host is QUARANTINED (SAN-008): its bytes may still hold tenant data, so no new allocation may be placed there, and the failure is journaled (host.quarantined.v1). The workspace stays in destroying (not destroyed) — the teardown did not complete — and the typed error is returned.

func (*WorkspaceRecovery) RecoverWorkspace 

func (r *WorkspaceRecovery) RecoverWorkspace(ctx context.Context, tenant coordinator.Tenant, in RecoverInput) (RecoverResult, error)

RecoverWorkspace drives leased→host_lost→recovering, mints a NEW allocation (fence+1) under a fresh host dir, restores the boundary snapshot into it verifying the create-side checksums (SAN-005), then recovering→ready and journals workspace.restored.v1. The logical id is unchanged; the old allocation is now a lower fence, so its writer-lease/snapshot attempts are rejected at the DB (SAN-006/ENG-007). If no restorable snapshot exists or the archive is missing/corrupt, it drives recovering→failed with a typed reason (ErrRecoveryImpossible) — never a silent drop.

func (*WorkspaceRecovery) SetTeardown 

func (r *WorkspaceRecovery) SetTeardown(remove func(string) error)

SetTeardown overrides how an allocation's host directory is reclaimed (default os.RemoveAll). It is a real seam — a remote/multi-host tier tears down over the wire rather than with a local RemoveAll — and also the point a destroy-failure fault injects through to exercise the quarantine path (SAN-008).

type WorkspaceStoreService 

type WorkspaceStoreService interface {
	// Grant mints presigned URLs. Write grants are fence-checked against the allocation; read grants are
	// not, because reading a digest you already know is not a privilege the fence protects.
	Grant(ctx context.Context, req workspacestore.GrantRequest, digests []string) ([]workspacestore.Grant, error)
	// Publish records the manifest and advances the head, under the fence.
	Publish(ctx context.Context, in workspacestore.PublishInput) (string, error)
	// HeadManifest is the manifest the workspace is currently at.
	HeadManifest(ctx context.Context, project, workspaceID string) (ws.Manifest, bool, error)
}

WorkspaceStoreService is what the gateway needs from the control plane's workspace store to answer a machine that is publishing its tree. It is an interface here rather than the concrete pair so this package does not depend on the store's construction, and so a test can drive the arms without an object store.

Source Files

View all Source files

Directories

Path Synopsis
Package tools defines the built-in model-facing tool surface (file, shell) that runs behind the sandbox-backed execution seam (spec §28.7-28.8).
 Package tools defines the built-in model-facing tool surface (file, shell) that runs behind the sandbox-backed execution seam (spec §28.7-28.8).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.