substore

Documentation

Overview

Package substore is where subharness bundles live on disk.

PRD §6 and §7. One directory per subharness under the state root, immutable version pages inside it, and the two mutable notes a subharness accumulates — what it has learnt, and how its last run went — beside them:

~/.codeaf/subharnesses/<name>/v1.json      the version record: hash, parent, why
~/.codeaf/subharnesses/<name>/v1/          the bundle that record describes
                              manifest.json
                              program.js
                              prompts/*.md
                              memory.md      the seed this version was minted with
                              evals/         inert until v3; the format exists now
~/.codeaf/subharnesses/<name>/v2.json
~/.codeaf/subharnesses/<name>/v2/
~/.codeaf/subharnesses/<name>/memory.md    the LIVE memory, written by remember()
~/.codeaf/subharnesses/<name>/last-run.json
~/.codeaf/subharnesses/<name>/.mint.lock   the gate one mint at a time holds

THERE IS NO HEAD FILE. The head is the highest version present, which is the rule internal/subharness/store.go states about its own pages and the reason it has never had a pointer that could disagree with what it pointed at.

THE RECORD IS A FILE BESIDE THE DIRECTORY, NOT A FILE INSIDE IT, and that is the whole of how a version is claimed. os.Link cannot claim a directory, so the exclusive-create discipline this package inherited would have had nothing to grip if the record lived under v2/. Written beside it, `v2.json` is exactly the page the old store already minted this way: the first writer to link it into place owns v2, a second writer racing it is told the name is taken — refused, never shifted along to v3, for the reason [Store.write] gives — and the bundle directory is renamed into place only by the winner. See Store.Mint.

THE CLAIM IS NOT ON ITS OWN ENOUGH, and that is what `.mint.lock` is for. An exclusive create refuses two writers who computed the SAME version number, and two writers who read the store a moment apart do not: one of them counts past a record whose bundle has not landed yet and mints a second child of the same parent. [Store.gated] holds one writer at a time across the whole read-check-claim so that cannot be read a moment apart, and the exclusive create stays underneath it as the floor on a filesystem that will not lock.

THE HOLD IS TAKEN UNDER A DEADLINE AND NEVER SIMPLY WAITED FOR. A mint that finds the gate held for the whole of [mintGateBound] is refused with ErrMintBusy rather than parked behind it, because a caller drawing to a person can survive a refusal and cannot survive a wait — see [Store.gated].

The store holds no cache. A bundle is small, read at launch and at dispatch, and edited by hand often enough that a stale read would be the more expensive mistake — the same judgement the harness store made, for the same reason.

Index

• Constants
• Variables
• func ProjectDir(repository string) string
• func ProjectReadDir(repository string) string
• type Build
• type Bundle
• type Files
• type Memory
  • func (m *Memory) Notes() ([]exec.Note, error)
  • func (m *Memory) Path() string
  • func (m *Memory) Read() (string, error)
  • func (m *Memory) Recall(ctx context.Context, query string) ([]exec.Note, error)
  • func (m *Memory) Remember(ctx context.Context, note string) error
• type RunNote
• type Store
  • func At(dir string) *Store
  • func Home() *Store
  • func (s *Store) Dir() string
  • func (s *Store) Head(name string) (int, error)
  • func (s *Store) LastRun(name string) (RunNote, bool)
  • func (s *Store) Lineage(name string) ([]Version, error)
  • func (s *Store) Load(name string, version int) (Bundle, error)
  • func (s *Store) Manifest(name string, version int) (exec.Manifest, error)
  • func (s *Store) Memory(name string) *Memory
  • func (s *Store) Mint(files Files, parent, why string) (Version, error)
  • func (s *Store) Names() ([]string, error)
  • func (s *Store) Record(name string, version int) (Version, error)
  • func (s *Store) RecordRun(name string, note RunNote) error
  • func (s *Store) Source(build Build) exec.BundleSource
  • func (s *Store) VersionDir(name string, version int) string
  • func (s *Store) Versions(name string) ([]int, error)
• type Version

Constants

const (
	// Root is the directory name under the state root, and it is the same word
	// under a project's `.codeaf/` — see [ProjectDir].
	Root = "subharnesses"

	// ManifestFile, ProgramFile, PromptsDir, MemoryFile and EvalsDir are PRD §6's
	// bundle layout, spelled once. A bundle written by hand, by the design flow,
	// or by a `git pull` is the same five names.
	ManifestFile = "manifest.json"
	ProgramFile  = "program.js"
	PromptsDir   = "prompts"
	MemoryFile   = "memory.md"
	EvalsDir     = "evals"
)

Variables

var ErrExists = errors.New("substore: that version is already written")

ErrExists is what Store.Mint answers when the version it was told to write is already on disk. It is the loud half of the exclusive create: a mint that would have replaced a version somebody has already run is refused, never merged.

var ErrMintBusy = errors.New("substore: another mint is in flight")

ErrMintBusy is what Store.Mint answers when another mint held the gate for the whole of [mintGateBound]. It is A REFUSAL AND NOT A FAILED WRITE: the store was never read and nothing was staged, so a caller with time to spare may simply mint again, and a caller on a turn path can say so and carry on. It is a sentinel because telling that apart from a real write failure is the entire reason the acquire is bounded.

var ErrNotFound = errors.New("substore: not found")

ErrNotFound is what a read answers for a subharness or a version that was never minted. Callers tell "no such subharness" from "the disk is broken", so it is a sentinel rather than a formatted string.

Functions

func ProjectDir

func ProjectDir(repository string) string

ProjectDir names where a repository's own bundles would live. It is PRD §7's layer 2 and it is a NAME AND NOTHING ELSE in this phase: the org-sharing story needs no machinery beyond a directory a `git pull` fills, and the lookup that would consult it is one Registry.UseBundles(exec.LayerProject, …) away when the phase that wants it arrives.

func ProjectReadDir

func ProjectReadDir(repository string) string

ProjectReadDir is the repository store a reader should open. The current directory wins; the former directory is accepted only while the current one is absent. Writers keep using ProjectDir and therefore never touch it.

Types

type Build

type Build func(Bundle) (exec.Runner, error)

Build turns a loaded bundle into the thing that runs it. It is the runtime lane's door into this package and the only one.

THE STORE DOES NOT KNOW WHAT JAVASCRIPT IS. PRD §3 says there is ONE generic Go runner parameterized by a bundle, not one runner per bundle, and this function type is that sentence written as a signature: the store's whole job is to hand it a Bundle, and everything about goja, fuel, guards and the deopt is on the other side of it.

type Bundle

type Bundle struct {
	// Name and Version are the identity. Version is the version this bundle
	// actually is, never the head at the time it was asked for, so a runner that
	// holds a bundle open across a mint keeps running what it loaded.
	Name    string
	Version int

	// Dir is the version directory on disk. A runner that wants to open a file
	// this struct did not read — an eval, an attachment a program writes beside
	// its prompts — starts here rather than rebuilding the layout.
	Dir string

	// Manifest is the bundle's manifest.json, validated. Its Provenance is
	// whatever the file claimed and means nothing: the registry stamps that field
	// from the layer the bundle was found at, which is the law
	// exec.Manifest.Provenance states.
	Manifest exec.Manifest

	// Program is program.js, as bytes. The store never interprets it; see
	// [Store.Parse] for the one check it makes, and who supplies it.
	Program []byte

	// Prompts is prompts/*.md by the name an ai() call site refers to them by —
	// the file's own name, "weekly-brief.md", not a path. Every ai() call names
	// one of these (PRD §6) and a promptRef that is not a key here is a bundle
	// that will fail at the call rather than at the load, because the store
	// cannot know which refs the program will reach.
	Prompts map[string][]byte

	// Seed is the memory.md this version was minted with. It is NOT the memory a
	// run reads and writes: a version is immutable, so what a run accumulates
	// cannot live inside one. See [Bundle.Memory] and [Store.Memory].
	Seed []byte

	// Evals names the checks in evals/, sorted. INERT UNTIL v3 and present from
	// v1 on purpose (PRD §6): the slot and the directory exist so that the
	// version that runs them is a new runner and not a bundle format migration.
	// Nothing in this build reads one.
	Evals []string

	// Memory is the live, subharness-scoped memory door — the backing for the
	// runtime's remember() and recall(). It is per NAME and not per version, and
	// it is seeded from [Bundle.Seed] the first time it is touched.
	Memory *Memory
}

Bundle is one version of one subharness, whole, in memory.

PROVISIONAL, AND FLAGGED AS SUCH. The runtime lane owns what a bundle has to carry in order to be run, and at the time this was written its worktree had no Bundle type of its own to code against. This is the store's honest answer to "what is on disk": the manifest, the program, the prompts every ai() call site names, the memory door, and the eval names that are inert until v3. If the runtime landed a different shape, Build is the seam where the two meet — the coordinator points it at an adapter, or this type moves into internal/exec beside the interface it feeds. Nothing else in this package depends on the field list.

type Files

type Files struct {
	// Manifest is the description. Its Name is the subharness being minted, and
	// [exec.Manifest.Validate] is what decides whether it may be written at all.
	Manifest exec.Manifest
	// Program is program.js. A bundle with no program is refused — that is the
	// one file that makes a directory a subharness rather than a description of
	// one.
	Program []byte
	// Prompts is prompts/*.md by filename. A name with no `.md` gets one, so a
	// caller that thought in refs and a caller that thought in files write the
	// same bundle.
	Prompts map[string][]byte
	// Memory is the seed memory.md. Empty is ordinary: most subharnesses learn
	// their domain notes rather than being shipped with them.
	Memory []byte
	// Evals is evals/* by filename, and the directory exists whether or not this
	// is empty.
	Evals map[string][]byte
}

Files is what Store.Mint is handed: a bundle before it has a version.

type Memory

type Memory struct {
	// contains filtered or unexported fields
}

Memory is one subharness's accumulated domain notes — the backing for the runtime's remember() and recall(), which are exec.Env.Remember and exec.Env.Recall.

IT IS SCOPED TO THE NAME AND NOT TO THE VERSION, and that is a decision worth being explicit about because PRD §6 draws memory.md inside the version directory. Both halves of that are kept and neither is bent: the version's memory.md is the SEED the bundle was minted with, immutable like everything else in a version page, and the file a run actually appends to lives one level up, beside the versions. Anything else would make a version mutable — which is the rule the whole store is built on — or would throw a subharness's learning away every time somebody minted a new version of it, which is exactly the thing accumulated domain notes exist not to do.

The file is markdown because a person reads it. It is the same file whether it was written by a run, by an editor, or by a `git pull` over a project store.

Memory.Remember and Memory.Recall TAKE A CONTEXT THEY BARELY USE, and that is deliberate: they are exec.Env.Remember and exec.Env.Recall, spelled the way the contract spells them, so the runtime hands this value straight to a bundle instead of wrapping it in an adapter whose only job is to drop an argument. An interface a store satisfies by having the methods is one nobody has to keep in sync.

func (*Memory) Notes

func (m *Memory) Notes() ([]exec.Note, error)

Notes is every note in the file, most recent first.

func (*Memory) Path

func (m *Memory) Path() string

Path names the file, for a surface that wants to open it in an editor.

func (*Memory) Read

func (m *Memory) Read() (string, error)

Read is the whole file, for a surface that wants to show it. A subharness that has never remembered anything reads as empty and is not an error.

func (*Memory) Recall

func (m *Memory) Recall(ctx context.Context, query string) ([]exec.Note, error)

Recall answers the notes that match, most recent first.

THE SEARCH IS A SUBSTRING AND THAT IS THE WHOLE OF IT. A subharness's memory is a handful of domain notes, not a corpus, and the model on the other end of this call is perfectly able to read fifteen lines and decide which two matter. An empty query is everything, which is what a program asking "what do I know about this domain" means.

func (*Memory) Remember

func (m *Memory) Remember(ctx context.Context, note string) error

Remember appends one note, stamped.

APPEND, NEVER REWRITE. A subharness's memory is a log of what it learnt and when, and the only operation on it is adding to the end — there is no door here that edits or forgets, because a program that could quietly rewrite its own history is a program whose recall nobody can trust. A person editing the file is a different matter: it is theirs, it is markdown, and nothing here objects.

type RunNote

type RunNote struct {
	// At is when the run ended, UTC.
	At time.Time `json:"at"`
	// Version is which version ran. A row that says "yesterday" about a
	// subharness somebody has since re-minted is answering about the old program,
	// and this is what lets a surface say so.
	Version int `json:"version,omitempty"`
	// Finished is exec.RunResult.Finished, asked once at the moment it was true
	// and written down. It is a bool and not a word because the word is the tui's.
	Finished bool `json:"finished"`
	// Why is exec.RunResult.Incomplete — why it did not finish, in a person's
	// words — and empty when it did. It is carried verbatim: the sentence was
	// written by whoever knew what ran out, and this store is not entitled to
	// rephrase it.
	Why string `json:"why,omitempty"`
	// CostUSD is the run's ledger, folded to the one figure a row draws. The
	// whole exec.Spend is the journal's; this is the number.
	CostUSD float64 `json:"cost_usd,omitempty"`
}

RunNote is the whole of what this store keeps about a run: when, how it ended, and what it cost.

IT IS NOT A RUN JOURNAL AND MUST NOT GROW INTO ONE. The journal is the runtime's — exec.JournalEntry, written per host call, kept by whoever is watching a run — and the door lane decides where a whole trace lands. What lives here is the one line a list row needs, and keeping it to that is what makes drawing forty rows forty small reads instead of forty directory scans.

It renders NOTHING. The tui lane turns this into the string session.SubharnessRow.LastRun carries, because the emptiness law, the word for an unfinished run, and how a cost is drawn are all that lane's to decide and are all stated in one place there.

type Store

type Store struct {

	// Notice is where this store says that it skipped something. A bundle that
	// does not validate is ABSENT FROM EVERY LIST — the codebase's law about a
	// capability that cannot work — and the line here is the only trace of it,
	// so that "my subharness vanished" is answerable by looking at the journal
	// rather than by guessing. Nil is silence, which is what a store nobody is
	// debugging should be; the surface that owns a journal wires it at launch.
	Notice func(line string)

	// Parse is the syntax check [Store.Mint] runs over program.js before it
	// writes anything. IT IS A FIELD BECAUSE THE PARSER IS THE RUNTIME'S, not
	// this package's: goja is not in this build's module graph, and a store that
	// imported a JavaScript engine to spell-check a file would make every surface
	// that lists subharnesses depend on the engine that runs them. Nil is no
	// check, and a bundle whose program is nonsense is then caught by the runtime
	// at its first run instead of by the mint — later than ideal, and honest,
	// which is the trade this field exists to let the runtime lane close.
	Parse func(program []byte) error

	// Now stamps version records and run notes. It is a field so a test can mint
	// twice in one second and still say which came first.
	Now func() time.Time
	// contains filtered or unexported fields
}

Store is the bundles at one directory.

func At

func At(dir string) *Store

At opens the store at a directory. The directory is created on the first mint and not here, so listing a machine that has never written a subharness is not itself a mutation.

func Home

func Home() *Store

Home opens the store codeaf owns: ~/.codeaf/subharnesses, moved wholesale by CODEAF_HOME like everything else durable, through the one package that reads that variable.

func (*Store) Dir

func (s *Store) Dir() string

Dir names the store's directory.

func (*Store) Head

func (s *Store) Head(name string) (int, error)

Head is the highest complete version. There is no head file; this is the whole of the rule.

func (*Store) LastRun

func (s *Store) LastRun(name string) (RunNote, bool)

LastRun answers the note, and false for a subharness nobody has run.

FALSE IS THE ORDINARY ANSWER AND NOT A FAULT. A subharness nobody has run has no last run, its row draws nothing under the emptiness law, and a store that returned a zero note with a zero time would hand the surface a date in 1970 to render. An unreadable note is the same answer with a line in the journal: the list is not where somebody learns their disk is broken.

func (*Store) Lineage

func (s *Store) Lineage(name string) ([]Version, error)

Lineage is every version of one subharness, oldest first — the log PRD §6 asks a subharness's history to read like.

func (*Store) Load

func (s *Store) Load(name string, version int) (Bundle, error)

Load reads one version of one subharness whole. Version 0 is the head, and any other number is that version and only that version — which is what makes a pin durable: v1 loads the same bytes after v2 is minted.

A BUNDLE THAT DOES NOT VALIDATE IS AN ERROR HERE AND ABSENT ABOVE. This door answers honestly, because whoever called it named a version and deserves to be told what is wrong with it. The doors a person's lists are drawn from — [Store.Runner] and [Store.Manifests] — turn that error into a journal line and an absence instead, which is the codebase's law about a capability that cannot work: absent, never present-and-broken.

func (*Store) Manifest

func (s *Store) Manifest(name string, version int) (exec.Manifest, error)

Manifest reads one version's manifest without its program, for the lists. Reading a whole bundle in order to draw a row would put every program.js in the tree into memory at launch.

func (*Store) Memory

func (s *Store) Memory(name string) *Memory

Memory opens one subharness's memory door.

It is safe to open for a name with no versions and no directory — the file is created on the first note, not here, so recalling from a subharness nobody has run is an empty answer rather than a mutation.

func (*Store) Mint

func (s *Store) Mint(files Files, parent, why string) (Version, error)

Mint writes a new version of a subharness. It is the only door that writes a bundle, and it validates before it writes anything at all, so every version a reader finds is one the runtime can be handed.

VERSIONS ARE MINTED, NEVER CHOSEN (PRD §6). The caller does not say which version it is making; it says which version it MADE THIS FROM, by that version's hash, and one line of why. That is optimistic concurrency in its smallest honest form: a design flow that read v2, thought for a minute and came back to save finds out here that somebody else saved v3 in the meantime, instead of silently minting a v4 that threw their work away.

• parent empty, nothing on disk: this is v1.
• parent empty, versions on disk: refused. Say what this came from.
• parent is the head's hash, and the content is the head's content: nothing changed, so nothing is minted and the head is returned. Minting is IDEMPOTENT, which is what "content-addressed" has to mean for a flow that re-saves an unedited bundle.
• parent is the head's hash, content differs: v(head+1), recording the parent hash and the why.
• parent is anything else: refused, naming the head it disagrees with.
• two mints racing: exactly one wins and every loser is refused, never shifted along to the next number. A loser normally reads the winner's version and is told its parent has moved on; where the gate could not be taken it gets as far as the claim and is told ErrExists. See [Store.gated] for how the race is decided and [Store.write] for why a retry would write a lineage that lies.

A CALLER ON A TURN PATH GETS AN ANSWER WITHIN [mintGateBound], NEVER A WAIT: a mint that cannot have the gate inside that bound is refused with ErrMintBusy and has written nothing, so the caller may say so, or mint again, but it never blocks a turn on somebody else's lock.

Content that an ANCESTOR once had is still a new version. A revert is a real event with its own place in the lineage, and collapsing it onto the version it restored would move the head backwards — which is the one thing the no-head-file rule cannot survive.

func (*Store) Names

func (s *Store) Names() ([]string, error)

Names is every subharness with at least one complete version, in name order.

A DIRECTORY IS A SUBHARNESS WHEN IT HOLDS A VERSION, which is deliberately not a check on the shape of its name. The rule for what a name may be lives in exec.Manifest.Validate and may live nowhere else; restating it here as a directory filter would be a second copy of it, and the manifest inside is validated for real at load anyway — a directory called something impossible simply has no loadable bundle in it.

func (*Store) Record

func (s *Store) Record(name string, version int) (Version, error)

Record reads one version's record. Version 0 is the head.

func (*Store) RecordRun

func (s *Store) RecordRun(name string, note RunNote) error

RecordRun writes the last-run note for one subharness, replacing whatever was there.

ONE NOTE PER NAME, OVERWRITTEN. It is called "last run" and it holds the last run; a history would be a different feature with a different door, and building the history first and calling it this is how a small file becomes an unbounded one nobody pruned. The write is atomic (see [replace]) rather than exclusive, because unlike a version this file is MEANT to change — the exclusive create that protects a version page would fail on the second run of every subharness.

A note for a name with no bundle is still written. That is deliberate: a Go-native subharness has no directory here until its first run, and the list row for one wants the same "when did I last run this" the stored ones get.

func (*Store) Source

func (s *Store) Source(build Build) exec.BundleSource

Source is this store seen as a place the registry loads bundles from — exec.BundleSource, and the door in is Registry.UseBundles(layer, source).

A STORE WITH NO RUNTIME IS NOT A SOURCE AT ALL, which is why this returns a nil interface when handed no builder rather than a source whose every lookup fails. That is the codebase's law about a capability that cannot work stated at the earliest possible place: a build with no JavaScript runtime in it offers no bundles, so a person is shown nothing about them instead of a list of names that refuse to run. Registry.UseBundles takes a nil source as a no-op, so the wiring reads the same either way:

registry.UseBundles(exec.LayerHome, substore.Home().Source(runtime.Build))

func (*Store) VersionDir

func (s *Store) VersionDir(name string, version int) string

VersionDir names one version's bundle directory. It is exported because a bundle's prompts and its program are files a runner opens by path, and a runtime that had to reconstruct this path itself would be a second copy of the layout this package is the authority on.

func (*Store) Versions

func (s *Store) Versions(name string) ([]int, error)

Versions lists one subharness's complete versions, ascending.

COMPLETE MEANS BOTH HALVES ARE THERE — the record and the directory it describes. A record with no directory is the one crash window Store.Mint has: the claim was linked and the process died before the bundle was renamed into place. Such a version is dead rather than empty, it is never re-minted (the next version is counted from the records, not from this list), and leaving it out here is what keeps a reader from ever seeing half a mint.

type Version

type Version struct {
	// Version is the page number, which is also the directory name.
	Version int `json:"version"`
	// Hash is the content address: sha256 over the bundle's files, canonically
	// ordered. See [hashFiles] for exactly what is covered.
	Hash string `json:"hash"`
	// Parent is the hash of the version this one was minted from, and empty for
	// the first.
	Parent string `json:"parent,omitempty"`
	// ParentVersion is that same version's number, carried for readability. The
	// hash is the authority; this is so a person reading the file does not have
	// to grep for it.
	ParentVersion int `json:"parent_version,omitempty"`
	// Why is one line: what changed and what it was for. It is required of every
	// version that has a parent, because a lineage of unexplained versions is a
	// list of hashes and not a log.
	Why string `json:"why,omitempty"`
	// At is when it was minted, UTC.
	At time.Time `json:"at"`
}

Version is one version's record — the v<N>.json beside the v<N>/ it describes.

A SUBHARNESS'S LINEAGE READS LIKE A LOG (PRD §6) and these three fields are what make it one: what this version is, what it came from, and why somebody made it. The parent is a HASH and not a version number, because a number says only "the one before" and a hash says "these exact bytes" — which is the question anybody reading a lineage after a directory has been copied, restored, or pulled actually has.