Documentation
¶
Overview ¶
Package brain gives each agent its own memory.
Every agent gets a private namespace plus a read-only grant on the shared project brain. That split is the whole design: an agent can learn what the team knows without being able to rewrite it, so one agent's wrong conclusion never becomes every agent's premise.
The default driver is pgvector in the app's own Postgres rather than a hosted service. A day-0 harness cannot have a hard dependency on an endpoint that might be down — and during research the hosted cabrain instance answered 502 on two endpoints, ran on a single workstation with no HA, and resolved tokenless callers as admin.
Index ¶
- Constants
- Variables
- func Chunk(text string, opt ChunkOptions) []string
- func DocSourceRef(name string, i int) string
- func DocumentName(m Memory) string
- func Extract(d Document) (string, error)
- func FleetName(ctx context.Context, db *sql.DB) string
- func IsProjectNamespace(ns string) bool
- func ProjectNamespace(fleet string) string
- type ChunkOptions
- type Document
- type Embedder
- type GraphEdge
- type GraphNode
- type HashEmbedder
- type IngestResult
- type Memory
- type Provenance
- type Store
- func (s *Store) EnsureProjectBrain(ctx context.Context, fleet string) string
- func (s *Store) Forget(ctx context.Context, id, agentSlug string) error
- func (s *Store) ForgetDocument(ctx context.Context, ns, name string) (int, error)
- func (s *Store) GrantProjectRead(ctx context.Context, ns, agentSlug string) error
- func (s *Store) Graph(ctx context.Context, namespace string, maxNodes int) ([]GraphNode, []GraphEdge, error)
- func (s *Store) IngestDocument(ctx context.Context, ns string, d Document, opt ChunkOptions) (IngestResult, error)
- func (s *Store) ProjectNamespaceFor(ctx context.Context) string
- func (s *Store) ReadableNamespaces(ctx context.Context, agentSlug string) (string, []string, error)
- func (s *Store) Recall(ctx context.Context, agentSlug, query string, limit int) ([]Memory, error)
- func (s *Store) Retain(ctx context.Context, ns, content, sourceKind, sourceRef string, ...) (string, error)
- func (s *Store) RetainProject(ctx context.Context, fleet, content, sourceKind, sourceRef string, ...) (string, error)
- func (s *Store) Routes(r chi.Router)
- func (s *Store) Writable(ctx context.Context, agentSlug string) (string, error)
Constants ¶
const ( DefaultChunkRunes = 1200 DefaultOverlapRunes = 200 )
Defaults, in runes rather than tokens because the embedder seam takes text.
1200 is roughly two paragraphs of prose: big enough that a claim and its qualification stay together, small enough that the vector is about one thing. 200 of overlap is about one sentence of run-up, which is what it takes for a sentence beginning "It must therefore…" to still have its antecedent.
const ( KindText = "text" KindCSV = "csv" KindPDF = "pdf" KindImage = "image" )
Document kinds. The kind decides the extractor, not the file extension.
const Dim = 1024
Dim is the embedding width. It must match the vector column in migration 0002 — a mismatch fails at insert time with a confusing cast error, so it is validated at boot instead.
const DocSourceKind = "document"
DocSourceKind is the source_kind every document chunk carries, so a citation can say "this came from an uploaded document" and a cleanup can find them all.
const MaxDistance = 0.85
MaxDistance is the cosine-distance cutoff for a vector hit.
Without it, vector search returns the top-k no matter how bad the match is, so a query about something the brain has never heard of still "succeeds" — the agent gets confidently irrelevant memories and no gap is ever recorded.
Measured with the default hash embedder on real rows: an identical document scores 0.000, a related one 0.705, and an unrelated one 1.000 (orthogonal — no shared tokens). 0.85 separates those cleanly.
This value is EMBEDDER-DEPENDENT. A semantic model produces a completely different distance distribution, so re-measure when swapping one in.
const MaxDocumentBytes = 25 << 20 // 25 MB
MaxDocumentBytes is the largest upload this package will read. It matches the attachment ceiling in internal/issues so a file that was accepted on upload cannot then be refused on ingestion.
const MaxDocumentChunks = 5000
MaxDocumentChunks bounds how many memories one upload may create.
With the default chunk size the rune cap binds first and this never fires; it exists for a caller that passes a tiny MaxRunes, where one file could otherwise become tens of thousands of rows in the brain every agent reads. It also keeps every index inside the width DocSourceRef pads to.
const MaxDocumentRunes = 2 << 20 // ~2 M runes
MaxDocumentRunes caps extracted text. A 25 MB text file is legal input and would otherwise become ~25 M runes of chunking work and several thousand embedder calls from a single upload.
const ProjectSlot = "project"
ProjectSlot is what a project brain puts where an agent brain puts its slug.
A project namespace is `<fleet>:project` — the same shape as `<fleet>:<slug>`, not a second convention. Three spellings were live before this constant existed: the issue said `project:<name>`, generate.go built `<fleet>:project`, and agents_api.go hardcoded the fleet as `default`. They disagreed about which string a grant pointed at, which is the whole reason the shared brain was unreachable. Everything that needs the name now comes through here.
`project` is a legal agent slug, so the namespaces could in principle collide with an agent actually called "project". builder_brains.namespace is UNIQUE, so that collision is a loud insert failure rather than two brains quietly sharing memories.
Variables ¶
var ErrNoText = errors.New("no extractable text")
ErrNoText means the file was understood but carries nothing worth retaining: a scanned PDF with no text layer, an empty file, an image with no caption.
It is deliberately distinct from a parse failure. "This PDF is images of paper" is a fact an operator can act on (caption it, or re-export it); "this PDF is malformed" is not the same problem and should not print the same message.
Functions ¶
func Chunk ¶
func Chunk(text string, opt ChunkOptions) []string
Chunk splits text into overlapping pieces that end on sentence boundaries.
Guarantees, all covered by tests:
- no chunk exceeds MaxRunes, unless a single word does
- a chunk never ends mid-sentence while a sentence boundary was available
- consecutive chunks overlap when OverlapRunes > 0
- the concatenation of the chunks contains every sentence of the input
func DocSourceRef ¶
DocSourceRef is the source_ref of one chunk: stable across re-uploads, so chunk 3 of a revised document updates chunk 3 rather than adding a row.
Zero-padded to a FIXED width, which is load-bearing twice over: it is what an operator reads in a citation (#10 must not sort before #2), and the prune below finds a shorter document's leftovers by string comparison, which is only the numeric order while every index is the same width.
func DocumentName ¶
DocumentName recovers the document a memory came from, for a citation. Returns "" for a memory that is not a document chunk.
func Extract ¶
Extract turns a document into the text that will be retained.
The returned string is normalised (LF newlines, no control characters) and credential-scrubbed. Scrubbing happens HERE rather than at the call site because this is the last point where the whole document is in one place: a key split across two chunks is a key nobody redacts. See redact.go.
func FleetName ¶
FleetName is the current fleet, or "default" when the wizard has not run.
Ordered by created_at like the wizard's own lookup, so both agree on which fleet is "the" fleet when someone has generated twice.
func IsProjectNamespace ¶
IsProjectNamespace reports whether a namespace is a project brain rather than an agent's private one. It matches the CHECK constraint in migration 0010.
func ProjectNamespace ¶
ProjectNamespace is the project brain for a fleet.
Types ¶
type ChunkOptions ¶
type ChunkOptions struct {
// MaxRunes is the ceiling for one chunk.
MaxRunes int
// OverlapRunes is how much of the tail of a chunk is repeated at the head
// of the next. Clamped to half of MaxRunes — at more than that the overlap
// consumes the chunk and ingestion never terminates.
OverlapRunes int
}
ChunkOptions tunes the cut. The zero value is the default, which is what every caller should use until there is a measurement saying otherwise.
type Document ¶
type Document struct {
// Name is the file's name as an operator would recognise it, and it is
// IDENTITY: re-ingesting the same name replaces that document's chunks.
// It is never used to build a filesystem path.
Name string
// Mime is the sniffed content type. Authoritative when present; the
// extension is only consulted when it is empty or too vague to act on.
Mime string
// Caption is operator-supplied prose. The only text an image has until
// there is an OCR step.
Caption string
Data []byte
}
Document is one uploaded file on its way into the brain.
type Embedder ¶
type Embedder interface {
Embed(ctx context.Context, texts []string) ([][]float32, error)
Dimensions() int
Name() string
}
Embedder turns text into a vector. The seam exists so a project can swap in a real model without the store knowing anything about it.
func EmbedderFromEnv ¶
func EmbedderFromEnv() Embedder
EmbedderFromEnv returns the configured embedder, or nil when none is set.
Returning nil rather than an error, and nil rather than a fallback: the caller decides what to do without an embedder, and a silent fallback to the hash embedder would mean an operator who typed the URL wrong gets keyword search that looks exactly like semantic search. The caller logs which one it got, so the answer to "is this real recall?" is in the boot log.
type GraphNode ¶
type GraphNode struct {
ID string `json:"id"`
Name string `json:"name"`
Kind string `json:"kind"`
Mentions int `json:"mentions"`
}
GraphNode and GraphEdge are the shape the UI draws.
type HashEmbedder ¶
type HashEmbedder struct{}
HashEmbedder is a deterministic local embedder: hashed bag-of-words projected into Dim and L2-normalized.
It is NOT a semantic model — it captures lexical overlap, not meaning. It exists so the brain works out of the box with no API key and no network, and so the hybrid recall path is exercised from day one. A real project swaps in a proper embedder through the same interface; the store never knows.
L2-normalized because pgvector's cosine operator assumes it, and unnormalized vectors make distance depend on document length rather than content.
func (HashEmbedder) Dimensions ¶
func (HashEmbedder) Dimensions() int
func (HashEmbedder) Name ¶
func (HashEmbedder) Name() string
type IngestResult ¶
type IngestResult struct {
Namespace string `json:"namespace"`
Document string `json:"document"`
Kind string `json:"kind"`
Runes int `json:"runes"`
Chunks int `json:"chunks"`
// Removed counts chunks of a PREVIOUS version of this document that the new
// one no longer has. Non-zero means the document got shorter.
Removed int `json:"removed"`
MemoryIDs []string `json:"memoryIds"`
}
IngestResult is what one ingestion did, for the log line and the API reply.
type Memory ¶
type Memory struct {
ID string `json:"id"`
Namespace string `json:"namespace"`
Content string `json:"content"`
SourceKind string `json:"sourceKind"`
SourceRef string `json:"sourceRef"`
Importance float64 `json:"importance"`
Score float64 `json:"score,omitempty"`
CreatedAt string `json:"createdAt"`
}
type Provenance ¶
type Provenance struct {
// Kind is the connector or origin: github, rss, crawl, slack, sql,
// document, issue, agent.
Kind string `json:"kind"`
// Source names the specific thing — the repository, the feed URL, the
// document's filename.
Source string `json:"source"`
// Ref is the item within that source: a path, an entry id, a chunk.
Ref string `json:"ref"`
// Label is the one-line rendering, so every surface says it the same way.
Label string `json:"label"`
}
Provenance is where one memory came from, in a form a screen can render.
func ProvenanceFor ¶
func ProvenanceFor(kind, ref string) Provenance
provenanceOf reads a memory's origin out of the fields it already carries.
No new table, and none needed: sources write `source:<kind>:<name>:<ref>` and documents write `doc:<name>#<index>`, which between them name the connector, the specific source and the item. The decision to parse rather than join was made deliberately — a join would need a foreign key onto a source row that may since have been deleted, and a memory outliving its pipe is the normal case, not an error. ProvenanceFor is exported for the chat surface, which cites the same origins on screen that this package renders in the brain view. One implementation, so a memory is described identically wherever it appears.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
func (*Store) EnsureProjectBrain ¶
EnsureProjectBrain creates the project brain row if it is missing and returns its namespace. Idempotent, so both the wizard and the hire path can call it without either owning it.
It is deliberately NOT an error for the row to be absent-and-uncreatable: on a database that has not run migration 0010 the INSERT fails the NOT NULL on agent_slug, and the caller still gets the namespace back. A fleet that cannot count its project memories yet is a smaller problem than a wizard run that refuses to finish.
func (*Store) Forget ¶
Forget marks a memory invalid rather than deleting it, so a wrong conclusion leaves a trace of having been believed.
func (*Store) ForgetDocument ¶
ForgetDocument removes every chunk of a document, for when the underlying file is deleted from the library.
The uploaded file and its chunks are one thing in an operator's head. Deleting the file and leaving the chunks means the brain keeps answering from a document that is no longer there and cannot be re-read to check.
func (*Store) GrantProjectRead ¶
GrantProjectRead gives an agent read-only access to the project brain.
Read-only is not a default that a caller may override: it is the invariant the whole split exists for. An agent that could write the shared brain would make its own wrong conclusion every other agent's premise, which is exactly what separate namespaces were built to prevent.
func (*Store) Graph ¶
func (s *Store) Graph(ctx context.Context, namespace string, maxNodes int) ([]GraphNode, []GraphEdge, error)
Graph returns the strongest part of a brain's entity graph.
Capped hard: a graph with 400 nodes is not a picture, it is a hairball. The most-mentioned entities and the heaviest edges between them are what an operator can actually read.
func (*Store) IngestDocument ¶
func (s *Store) IngestDocument(ctx context.Context, ns string, d Document, opt ChunkOptions) (IngestResult, error)
IngestDocument extracts a document, chunks it, and retains every chunk.
Re-ingesting the same Document.Name REPLACES that document rather than duplicating it: chunk N upserts on (namespace, source_ref), and any chunk the new version does not have is deleted. Without the second half a spec that loses a section keeps answering from the section it lost, which is worse than not having ingested it — the stale chunk is indistinguishable from a current one and outranks nothing.
func (*Store) ProjectNamespaceFor ¶
ProjectNamespaceFor is the project brain this installation reads.
func (*Store) ReadableNamespaces ¶
ReadableNamespaces is what an agent may recall from: its own brain first, then every namespace it has been granted. Exposed so a screen can show an agent which brains it reads without duplicating the grant logic.
func (*Store) Recall ¶
Recall searches the agent's own namespace plus anything it has been granted — which now includes the shared project brain, so a run sees what the team knows alongside what it worked out itself.
Hybrid: vector similarity when an embedder is configured, full-text always, fused with reciprocal rank. Keyword-only recall misses paraphrase; vector-only recall misses exact identifiers like a function name — and agents search for identifiers constantly.
func (*Store) Retain ¶
func (s *Store) Retain(ctx context.Context, ns, content, sourceKind, sourceRef string, importance float64) (string, error)
Retain writes a memory.
source_ref is identity: retaining the same source twice updates in place rather than growing a duplicate. Without it an agent that re-reads a file on every run accumulates one copy per run and its recall degrades into noise.
func (*Store) RetainProject ¶
func (s *Store) RetainProject(ctx context.Context, fleet, content, sourceKind, sourceRef string, importance float64) (string, error)
RetainProject writes to the project brain. This is the INGESTION path, and it is the only way anything writes there.
It takes no agent slug, and that is the point. `Retain` is reached by agents through Writable(), which resolves a namespace by agent_slug — a project brain has none, so no agent can route a write here however it is called. Ingestion (sources, webhooks, documents) has no agent to be, and calls this instead.
sourceRef is required. An agent memory without one is merely a duplicate risk; a project memory without one is unattributable — nobody reading the shared brain can tell where a claim about the project came from, and nobody can re-ingest a corrected version over it.
Source Files