brain

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package brain gives each agent its own memory.

Every agent gets a private namespace plus a read-only grant on the shared project brain. That split is the whole design: an agent can learn what the team knows without being able to rewrite it, so one agent's wrong conclusion never becomes every agent's premise.

The default driver is pgvector in the app's own Postgres rather than a hosted service. A day-0 harness cannot have a hard dependency on an endpoint that might be down — and during research the hosted cabrain instance answered 502 on two endpoints, ran on a single workstation with no HA, and resolved tokenless callers as admin.

Index

Constants

View Source 
const (
	DefaultChunkRunes   = 1200
	DefaultOverlapRunes = 200
)

Defaults, in runes rather than tokens because the embedder seam takes text.

1200 is roughly two paragraphs of prose: big enough that a claim and its qualification stay together, small enough that the vector is about one thing. 200 of overlap is about one sentence of run-up, which is what it takes for a sentence beginning "It must therefore…" to still have its antecedent.

View Source 
const (
	KindText  = "text"
	KindCSV   = "csv"
	KindPDF   = "pdf"
	KindImage = "image"
)

Document kinds. The kind decides the extractor, not the file extension.

View Source 
const DefaultCandidates = 50

DefaultCandidates is how many rows the hybrid search hands to the reranker.

Fifty because it is what the two arms of the fused query already fetch, so the default costs one extra HTTP call and no extra database work. Recall@50 on a bi-encoder is high enough that the right row is nearly always inside it; raising this buys accuracy at a linear cost in reranker time, and lowering it below `limit` would silently truncate results, which is why it is clamped.

View Source 
const Dim = 1024

Dim is the embedding width. It must match the vector column in migration 0002 — a mismatch fails at insert time with a confusing cast error, so it is validated at boot instead.

View Source 
const DocSourceKind = "document"

DocSourceKind is the source_kind every document chunk carries, so a citation can say "this came from an uploaded document" and a cleanup can find them all.

View Source 
const MaxCandidates = 200

MaxCandidates bounds it. A cross-encoder is a forward pass per candidate, so an operator who sets 5000 has configured a timeout, not a search.

View Source 
const MaxDistance = 0.85

MaxDistance is the cosine-distance cutoff for a vector hit.

Without it, vector search returns the top-k no matter how bad the match is, so a query about something the brain has never heard of still "succeeds" — the agent gets confidently irrelevant memories and no gap is ever recorded.

Measured with the default hash embedder on real rows: an identical document scores 0.000, a related one 0.705, and an unrelated one 1.000 (orthogonal — no shared tokens). 0.85 separates those cleanly.

This value is EMBEDDER-DEPENDENT. A semantic model produces a completely different distance distribution, so re-measure when swapping one in.

View Source 
const MaxDocumentBytes = 25 << 20 // 25 MB

MaxDocumentBytes is the largest upload this package will read. It matches the attachment ceiling in internal/issues so a file that was accepted on upload cannot then be refused on ingestion.

View Source 
const MaxDocumentChunks = 5000

MaxDocumentChunks bounds how many memories one upload may create.

With the default chunk size the rune cap binds first and this never fires; it exists for a caller that passes a tiny MaxRunes, where one file could otherwise become tens of thousands of rows in the brain every agent reads. It also keeps every index inside the width DocSourceRef pads to.

View Source 
const MaxDocumentRunes = 2 << 20 // ~2 M runes

MaxDocumentRunes caps extracted text. A 25 MB text file is legal input and would otherwise become ~25 M runes of chunking work and several thousand embedder calls from a single upload.

View Source 
const ProjectSlot = "project"

ProjectSlot is what a project brain puts where an agent brain puts its slug.

A project namespace is `<fleet>:project` — the same shape as `<fleet>:<slug>`, not a second convention. Three spellings were live before this constant existed: the issue said `project:<name>`, generate.go built `<fleet>:project`, and agents_api.go hardcoded the fleet as `default`. They disagreed about which string a grant pointed at, which is the whole reason the shared brain was unreachable. Everything that needs the name now comes through here.

`project` is a legal agent slug, so the namespaces could in principle collide with an agent actually called "project". builder_brains.namespace is UNIQUE, so that collision is a loud insert failure rather than two brains quietly sharing memories.

View Source 
const SemanticMaxDistance = 0.50

SemanticMaxDistance is the same cutoff for a real embedding model.

Re-measured, as the note above demands, against bge-m3 on the sentences the package documentation uses as its example: 

identical text                                        0.000
"authentication fails" ~ "the login button is broken"  0.223
"authentication fails" ~ "تعذر تسجيل الدخول"           0.189   (cross-lingual)
"authentication fails" ~ "the deploy pipeline runs…"   0.540
"authentication fails" ~ "bananas are yellow"          0.547

A real model's distances are COMPRESSED compared to the hash embedder's, which scores unrelated text at 1.000 because it shares no tokens. Nothing a transformer embeds is orthogonal to anything else, so unrelated text lands near 0.55 — and the inherited 0.85 cutoff, applied to bge-m3, admits every row in the table. The floor stops being a floor, no query ever comes back empty, and no gap is ever recorded.

0.50 sits below the unrelated band and well above the related one. It is a coarse pre-filter, not the ranking: the cross-encoder is what decides the order of what survives it, so this only has to exclude the obviously wrong. Override with BUILDER_RECALL_MAX_DISTANCE after measuring your own corpus — long chunks score differently from the short sentences above.

Variables

View Source 
var ErrNoText = errors.New("no extractable text")

ErrNoText means the file was understood but carries nothing worth retaining: a scanned PDF with no text layer, an empty file, an image with no caption.

It is deliberately distinct from a parse failure. "This PDF is images of paper" is a fact an operator can act on (caption it, or re-export it); "this PDF is malformed" is not the same problem and should not print the same message.

Functions

func BackfillEnabled added in v0.2.0 

func BackfillEnabled() bool

BackfillEnabled reports whether boot should start a backfill. On by default: the whole point of configuring a real embedder is that recall gets better, and an operator who has to find and run a separate command to make the memories they already have participate will reasonably conclude it did not work.

func BackfillPause added in v0.2.0 

func BackfillPause() time.Duration

BackfillPause is the gap between batches.

Non-zero by default. The endpoint that serves the backfill also serves live recall, and a backfill that saturates it turns "recall got better" into "recall got slow" on the day it is switched on. 250ms across 64-row batches drains ten thousand memories in about a minute while leaving the model mostly idle for real queries.

func Candidates added in v0.2.0 

func Candidates(limit int) int

Candidates is how many rows to fetch before reranking, clamped to something that can be served. Never below `limit`, or recall would return fewer results with the reranker on than without it.

func Chunk 

func Chunk(text string, opt ChunkOptions) []string

Chunk splits text into overlapping pieces that end on sentence boundaries.

Guarantees, all covered by tests:

  • no chunk exceeds MaxRunes, unless a single word does
  • a chunk never ends mid-sentence while a sentence boundary was available
  • consecutive chunks overlap when OverlapRunes > 0
  • the concatenation of the chunks contains every sentence of the input

func DocSourceRef 

func DocSourceRef(name string, i int) string

DocSourceRef is the source_ref of one chunk: stable across re-uploads, so chunk 3 of a revised document updates chunk 3 rather than adding a row.

Zero-padded to a FIXED width, which is load-bearing twice over: it is what an operator reads in a citation (#10 must not sort before #2), and the prune below finds a shorter document's leftovers by string comparison, which is only the numeric order while every index is the same width.

func DocumentName 

func DocumentName(m Memory) string

DocumentName recovers the document a memory came from, for a citation. Returns "" for a memory that is not a document chunk.

func Extract 

func Extract(d Document) (string, error)

Extract turns a document into the text that will be retained.

The returned string is normalised (LF newlines, no control characters) and credential-scrubbed. Scrubbing happens HERE rather than at the call site because this is the last point where the whole document is in one place: a key split across two chunks is a key nobody redacts. See redact.go.

func FleetName 

func FleetName(ctx context.Context, db *sql.DB) string

FleetName is the current fleet, or "default" when the wizard has not run.

Ordered by created_at like the wizard's own lookup, so both agree on which fleet is "the" fleet when someone has generated twice.

func IsProjectNamespace 

func IsProjectNamespace(ns string) bool

IsProjectNamespace reports whether a namespace is a project brain rather than an agent's private one. It matches the CHECK constraint in migration 0010.

func IsSemantic added in v0.2.0 

func IsSemantic(e Embedder) bool

IsSemantic reports whether recall over this embedder means anything.

Three things can make it false and only one of them is a configuration choice: no embedder at all, the hash embedder (lexical overlap, no meaning), or a real embedder whose endpoint has stopped answering. A screen that says "semantic" has to be wrong in none of those cases, so the check lives here rather than as an `.(HashEmbedder)` assertion repeated at each call site — there were two, and neither noticed the third case.

func Probe added in v0.2.0 

func Probe(ctx context.Context, e Embedder) error

Probe runs an embedder's boot check when it has one. HashEmbedder does not, and does not need one.

func ProjectNamespace 

func ProjectNamespace(fleet string) string

ProjectNamespace is the project brain for a fleet.

Types

type BackfillResult added in v0.2.0 

type BackfillResult struct {
	Scanned int
	Written int
	Failed  int
}

BackfillResult is what one backfill pass did.

type ChunkOptions 

type ChunkOptions struct {
	// MaxRunes is the ceiling for one chunk.
	MaxRunes int
	// OverlapRunes is how much of the tail of a chunk is repeated at the head
	// of the next. Clamped to half of MaxRunes — at more than that the overlap
	// consumes the chunk and ingestion never terminates.
	OverlapRunes int
}

ChunkOptions tunes the cut. The zero value is the default, which is what every caller should use until there is a measurement saying otherwise.

type Document 

type Document struct {
	// Name is the file's name as an operator would recognise it, and it is
	// IDENTITY: re-ingesting the same name replaces that document's chunks.
	// It is never used to build a filesystem path.
	Name string
	// Mime is the sniffed content type. Authoritative when present; the
	// extension is only consulted when it is empty or too vague to act on.
	Mime string
	// Caption is operator-supplied prose. The only text an image has until
	// there is an OCR step.
	Caption string
	Data    []byte
}

Document is one uploaded file on its way into the brain.

func (Document) Kind 

func (d Document) Kind() string

Kind reports which extractor a document will go through.

type Embedder 

type Embedder interface {
	Embed(ctx context.Context, texts []string) ([][]float32, error)
	Dimensions() int
	Name() string
}

Embedder turns text into a vector. The seam exists so a project can swap in a real model without the store knowing anything about it.

func EmbedderFromEnv 

func EmbedderFromEnv() Embedder

EmbedderFromEnv returns the configured embedder, or nil when none is set.

Returning nil rather than an error, and nil rather than a fallback: the caller decides what to do without an embedder, and a silent fallback to the hash embedder would mean an operator who typed the URL wrong gets keyword search that looks exactly like semantic search. The caller logs which one it got, so the answer to "is this real recall?" is in the boot log.

type GraphEdge 

type GraphEdge struct {
	From   string `json:"from"`
	To     string `json:"to"`
	Weight int    `json:"weight"`
}

type GraphNode 

type GraphNode struct {
	ID       string `json:"id"`
	Name     string `json:"name"`
	Kind     string `json:"kind"`
	Mentions int    `json:"mentions"`
}

GraphNode and GraphEdge are the shape the UI draws.

type HashEmbedder 

type HashEmbedder struct{}

HashEmbedder is a deterministic local embedder: hashed bag-of-words projected into Dim and L2-normalized.

It is NOT a semantic model — it captures lexical overlap, not meaning. It exists so the brain works out of the box with no API key and no network, and so the hybrid recall path is exercised from day one. A real project swaps in a proper embedder through the same interface; the store never knows.

L2-normalized because pgvector's cosine operator assumes it, and unnormalized vectors make distance depend on document length rather than content.

func (HashEmbedder) Dimensions 

func (HashEmbedder) Dimensions() int

func (HashEmbedder) Embed 

func (h HashEmbedder) Embed(_ context.Context, texts []string) ([][]float32, error)

func (HashEmbedder) Name 

func (HashEmbedder) Name() string

type IngestResult 

type IngestResult struct {
	Namespace string `json:"namespace"`
	Document  string `json:"document"`
	Kind      string `json:"kind"`
	Runes     int    `json:"runes"`
	Chunks    int    `json:"chunks"`
	// Removed counts chunks of a PREVIOUS version of this document that the new
	// one no longer has. Non-zero means the document got shorter.
	Removed   int      `json:"removed"`
	MemoryIDs []string `json:"memoryIds"`
}

IngestResult is what one ingestion did, for the log line and the API reply.

type Memory 

type Memory struct {
	ID         string  `json:"id"`
	Namespace  string  `json:"namespace"`
	Content    string  `json:"content"`
	SourceKind string  `json:"sourceKind"`
	SourceRef  string  `json:"sourceRef"`
	Importance float64 `json:"importance"`
	Score      float64 `json:"score,omitempty"`
	CreatedAt  string  `json:"createdAt"`
}

type Provenance 

type Provenance struct {
	// Kind is the connector or origin: github, rss, crawl, slack, sql,
	// document, issue, agent.
	Kind string `json:"kind"`
	// Source names the specific thing — the repository, the feed URL, the
	// document's filename.
	Source string `json:"source"`
	// Ref is the item within that source: a path, an entry id, a chunk.
	Ref string `json:"ref"`
	// Label is the one-line rendering, so every surface says it the same way.
	Label string `json:"label"`
}

Provenance is where one memory came from, in a form a screen can render.

func ProvenanceFor 

func ProvenanceFor(kind, ref string) Provenance

provenanceOf reads a memory's origin out of the fields it already carries.

No new table, and none needed: sources write `source:<kind>:<name>:<ref>` and documents write `doc:<name>#<index>`, which between them name the connector, the specific source and the item. The decision to parse rather than join was made deliberately — a join would need a foreign key onto a source row that may since have been deleted, and a memory outliving its pipe is the normal case, not an error. ProvenanceFor is exported for the chat surface, which cites the same origins on screen that this package renders in the brain view. One implementation, so a memory is described identically wherever it appears.

type RerankHit added in v0.2.0 

type RerankHit struct {
	Index int     `json:"index"`
	Score float64 `json:"score"`
}

RerankHit is one scored candidate, referring to the input by index.

type Reranker added in v0.2.0 

type Reranker interface {
	Rerank(ctx context.Context, query string, texts []string) ([]RerankHit, error)
	Name() string
}

Reranker orders candidate texts against a query. A separate seam from Embedder because the two are independently available: an operator can have a working embedder and no reranker, and recall must still be better than it was.

func RerankerFromEnv added in v0.2.0 

func RerankerFromEnv(embedURL string) Reranker

RerankerFromEnv returns the configured reranker, or nil.

embedURL is where the default comes from: the reference deployment puts the embedding and reranking models behind one host, so an operator who has configured embeddings has almost certainly got reranking at /rerank on the same origin. Derived rather than assumed silently — the caller logs the URL it ended up with, and a derived URL that 404s disables reranking with a warning rather than failing recall.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

func New 

func New(db *sql.DB, log *slog.Logger, emb Embedder) (*Store, error)

func (*Store) BackfillEmbeddings added in v0.2.0 

func (s *Store) BackfillEmbeddings(ctx context.Context, pause time.Duration) (BackfillResult, error)

BackfillEmbeddings rewrites every memory whose vector did not come from the current embedder, until there are none left or ctx ends.

Returns the counts rather than an error for a partial run: a backfill that re-embedded 900 of 1000 rows and then lost the endpoint has done 900 rows of real good, and reporting that as a failure would invite a caller to discard it and start over.

func (*Store) Embedder added in v0.2.0 

func (s *Store) Embedder() Embedder

Embedder is what this store embeds with, for a surface that reports it.

func (*Store) EmbedderName added in v0.2.0 

func (s *Store) EmbedderName() string

EmbedderName is the embedder's name, or "none" — a store can be constructed without one and s.emb.Name() on a nil interface panics.

func (*Store) EnsureProjectBrain 

func (s *Store) EnsureProjectBrain(ctx context.Context, fleet string) string

EnsureProjectBrain creates the project brain row if it is missing and returns its namespace. Idempotent, so both the wizard and the hire path can call it without either owning it.

It is deliberately NOT an error for the row to be absent-and-uncreatable: on a database that has not run migration 0010 the INSERT fails the NOT NULL on agent_slug, and the caller still gets the namespace back. A fleet that cannot count its project memories yet is a smaller problem than a wizard run that refuses to finish.

func (*Store) Forget 

func (s *Store) Forget(ctx context.Context, id, agentSlug string) error

Forget marks a memory invalid rather than deleting it, so a wrong conclusion leaves a trace of having been believed.

func (*Store) ForgetDocument 

func (s *Store) ForgetDocument(ctx context.Context, ns, name string) (int, error)

ForgetDocument removes every chunk of a document, for when the underlying file is deleted from the library.

The uploaded file and its chunks are one thing in an operator's head. Deleting the file and leaving the chunks means the brain keeps answering from a document that is no longer there and cannot be re-read to check.

func (*Store) GrantProjectRead 

func (s *Store) GrantProjectRead(ctx context.Context, ns, agentSlug string) error

GrantProjectRead gives an agent read-only access to the project brain.

Read-only is not a default that a caller may override: it is the invariant the whole split exists for. An agent that could write the shared brain would make its own wrong conclusion every other agent's premise, which is exactly what separate namespaces were built to prevent.

func (*Store) Graph 

func (s *Store) Graph(ctx context.Context, namespace string, maxNodes int) ([]GraphNode, []GraphEdge, error)

Graph returns the strongest part of a brain's entity graph.

Capped hard: a graph with 400 nodes is not a picture, it is a hairball. The most-mentioned entities and the heaviest edges between them are what an operator can actually read.

func (*Store) IngestDocument 

func (s *Store) IngestDocument(ctx context.Context, ns string, d Document, opt ChunkOptions) (IngestResult, error)

IngestDocument extracts a document, chunks it, and retains every chunk.

Re-ingesting the same Document.Name REPLACES that document rather than duplicating it: chunk N upserts on (namespace, source_ref), and any chunk the new version does not have is deleted. Without the second half a spec that loses a section keeps answering from the section it lost, which is worse than not having ingested it — the stale chunk is indistinguishable from a current one and outranks nothing.

func (*Store) PendingReembed added in v0.2.0 

func (s *Store) PendingReembed(ctx context.Context) (int, error)

PendingReembed counts memories not yet on the current embedder, so a boot log or a status page can say how much of the brain semantic recall can currently see.

func (*Store) ProjectNamespaceFor 

func (s *Store) ProjectNamespaceFor(ctx context.Context) string

ProjectNamespaceFor is the project brain this installation reads.

func (*Store) ReadableNamespaces 

func (s *Store) ReadableNamespaces(ctx context.Context, agentSlug string) (string, []string, error)

ReadableNamespaces is what an agent may recall from: its own brain first, then every namespace it has been granted. Exposed so a screen can show an agent which brains it reads without duplicating the grant logic.

func (*Store) Recall 

func (s *Store) Recall(ctx context.Context, agentSlug, query string, limit int) ([]Memory, error)

Recall searches the agent's own namespace plus anything it has been granted — which now includes the shared project brain, so a run sees what the team knows alongside what it worked out itself.

Hybrid: vector similarity when an embedder is configured, full-text always, fused with reciprocal rank. Keyword-only recall misses paraphrase; vector-only recall misses exact identifiers like a function name — and agents search for identifiers constantly.

Then RERANKED, when a cross-encoder is configured. The fused query is a recall-oriented first pass: it is asked for a wide candidate set and judged on whether the right row is anywhere in it, not on where. The cross-encoder reads each candidate against the query and decides the order that is actually returned. See rerank.go for why the two passes cannot be one.

func (*Store) RerankerName added in v0.2.0 

func (s *Store) RerankerName() string

RerankerName is the reranker's name, or "" when recall returns the fused order unchanged.

func (*Store) Retain 

func (s *Store) Retain(ctx context.Context, ns, content, sourceKind, sourceRef string, importance float64) (string, error)

Retain writes a memory.

source_ref is identity: retaining the same source twice updates in place rather than growing a duplicate. Without it an agent that re-reads a file on every run accumulates one copy per run and its recall degrades into noise.

func (*Store) RetainProject 

func (s *Store) RetainProject(ctx context.Context, fleet, content, sourceKind, sourceRef string, importance float64) (string, error)

RetainProject writes to the project brain. This is the INGESTION path, and it is the only way anything writes there.

It takes no agent slug, and that is the point. `Retain` is reached by agents through Writable(), which resolves a namespace by agent_slug — a project brain has none, so no agent can route a write here however it is called. Ingestion (sources, webhooks, documents) has no agent to be, and calls this instead.

sourceRef is required. An agent memory without one is merely a duplicate risk; a project memory without one is unattributable — nobody reading the shared brain can tell where a claim about the project came from, and nobody can re-ingest a corrected version over it.

func (*Store) Routes 

func (s *Store) Routes(r chi.Router)

Routes mounts the project brain surface.

func (*Store) Semantic added in v0.2.0 

func (s *Store) Semantic() bool

Semantic reports whether recall over this store means anything, which is the embedder question plus the liveness of the endpoint behind it.

func (*Store) SetReranker added in v0.2.0 

func (s *Store) SetReranker(rr Reranker)

SetReranker installs the second-pass ranker. Optional and separate from New so a store without one behaves exactly as it did — the reranker improves the order of results the fused query already found, it is not load-bearing.

func (*Store) Writable 

func (s *Store) Writable(ctx context.Context, agentSlug string) (string, error)

Writable is the agent's own namespace only. A read grant never implies write: the shared project brain is readable by everyone and writable by no agent.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.