memory

package
v0.13.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 8, 2026 License: MIT Imports: 30 Imported by: 0

Documentation

Overview

Package memory is abcd's transport-agnostic curated-knowledge substrate at .abcd/memory/ (itd-36 / adr-13). Every verb — ingest, ask, lint, and the bare status render — is a function taking a structured request and returning a structured result plus error; nothing here writes stdout or knows about a CLI, MCP, or prompt surface. Surfaces under internal/surface/* marshal these results for their transport.

WritePages is the ONE mutating entry to the store; it honours the ADR-13 single-writer, atomic-rename crash model (advisory flock + six-step durable write + idempotent sibling reconciliation, no journal). Read-only paths (Ask, Bare, and the read half of Ingest) never heal drift — they report it.

Index

Constants

View Source 
const AskReportHeading = "abcd memory ask"

AskReportHeading heads every ask render. It is the BINARY invocation, not the `/abcd:memory ask` slash command, because this is core: the same bytes are rendered whether the caller came through the plugin surface, the bare CLI, or a later transport, and a heading naming one front door is a falsehood in the other two (iss-44). A reader who ran `abcd memory ask` in a terminal must not be told they invoked a plugin command they may not even have installed.

View Source 
const AskTopN = 5

AskTopN is the pinned default retrieval depth.

View Source 
const LintReportHeading = "abcd memory lint"

LintReportHeading heads the run-log report, on the same rule as AskReportHeading: core names the binary invocation, never one front door.

View Source 
const RelDir = ".abcd/memory"

RelDir is the store's repo-relative directory, slash-separated. It is what a front door names when it has a checkout root and no store path yet — the stray-store walk a resolving front door makes reports a substrate sitting below the checkout root, and it has only the relative shape to look for.

View Source 
const TokenCountVersion = 1

TokenCountVersion is the pinned deterministic tokenizer version: count of regex \w+ word tokens over the normalised text. The registry stores it so consumers read a stable basis.

Variables

This section is empty.

Functions

func Dir 

func Dir(repoRoot string) string

Dir returns the canonical store path <repoRoot>/.abcd/memory.

func IsMemoryPageName 

func IsMemoryPageName(filename string) bool

IsMemoryPageName reports whether filename is an individual memory page — a .md directly in the store that is not a sibling and not a dotfile.

func LoadRegistry 

func LoadRegistry(path string) (map[string]any, error)

LoadRegistry reads the registry JSON. A missing file yields an empty registry; a present-but-unparseable file raises RegistryFormatError — durable source metadata must fail loudly, never be silently replaced.

func MergeIngest 

func MergeIngest(registry map[string]any, ev IngestEvent) (map[string]any, error)

MergeIngest merges one ingest event into registry and returns a NEW map (the input is never mutated). Pure — no I/O, no lock.

func ParsePageFilename 

func ParsePageFilename(filename string) (typ, domain, slug string, ok bool)

ParsePageFilename parses <type>_<domain>_<slug>.md, reporting ok=false on a non-conforming name.

func RenderContradictions 

func RenderContradictions(pages []PageInfo) string

RenderContradictions renders contradictions.md deterministically from page facts.

func RenderIndex 

func RenderIndex(pages []PageInfo) string

RenderIndex renders index.md deterministically (sorted by filename).

func SerializeRegistry 

func SerializeRegistry(registry map[string]any) string

SerializeRegistry deterministically serialises the registry (sorted keys, 2-space indent, trailing newline).

func SourceClasses 

func SourceClasses(source map[string]any) []string

SourceClasses returns the page's declared source classes: the UNION of the scalar `class` and the plural `classes` list (deduplicated). This is the same derivation the lint makes (memoryLinter.derivedClasses) — reading only the scalar when both shapes are present shadowed the plural list in the opposite direction to lint, so a both-shapes page rendered under only its scalar class in the generated index and write log while lint counted the union (iss-2608270945468534). The two shapes are mutually exclusive only on the write path (validateSourceBlock); readers see raw on-disk bytes and must read both.

func SourceHashes 

func SourceHashes(source map[string]any) []string

SourceHashes returns the page's full contributing source-hash set: source.source_hash (single) and every source.sources[].source_hash (multi).

func SourcesIndexPath 

func SourcesIndexPath(repoRoot string) string

SourcesIndexPath returns .abcd/memory/.sources_index.json.

func WithStoreLock 

func WithStoreLock(repoRoot string, fn func() error) error

WithStoreLock holds the exclusive non-blocking advisory lock on .abcd/memory/.lock for fn's extent. The closure form keeps the locked/unlocked split structural — flock does not nest. The lock is fsutil.WithFileLock with no wait: a lock another process holds fails closed at once, as *StoreLockHeldError, and a lock path that is a symlink or not a regular file is *UnsafeStorePathError, judged on the path first and again on the opened descriptor. fn's own error passes through unchanged.

Types

type AskCitation 

type AskCitation struct {
	SourceClass string         `json:"source_class"`
	Citation    map[string]any `json:"citation"`
	SourceHash  string         `json:"source_hash"`
	Licence     string         `json:"licence"`
	IngestedAt  string         `json:"ingested_at"`
}

AskCitation carries the per-source provenance facts of one matched page. Fields are "" when absent (a legacy / backfilled page), never invented.

type AskError 

type AskError struct{ Msg string }

AskError is a pre-write ask failure (unusable file-back payload, no matches to file back against, cited pages without complete provenance). Raised BEFORE any write; the surface reports it and exits 1.

func (*AskError) Error 

func (e *AskError) Error() string

type AskRequest 

type AskRequest struct {
	RepoRoot       string
	Question       string
	TopN           int
	Synthesizer    Synthesizer
	FileBackPage   map[string]any
	DecideFileBack FileBackDecision
	Now            time.Time
}

AskRequest is the input to Ask.

type AskResult 

type AskResult struct {
	Question string          `json:"question"`
	Matches  []MatchedPage   `json:"matches"`
	Answer   string          `json:"answer"`
	FileBack *FileBackResult `json:"file_back"`
}

AskResult is the structured result of one Ask call.

func Ask 

func Ask(req AskRequest) (AskResult, error)

Ask runs deterministic retrieval -> synthesis -> optional file-back.

type BareStatus 

type BareStatus struct {
	StorePresent   bool         `json:"store_present"`
	Pages          int          `json:"pages"`
	ByClass        []ClassCount `json:"by_class"`
	LastIngest     string       `json:"last_ingest"`
	Contradictions []string     `json:"contradictions"`
	Drift          []string     `json:"drift"`
	Headroom       []string     `json:"headroom"`
}

BareStatus is the structured result of Bare.

func Bare 

func Bare(repoRoot string) (BareStatus, error)

Bare renders the read-only store status.

type ClassCount 

type ClassCount struct {
	Class string `json:"class"`
	Count int    `json:"count"`
}

ClassCount is one page-count-by-class row.

type DistilledPage 

type DistilledPage struct {
	Type        string
	Domain      string
	Slug        string
	Body        string
	TopicHash   string
	Source      map[string]any
	Contradicts []string
	Recall      []string
}

DistilledPage is one validated distiller output page. TopicHash is computed by validateDistilledPage (a supplied one is rejected) and persisted into the written page's frontmatter.

func (DistilledPage) Filename 

func (p DistilledPage) Filename() string

Filename returns <type>_<domain>_<slug>.md.

type Distiller 

type Distiller func(normalisedText string, sourceBlock map[string]any) ([]map[string]any, error)

Distiller is the host-delegated seam: (normalisedText, sourceBlock) -> raw page maps. A map omitting "source" gets sourceBlock injected. The core validates every page.

type FetchedSource 

type FetchedSource struct {
	FinalURL string
	Headers  map[string]string
	Body     []byte
}

FetchedSource is the raw result of fetching a URL (the injectable Fetcher contract). Content-type / size / decode checks are applied uniformly by the ingest path after the fetcher returns.

type Fetcher 

type Fetcher func(url string) (FetchedSource, error)

Fetcher fetches a URL; a nil Fetcher uses the bounded default fetch.

type FileBackDecision 

type FileBackDecision func(page DistilledPage) bool

FileBackDecision is consulted after validation and before any write; false declines (no partial write).

type FileBackResult 

type FileBackResult struct {
	Status         string       `json:"status"`
	Pages          []string     `json:"pages"`
	Linked         [][2]string  `json:"linked"`
	Contradictions [][2]string  `json:"contradictions"`
	WriteReport    *WriteReport `json:"write_report"`
}

FileBackResult records what the file-back branch did.

type Finding 

type Finding struct {
	Code       string `json:"code"`
	Severity   string `json:"severity"`
	File       string `json:"file"`
	Line       int    `json:"line"`
	Message    string `json:"message"`
	Suggestion string `json:"suggestion"`
}

Finding is a single memory-lint finding.

type IngestError 

type IngestError struct{ Msg string }

IngestError is a pre-dispatch ingest failure — raised BEFORE any memory-store write (bad source path, fetch failure, binary source, zero distilled pages, repair collision). The surface reports it fail-closed and exits 1.

func (*IngestError) Error 

func (e *IngestError) Error() string

type IngestEvent 

type IngestEvent struct {
	ContentHash       string
	Consumer          string
	SourceClass       string
	Citation          map[string]any
	Origin            string
	Licence           string
	IngestedAt        string
	Pages             []string
	SourceTokenCount  int
	TokenCountVersion int
}

IngestEvent is one ingest event to merge into the registry. A zero SourceTokenCount (or TokenCountVersion) is treated as "not provided" — the durable value is filled only when currently null, never overwritten.

type IngestRequest 

type IngestRequest struct {
	RepoRoot     string
	Source       string
	Distiller    Distiller
	KeepOriginal bool
	Fetcher      Fetcher
	PDFExtractor PDFExtractor
	Now          time.Time
}

IngestRequest is the input to Ingest.

type IngestResult 

type IngestResult struct {
	Status           string         `json:"status"`
	ContentHash      string         `json:"content_hash"`
	Licence          string         `json:"licence"`
	SourceTokenCount int            `json:"source_token_count"`
	Pages            []string       `json:"pages"`
	Citation         map[string]any `json:"citation"`
	KeptOriginal     string         `json:"kept_original"`
	// KeepOriginalError records a --keep-original copy failure that occurred
	// AFTER the pages and registry were durably written. The ingest itself
	// succeeded; only the best-effort original copy did not. Empty when
	// --keep-original was not requested or the copy succeeded.
	KeepOriginalError string       `json:"keep_original_error,omitempty"`
	Linked            [][2]string  `json:"linked"`
	Contradictions    [][2]string  `json:"contradictions"`
	WriteReport       *WriteReport `json:"write_report"`
	// ScanGap names the coverage the repository asked for and did not get: a
	// scanner augmenter it configured (gitleaks, in .abcd/config/gitleaks.json)
	// whose tool is not installed. The ingest still writes, redacted by the
	// native scanner, and this says so (the 2026-09-25 ruling on
	// iss-2608291814575788). Empty when there is no gap.
	ScanGap string `json:"scan_gap,omitempty"`
}

IngestResult is the structured result of one Ingest call.

func Ingest 

func Ingest(req IngestRequest) (IngestResult, error)

Ingest runs the full ingest flow for one source (a local path or an https URL; a plaintext http source is refused — see isURL).

type LicenceDetection 

type LicenceDetection struct {
	Licence     string // SPDX id, verbatim compound expression, or "unknown"
	Restrictive bool
	Source      string // where it was detected: spdx_header | manifest | licence_file | http_header | none
}

LicenceDetection is the result of detectLicence.

type LintRequest 

type LintRequest struct {
	RepoRoot string
	Now      time.Time
}

LintRequest is the input to Lint.

type LintResult 

type LintResult struct {
	Findings      []Finding      `json:"findings"`
	Summary       LintSummary    `json:"summary"`
	CoverageIndex map[string]any `json:"coverage_index"`
	ReportDir     string         `json:"report_dir"`
	GeneratedAt   string         `json:"generated_at"`
	StorePath     string         `json:"store_path"`
	ExitCode      int            `json:"exit_code"`
}

LintResult is the structured result of Lint.

func Lint 

func Lint(req LintRequest) (LintResult, error)

Lint runs the full-store curator health-check and writes one run-log report. Mutates no memory-store state (only the regenerable coverage index + report).

type LintSummary 

type LintSummary struct {
	Blockers int `json:"blockers"`
	Warnings int `json:"warnings"`
	Infos    int `json:"infos"`
}

LintSummary tallies findings by severity.

type MatchedPage 

type MatchedPage struct {
	Filename  string        `json:"filename"`
	Score     int           `json:"score"`
	Classes   []string      `json:"classes"`
	Domain    string        `json:"domain"`
	Summary   string        `json:"summary"`
	Body      string        `json:"body"`
	Citations []AskCitation `json:"citations"`
}

MatchedPage is one retrieval hit.

type MemorySchemaError 

type MemorySchemaError struct{ Msg string }

MemorySchemaError signals a distilled page / source block that does not conform to the memory schema. No artifact is produced.

func (*MemorySchemaError) Error 

func (e *MemorySchemaError) Error() string

type PDFExtractor 

type PDFExtractor func(data []byte) (string, error)

PDFExtractor extracts text from PDF bytes; a nil extractor rejects with a clear error (never silently pulls in a parser dependency).

type PageInfo 

type PageInfo struct {
	Filename    string
	Classes     []string
	Domain      string
	Summary     string
	Contradicts []string
}

PageInfo carries the per-page facts the derived siblings render from.

type PageWrite 

type PageWrite struct {
	Filename    string
	Frontmatter map[string]any
	Body        string
}

PageWrite is one page to materialise: <type>_<domain>_<slug>.md, full frontmatter (must carry a schema-valid source: block), and the markdown body.

type PlannedWrite 

type PlannedWrite struct {
	Filename    string
	Frontmatter map[string]any
	Body        string
}

PlannedWrite is one page the write plan materialises (field-compatible with PageWrite).

type RegistryFormatError 

type RegistryFormatError struct{ Msg string }

RegistryFormatError signals .sources_index.json exists but is not parseable as a JSON object — durable metadata must fail loudly, never be silently replaced with an empty index.

func (*RegistryFormatError) Error 

func (e *RegistryFormatError) Error() string

type RegistryMerge 

type RegistryMerge func(current map[string]any) (map[string]any, error)

RegistryMerge recomputes the COMPLETE new registry from the registry as freshly read under the store lock. WritePages calls it with the on-disk registry loaded INSIDE the lock, so the merged result is never derived from a stale pre-lock snapshot. This closes the load-merge-write lost-update: two concurrent ingests that each loaded the registry before either wrote would, if each wrote its own wholesale pre-computed registry, have the last writer silently clobber the other's entry (and orphan its pages). Re-running the merge against the locked read makes each write additive. A nil RegistryMerge leaves the registry untouched (a heal-only pass).

type StoreLockHeldError 

type StoreLockHeldError struct{ Path string }

StoreLockHeldError signals a live process holds .abcd/memory/.lock — the writer fails closed (LOCK_NB; the lock file is never deleted, flock releases on process exit).

func (*StoreLockHeldError) Error 

func (e *StoreLockHeldError) Error() string

type Synthesizer 

type Synthesizer func(question string, matches []MatchedPage) string

Synthesizer turns matches into answer prose; nil uses renderCitedMatches.

type UnsafeStorePathError 

type UnsafeStorePathError struct{ Msg string }

UnsafeStorePathError signals a memory-store path (.abcd, .abcd/memory, or the lock leaf) is a symlink or non-regular filesystem object.

func (*UnsafeStorePathError) Error 

func (e *UnsafeStorePathError) Error() string

type WritePlan 

type WritePlan struct {
	Writes         []PlannedWrite
	RegistryPages  map[string][]string
	Linked         [][2]string
	Contradictions [][2]string
}

WritePlan is the output of resolveDistilledPages.

type WriteReport 

type WriteReport struct {
	CreatedSiblings []string `json:"created_siblings"`
	Backfilled      []string `json:"backfilled"`
	Pruned          []string `json:"pruned"`
	Reconciled      []string `json:"reconciled"`
	WrotePages      []string `json:"wrote_pages"`
}

WriteReport records what one WritePages call did (all lists sorted).

func WritePages 

func WritePages(repoRoot string, writes []PageWrite, merge RegistryMerge, now time.Time) (WriteReport, error)

WritePages is the single mutating entry point for .abcd/memory/. It acquires the advisory store lock, runs the full locked sequence (skeleton ensure -> legacy backfill -> orphan prune -> pre-reconcile -> page writes -> registry load+merge+write when given -> post-reconcile -> log append), and returns a report. An empty writes slice is a valid heal-only pass. merge may be nil (no registry mutation); when non-nil it is invoked with the registry read fresh under the lock and must return the COMPLETE new mapping. A zero now is sampled as time.Now().UTC() AFTER the lock is held.

type WriterContractError 

type WriterContractError struct{ Msg string }

WriterContractError is a writer-side contract violation (invalid write request, unreadable user-visible file the writer refuses to overwrite). No artifact is produced.

func (*WriterContractError) Error 

func (e *WriterContractError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.