Documentation
¶
Overview ¶
Package sources runs saved read-only queries against real databases on a schedule and retains the rendered result as project memory.
This file is the part that has to be right. A source holds a credential for somebody's production database, so the query it runs is treated as hostile input even though an operator typed it: the operator who typed it is not necessarily the one whose database it reaches.
Read-only is enforced at four independent layers, because any one of them can be wrong:
- ValidateReadOnly, here — a single statement that begins SELECT or WITH, with nothing data-modifying hidden in it.
- A READ ONLY transaction — Postgres itself refuses every write, whatever this file failed to notice. This is the layer that actually guarantees it.
- statement_timeout and a row cap — see sql.go.
- A read-only database role in the DSN, which is the operator's job and the only layer that also survives a bug in pgx.
Layer 1 exists to give a clear error at save time instead of a confusing one at 03:00, and to catch the things layer 2 permits — pg_read_file is a *read*, so a READ ONLY transaction is perfectly happy to hand back /etc/passwd.
Package sources pulls knowledge from outside this repository into the brain.
A source is a thing an agent should know about but cannot read for itself: a repository's README, a runbook, a changelog. The package is a registry of source KINDS plus one function that runs a source and retains what it found.
Three seams are deliberate, and all three exist so that a credential, a namespace and a schema decision stay outside this package:
- Secrets is reveal-BY-NAME. A source config holds the NAME of a vault secret, never its value (Rule 34), and the caller binds which principal is doing the revealing. Nothing here logs, returns or retains a revealed value — it reaches exactly one place, an Authorization header.
- Retainer is the brain narrowed to its one write. The NAMESPACE is the caller's decision, because "which brain may an unattended ingester write to" is a grant question, not a connector question (Rule 40).
- CursorStore is where a source's incremental position lives. The obvious home is a builder_sources row, but that is a new table and therefore a `schema_change` — a human's call. Keeping it an interface means the connector is finished and tested today and gains persistence with one adapter later, rather than waiting on a migration.
Index ¶
- Constants
- Variables
- func Kinds() []string
- func LooksLikeCredential(s string) bool
- func Match(pattern, name string) bool
- func MatchAny(patterns []string, name string) bool
- func MemoryRef(kind, name, ref string) string
- func Redact(s string) string
- func Register(kind string, f Factory)
- func Scrub(s string) string
- func ScrubErr(err error) string
- func ValidateReadOnly(q string) (string, error)
- type Batch
- type CursorStore
- type Doc
- type Factory
- type GitHubConfig
- type MemCursors
- type RSSConfig
- type Report
- type Result
- type Retainer
- type Revealer
- type SQLConfig
- type Secrets
- type Skip
- type SlackConfig
- type Source
- type Store
Constants ¶
const ( DefaultMaxRows = 200 MaxMaxRows = 5000 DefaultTimeoutMS = 10_000 MaxTimeoutMS = 60_000 // The rendered text is one memory row that an agent pastes into its // context on every recall. Past a few KB it stops being a fact and starts // being a document that crowds out everything else the agent knows. MaxRenderedBytes = 8192 )
Caps. A source exists to answer a question in one paragraph, so these are sized for that rather than for reporting. A query that wants more than this wants a dashboard, not a memory.
const KindCrawl = "crawl"
KindCrawl is the registered kind for a web crawl source.
const KindGitHub = "github"
KindGitHub is the registered kind for a GitHub repository source.
const KindRSS = "rss"
KindRSS is the registered kind for an RSS/Atom feed source.
const KindSlack = "slack"
KindSlack is the registered kind for a single Slack channel source.
const KindWhatsApp = "whatsapp"
KindWhatsApp is the registered kind for a single WhatsApp chat source.
Variables ¶
var ErrNoNamespace = errors.New("sources: refresh needs a target namespace")
ErrNoNamespace is returned when a refresh is asked to write nowhere.
Functions ¶
func LooksLikeCredential ¶
LooksLikeCredential reports whether a value that is supposed to be a NAME is in fact a secret somebody pasted in.
This is the check behind "uses the vault's token, never a token pasted into the config": a config field holding a credential is rejected at parse time rather than quietly working, because the version that quietly works is the one that ends up in a git repository.
func Match ¶
Match reports whether a slash-separated path matches a gitignore-style pattern.
path.Match alone is not enough: its `*` does not cross a separator and it has no `**` at all, so `docs/**` — the single most obvious include pattern anyone will type — matches nothing. Rather than leave that as a footgun, `**` is handled here and every other segment is delegated to path.Match.
Rules:
- `**` matches zero or more whole segments, so `docs/**` matches `docs/a.md` and `docs/adr/1.md`, and also `docs` itself.
- `*` and `?` match within one segment, via path.Match.
- A pattern with no `/` matches against the BASENAME, so `README*` finds `README.md` at the root and `sub/README.md` alike. This is what people mean when they type it.
- Matching is case-insensitive: GitHub repositories contain README.md, readme.md and Readme.md, and an operator should not have to guess.
func MatchAny ¶
MatchAny reports whether any pattern matches. An empty pattern list matches nothing — an include list that silently meant "everything" would pull an entire repository into the brain on a typo.
func MemoryRef ¶
MemoryRef is the source_ref a doc is retained under.
Namespaced by kind and source name so that two repositories with a README each do not overwrite one another, and so an operator can see at a glance where a memory came from.
func Register ¶
Register adds a source kind. Called from a package init, so that importing the connector is all it takes to make the kind available.
func Scrub ¶
Scrub removes credentials from a string that is about to be logged or stored.
Always call this on an error before it reaches a log line or an API response.
func ValidateReadOnly ¶
ValidateReadOnly checks that q is a single, read-only SELECT and returns it trimmed of a trailing semicolon.
It reports the FIRST problem it finds rather than a list: an operator fixing a query wants one thing to change, and a validator that says "and also" tends to be argued with rather than obeyed.
Types ¶
type Batch ¶
type Batch struct {
Docs []Doc
Skipped []Skip
// Removed lists refs the source knows are gone upstream. Invalidating the
// matching memories needs a writer identity this package does not have, so
// it reports them and lets the caller decide.
Removed []string
// Cursor is opaque to everything except the source that produced it. It is
// persisted only after every Doc in the batch has been retained, so a
// refresh that dies halfway is retried rather than silently skipped.
Cursor string
// Unchanged is true when the source found nothing new. It is the whole
// point of the cursor: a re-run against an unchanged repository should cost
// one HTTP request and zero writes.
Unchanged bool
}
Batch is one fetch's worth of work.
type CursorStore ¶
type CursorStore interface {
Cursor(ctx context.Context, kind, name string) (string, error)
SetCursor(ctx context.Context, kind, name, cursor string) error
}
CursorStore persists where each source got to.
type Doc ¶
type Doc struct {
// Ref is stable across refreshes and unique within the source. It becomes
// part of the memory's source_ref, and the brain's unique index on
// (namespace, source_ref) turns the second refresh of the same Ref into an
// UPDATE. A source that minted a fresh Ref every run would bury its own
// current answer under a pile of stale copies within a fortnight.
Ref string
Title string
Text string
Importance float64
}
Doc is one unit of knowledge on its way to becoming a memory.
type Factory ¶
type Factory func(cfg json.RawMessage, sec Secrets) (Source, error)
Factory builds a source from its stored config.
type GitHubConfig ¶
type GitHubConfig struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Branch string `json:"branch,omitempty"` // empty means the repo's default branch
Include []string `json:"include,omitempty"`
Exclude []string `json:"exclude,omitempty"`
// ReadIssues pulls issue and pull-request titles as well as files.
ReadIssues bool `json:"readIssues,omitempty"`
// TokenRef is the NAME of a vault secret, never a token. Empty is legal and
// means unauthenticated, which works for a public repository at a much
// lower rate limit.
TokenRef string `json:"tokenRef,omitempty"`
MaxFileBytes int `json:"maxFileBytes,omitempty"`
MaxCommits int `json:"maxCommits,omitempty"`
MaxIssues int `json:"maxIssues,omitempty"`
// APIBase supports GitHub Enterprise, and lets a test point the connector
// at an httptest server.
APIBase string `json:"apiBase,omitempty"`
}
GitHubConfig is one configured repository.
There is deliberately no `token` field. The credential lives in the vault and this config carries its NAME; a config that carries a value is rejected at parse time rather than quietly working, because the version that quietly works is the one that gets committed to a repository (Rule 34).
type MemCursors ¶
type MemCursors struct {
// contains filtered or unexported fields
}
MemCursors is an in-process CursorStore. It is what the tests use, and what a single-process run uses until source rows have somewhere durable to live.
func NewMemCursors ¶
func NewMemCursors() *MemCursors
type RSSConfig ¶
type RSSConfig struct {
FeedURL string `json:"feedURL"`
MaxEntries int `json:"maxEntries,omitempty"`
// FetchFull follows each new entry's link and retains the article body in
// place of the feed's own summary. Off by default because it multiplies
// outbound requests by the number of new entries every run.
FetchFull bool `json:"fetchFull,omitempty"`
}
RSSConfig is one configured feed.
There is no credential field. A feed source reads a public URL; a feed that needs authentication is out of scope for this connector rather than a reason to smuggle a token into feedURL (see normalise's userinfo check).
type Report ¶
Report is what one refresh did.
func Refresh ¶
func Refresh(ctx context.Context, src Source, cur CursorStore, brain Retainer, ns string) (Report, error)
Refresh runs one source and retains what it found.
The cursor is advanced only after every doc has landed. The alternative — saving first, retaining after — turns a transient network error into a permanent hole in the brain, because the next run starts after the documents that were never written.
type Retainer ¶
type Retainer interface {
Retain(ctx context.Context, ns, content, sourceKind, sourceRef string, importance float64) (string, error)
}
Retainer is the brain, narrowed to its one write. *brain.Store satisfies it.
type Revealer ¶
type Revealer interface {
RevealFor(ctx context.Context, name, agentSlug, runID string) (string, error)
}
Revealer is the vault, narrowed to the one call this package makes.
The interface exists so the runner cannot reach anything else in the vault by accident, and so tests can exercise it without a real secret. *vault.Store satisfies it.
type SQLConfig ¶
type SQLConfig struct {
// DSNSecret is the NAME of a vault secret holding the connection string.
// Never the connection string. A source row is ordinary table data — it is
// in every backup and readable by anything with SELECT on the table.
DSNSecret string `json:"dsnSecret"`
// RunAs is the agent slug the reveal is audited under, and whose grant
// decides whether the reveal is allowed at all.
//
// This is not decoration. builder_secret_grants.agent_slug is a foreign key
// to builder_agents(slug), so a source can only run as an agent that
// actually exists, and only if a human has granted that agent can_reveal on
// this secret. Nothing in this package creates such a grant: widening who
// can read a production credential is a human decision (Rule 34), and a
// scheduler that could grant itself access would make the audit trail
// meaningless.
RunAs string `json:"runAs"`
SQL string `json:"sql"`
// Template renders rows to the text an agent quotes. Empty means the
// built-in table rendering.
Template string `json:"template"`
MaxRows int `json:"maxRows"`
TimeoutMS int `json:"timeoutMs"`
}
SQLConfig is builder_sources.config for kind='sql'.
It is the schema of record for that column: the migration stores jsonb precisely so this struct, and the validation below it, is the only place the shape is defined.
type Secrets ¶
Secrets is the vault, narrowed to the one call a source may make.
The implementation binds the principal and the run, so that the audit row says who read the credential. A source never sees, stores or logs the value it gets back.
type Skip ¶
Skip is something the source deliberately did not ingest. Skips are reported rather than dropped: "the README never appeared" and "the README was 4MB and was skipped" are very different failures, and only one of them is a bug.
type SlackConfig ¶
type SlackConfig struct {
ChannelID string `json:"channelID"`
// TokenSecret is the NAME of a vault secret holding a bot token — never
// the token itself (Rule 34).
TokenSecret string `json:"tokenSecret"`
MaxMessages int `json:"maxMessages,omitempty"`
// APIBase lets a test point the connector at an httptest server. Slack has
// no self-hosted deployment, so in production this is always the default.
APIBase string `json:"apiBase,omitempty"`
}
SlackConfig is one configured channel.
There is deliberately no `token` field, for the same reason GitHubConfig has none: the credential lives in the vault and this config carries its NAME.
type Source ¶
type Source interface {
Kind() string
Name() string
Fetch(ctx context.Context, cursor string) (Batch, error)
}
Source is one configured thing to read from.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store runs sources.
func (*Store) RefreshDue ¶
RefreshDue runs every enabled source whose next_run_at has passed.
Each source runs AT MOST ONCE per pass. The loop used to trust next_run_at to move forward, which is true only if markOK's UPDATE succeeds — and against a database whose builder_sources predated total_runs it did not, silently. The source stayed due, the loop claimed it again immediately, and one connector refetched the same repository in a tight loop for as long as the process ran.
A scheduler that can spin is worse than one that skips a tick: the tick comes round again in a minute, whereas a loop hammers somebody else's API and fills the brain until an operator notices. Seeing a source twice means something is wrong with the row, so stop and let the next pass — and the logged error — deal with it.
func (*Store) RefreshNow ¶
RefreshNow runs one source by name, ignoring its schedule but not its lease. `enabled` is also ignored: this is the "does my query work?" path, and being unable to test a source before turning it on is how a broken one gets turned on anyway.
Source Files