tools

Documentation

Overview

casefold.go — case-insensitive structured-record parsing at the boundary (portable-agent-architecture.md §3.7).

Humans write fixtures on different days as EventType, event_type, eventType, or event-type. Go's encoding/json matches a JSON key to a struct tag case-insensitively, but it does NOT translate between casing CONVENTIONS: "EventType" matches struct tag `json:"EventType"` but does NOT match struct tag `json:"event_type"`. The failure is the worst kind — the tool runs without error, returns shape-correct data, but every field is the zero value because no key matched. No error log to grep. This is bug #5 in the brief.

The fix: normalize every JSON object key to a canonical snake_case form at the parse boundary BEFORE unmarshalling into the typed struct. Then snake_case, camelCase, PascalCase, and kebab-case fixtures all parse identically.

normalizeRecordKeys rewrites the top level of a JSON object so each key is snake_case. It is applied to each record's raw JSON before json.Unmarshal in the event/finding decode paths.

check_baseline.go — the check-baseline pure read tool, reusing pkg/baseline.

CheckBaseline answers "is this entity known to the baseline, and what do we know about it?" It is a PURE function over an already-loaded *baseline.Baseline — no disk access, no inference, no side effects. Callers load the baseline (baseline.Load, or reconstruction from core/store) and pass the typed value.

envelope.go — the one-shape-always tool envelope and self-resolving config root, the two cross-cutting pieces of the tool-contract discipline (portable-agent-architecture.md §3.3, §3.5).

Envelope (§3.3, §3.4)

Every tool that the agent calls returns the SAME JSON shape on EVERY call. Every key is present every time — empty arrays / empty strings, never an omitted field, never a bare null, never a conditional branch on input. A tool with "nothing to return" returns the canonical envelope with empty slices and a notes string, NOT an error and NOT a "no results" string. Errors are reserved for genuine schema violations (nil store, malformed record JSON).

Self-resolving config root (§3.5)

Tools locate their static config (the operator-decisions rule corpus, fixtures) by walking UP from the binary's own location to a project marker — never from the current working directory (the agent runner sets CWD to a sandbox / worktree / /tmp) and never from an env var that "should be set" by the parent (the sandbox inherits a clean env). An env-var override is accepted as a LAST resort, but the walk is the primary path. This kills the entire class of "tool silently returns empty because it looked in the wrong place" failures (bug #1 and bug #6 in the brief).

github_actor.go — the github_actor LIVE lookup tool, the one net-new tool in the chat<->investigate protocol (mallcop-pro docs/chat-investigate-protocol.md).

Unlike the rest of this package (pure functions over an already-loaded store/baseline), GithubActor makes a real, read-only HTTP call to api.github.com: GET /users/{login} for the public profile, then GET /users/{login}/events/public for recent public activity. Both are public GitHub REST endpoints — unauthenticated calls work (60 req/hr); a GITHUB_TOKEN in the environment is sent as a bearer token when present (10x'ing the rate limit), the same env var connect/github.NewFromEnv reads for its PAT auth path. This tool does not construct a connect/github Connector (that type is org-events-scoped and carries state this per-actor lookup does not need) — it reuses only the auth/transport DISCIPLINE: a bearer header built from GITHUB_TOKEN, and an SSRF guard on every URL fetched, mirroring connect/github.validateNextLink (https + exact allowlisted host, refuse anything else).

The ghost tombstone

github.com/ghost is GitHub's real, reserved placeholder account: commits and other activity originally authored by a DELETED GitHub account are permanently reattributed to `ghost` rather than left dangling. `ghost` is not itself a deleted account — it is a live, real, public profile (created 2008) that exists specifically to be "this used to be someone." This is the exact question the current chat invents three wrong theories about ("who is ghost?"); GithubActor answers it for real by asking GitHub.

Envelope discipline (envelope.go)

GithubActorEnvelope carries the same keys on every call — empty string/ slice, never an omitted field, never bare null. GithubActor returns an error only for a genuine schema violation (empty login, unparseable API base URL, a transport failure, or an unexpected non-200/404 status). "Login not found" (404) is a legitimate, common result — it comes back as Found=false with an explanatory Notes, never an error.

lookup_rules.go — the lookup-rules pure read tool + operator-decisions.yaml loader, ported from cmd/mallcop-investigate-tools/tools_lookup_rules.go.

LookupRules is a PURE function: given a repo root, a finding family, and a flat metadata predicate, it returns the operator rules that match. It reads only the flat-file rule corpus at agents/rules/operator-decisions.yaml. It performs no inference, opens no channel, and produces no side effects beyond reading the corpus from disk.

YAML schema

rules:
  - id: "R-001"
    applies_to:
      family: "unusual-timing"            # detector family / finding.detector
      metadata_match:
        <key>: <value>                    # conjunctive flat-map predicate
    operator_directive: |
      <evidence-grounded resolution rationale>

File location

The rules file lives at agents/rules/operator-decisions.yaml (repo-relative). The caller resolves the repo root and passes it in — this package does not guess at process environment.

Security note

rule_id forgery is prevented downstream by requiring a cited rule_id to actually load from the YAML file. A consumer that invents "R-999" gets zero rules from LookupRules because no such rule exists in the corpus.

search_events.go — the search-events pure read tool, reusing pkg/event and reading the events stream from core/store.

SearchEvents replays the events stream from a *store.Store and returns the typed event.Event records that pass the actor/source/type/time filters. It is a PURE read: it opens no channel, runs no inference, and never writes. Its only effect is to read committed records from the git-backed store.

search_findings.go — the search-findings pure read tool, reusing pkg/finding and reading the findings stream from core/store.

SearchFindings replays the findings stream from a *store.Store and returns the typed finding.Finding records that pass the actor/source/since filters. It is a PURE read: it opens no channel, runs no inference, and never writes. Its only effect is to read committed records from the git-backed store.

Index

• func SearchEvents(s *store.Store, in SearchEventsInput) (events []event.Event, timeFilterFellBack bool, err error)
• func SearchFindings(s *store.Store, in SearchFindingsInput) ([]finding.Finding, error)
• type CheckBaselineInput
• type CheckBaselineResult
  • func CheckBaseline(b *baseline.Baseline, in CheckBaselineInput) (CheckBaselineResult, error)
• type EventView
  • func EventViewsFor(events []event.Event) []EventView
• type FilterApplied
• type GithubActorEnvelope
  • func GithubActor(ctx context.Context, in GithubActorInput) (GithubActorEnvelope, error)
• type GithubActorEvent
• type GithubActorInput
• type LookupRulesInput
• type LookupRulesOutput
  • func LookupRules(repoRoot string, input LookupRulesInput) (LookupRulesOutput, error)
• type OperatorRule
  • func FindRuleByID(rules []OperatorRule, ruleID string) (OperatorRule, bool)
  • func LoadOperatorRules(repoRoot string) ([]OperatorRule, error)
  • func LoadOperatorRulesResolved(repoRoot string, rootErr error) ([]OperatorRule, error)
• type RuleAppliesTo
• type SearchEventsEnvelope
  • func SearchEventsWrapped(s *store.Store, in SearchEventsInput, findingFamily string, ...) (SearchEventsEnvelope, error)
• type SearchEventsInput
• type SearchFindingsInput

Functions

func SearchEvents

func SearchEvents(s *store.Store, in SearchEventsInput) (events []event.Event, timeFilterFellBack bool, err error)

SearchEvents reads the events stream from the store and returns the events matching the filter, oldest first.

Time-filter fallback: if a Since/Until window excludes EVERY event but the non-time filters matched some, SearchEvents returns the non-time-filtered set instead. A caller-supplied window is frequently anchored to a different "now" than the stored fixtures (an LLM hallucinating a date range a year off from the data); treating an all-excluding window as a no-op keeps the read useful rather than silently empty. The boolean second return reports whether this fallback fired, so the caller can annotate the result if it cares.

SearchEvents returns an error only when the store cannot be read or a record is not valid event JSON.

func SearchFindings

func SearchFindings(s *store.Store, in SearchFindingsInput) ([]finding.Finding, error)

SearchFindings reads the findings stream from the store and returns the findings matching the filter, oldest first.

A finding with a zero timestamp is dropped when a Since bound is set (it cannot be proven to fall within the window), consistent with the original tool which skipped findings whose timestamp could not be parsed. Without a Since bound, timestamp is ignored entirely.

SearchFindings returns an error only when the store cannot be read or a record is not valid finding JSON.

Types

type CheckBaselineInput

type CheckBaselineInput struct {
	Entity    string `json:"entity"`
	Source    string `json:"source,omitempty"`
	EventType string `json:"event_type,omitempty"`
}

CheckBaselineInput is the input for CheckBaseline.

Entity is the actor ID / email to look up (required). Source, when set, additionally requires the actor's profile to carry that source-derived signal — currently a known geo, the only source-shaped field the typed baseline exposes. EventType, when set, selects the frequency bucket reported in FrequencyForType.

type CheckBaselineResult

type CheckBaselineResult struct {
	Known            bool                             `json:"known"`
	LastSeen         string                           `json:"last_seen"`
	Frequency        int                              `json:"frequency"`
	FrequencyByType  map[string]int                   `json:"frequency_by_type"`
	FrequencyForType int                              `json:"frequency_for_type"`
	EventType        string                           `json:"event_type,omitempty"`
	Roles            []string                         `json:"roles"`
	Relationships    map[string]baseline.Relationship `json:"relationships"`
}

CheckBaselineResult is the output contract for check-baseline.

Known reports whether the entity appears in the baseline. LastSeen is the actor's last-seen timestamp (RFC3339, empty when unknown). Frequency is the aggregate baseline event count across all event types for the entity; FrequencyByType breaks that aggregate down per event type. FrequencyForType is the count for the specific EventType the caller asked about (zero when the caller omits EventType or no events of that type are baselined). Roles is the actor's known role/permission keys. Relationships surfaces the entity's historical actor↔target relationships (FIX 4): each entry is "has this actor acted on this target before, how often, and when". It answers the academy's relationship question the portable eval was previously blind to. Empty map (never nil) when the baseline carries none.

func CheckBaseline

func CheckBaseline(b *baseline.Baseline, in CheckBaselineInput) (CheckBaselineResult, error)

CheckBaseline reports what the baseline knows about an entity.

A nil baseline is treated as "no baseline data": the entity is unknown, frequencies are zero, roles empty. This mirrors the original tool's graceful-not-found behaviour (a missing baseline file means "unknown entity", never an error). CheckBaseline returns an error only when Entity is empty.

type EventView

type EventView struct {
	ID     string `json:"id"`
	Source string `json:"source"`
	Type   string `json:"type"`
	Actor  string `json:"actor"`
	Target string `json:"target"`
	Action string `json:"action"`
	// Metadata carries the discriminating per-event fields (volume magnitude,
	// origin ip/location, grant principal_id/role) the model weighs to tell an
	// attack apart from benign load. Each VALUE is sanitized individually; the map
	// is empty (never nil) when the payload carries none of the projected keys.
	Metadata  map[string]string `json:"metadata"`
	Timestamp string            `json:"timestamp"`
}

EventView is the per-event projection returned in the envelope. It is a flat, stable view over the typed event.Event — id / source / type / actor / target / action / timestamp (RFC3339, empty when zero) PLUS the discriminating per-event metadata fields the model needs to weigh magnitude/origin. The rest of the raw payload is omitted to keep the envelope flat and the schema fixed; consumers that need the full payload read the store directly.

EVAL FIDELITY (FIX 4): Target and Action are projected from the event payload so the model sees WHAT an event did (action) and to WHAT (target) — the same per-event detail legion's academy fed its agent. Without them the portable eval projected only "actor did <event_type>" and the parity number measured a model blind to the relationship structure the academy showed. Each string is sanitized INDIVIDUALLY (control chars stripped, length-capped) on projection so a payload field cannot smuggle control bytes / unbounded text into the transcript.

FIX 1 (DISCRIMINATION, not blunter escalation): the discriminating metadata the model needs to TELL an attack apart from benign load is also projected — the VOLUME magnitude (operation_count / blobs_accessed / bytes_read), the ORIGIN (ip / location), and the grant target (principal_id / role). These are the exact fields VA-03 / CO-02 / UT-01 turn on (847 GetObject calls; a role_assignment to an unknown principal_id at 03:14). Each is projected as a flat string in the Metadata map and sanitized INDIVIDUALLY (sanitizeEventField) — never one concatenated blob — so one field cannot bleed into another and a payload field cannot smuggle control bytes / unbounded text into the boxed transcript.

func EventViewsFor

func EventViewsFor(events []event.Event) []EventView

EventViewsFor projects typed events into the flat EventView the envelope carries, with the discriminating per-event metadata populated and each string sanitized individually (FIX 1). Exported so the eval harness can reason over the SAME projection the model sees (target/action/metadata) when computing the observable force-escalate predicates, instead of re-deriving the projection.

type FilterApplied

type FilterApplied struct {
	Actor     string `json:"actor"`
	Source    string `json:"source"`
	Type      string `json:"type"`
	Since     string `json:"since"`
	Until     string `json:"until"`
	Effective string `json:"effective"`
}

FilterApplied echoes the filter that actually constrained the result. Actor / Source / Type are the equality filters as supplied. Since / Until are the time-window bounds as supplied (RFC3339, empty when unbounded). Effective reports how the time window was applied: "applied" when it constrained the result, "none" when no time bound was supplied, and "dropped" when the window excluded every candidate and the date-hallucination fallback discarded it (§3.6).

type GithubActorEnvelope

type GithubActorEnvelope struct {
	Login        string             `json:"login"`
	Found        bool               `json:"found"`
	Ghost        bool               `json:"ghost"`
	LooksDeleted bool               `json:"looks_deleted"`
	AccountType  string             `json:"account_type"`
	Name         string             `json:"name"`
	Bio          string             `json:"bio"`
	ProfileURL   string             `json:"profile_url"`
	CreatedAt    string             `json:"created_at"`
	PublicRepos  int                `json:"public_repos"`
	Followers    int                `json:"followers"`
	RecentEvents []GithubActorEvent `json:"recent_events"`
	Notes        string             `json:"notes"`
}

GithubActorEnvelope is the canonical, always-same-shape output of GithubActor.

• Found reports whether GET /users/{login} returned a real profile (200). A 404 is Found=false with Notes explaining why — not an error.
• Ghost is true exactly when Login is the reserved `ghost` tombstone login (see package doc) — the one hardcoded, load-bearing signal this tool exists to surface.
• LooksDeleted is true for Ghost, or (best-effort heuristic) when a non-ghost profile's name/bio reads like a deactivated-account placeholder. The heuristic case is always explained in Notes — it is not an API-confirmed fact the way Ghost is.
• AccountType mirrors GitHub's `type` field ("User", "Organization", "Bot"); empty when Found is false.
• RecentEvents is empty (never nil) when the account has no public activity, the events call failed, or (see Notes) was rate-limited.

func GithubActor

func GithubActor(ctx context.Context, in GithubActorInput) (GithubActorEnvelope, error)

GithubActor performs a live, read-only lookup of a GitHub login: its public profile plus recent public activity. Returns an error only for a genuine input/transport failure — never for "user not found", which is a valid Found=false result.

type GithubActorEvent

type GithubActorEvent struct {
	Type      string `json:"type"`
	Repo      string `json:"repo"`
	Timestamp string `json:"timestamp"`
}

GithubActorEvent is one recent public activity entry projected from GET /users/{login}/events/public.

type GithubActorInput

type GithubActorInput struct {
	Login string `json:"login"`
}

GithubActorInput is the input for GithubActor. Login is the GitHub username to look up (required).

type LookupRulesInput

type LookupRulesInput struct {
	FindingID     string `json:"finding_id"`
	FindingFamily string `json:"finding_family"`

	// Flat named predicate fields (the canonical shape).
	MaintenanceWindow    string `json:"maintenance_window,omitempty"`
	Scheduled            string `json:"scheduled,omitempty"`
	ResolutionEvent      string `json:"resolution_event,omitempty"`
	LocationChange       string `json:"location_change,omitempty"`
	AutomationProvenance string `json:"automation_provenance,omitempty"`
	DeployRelease        string `json:"deploy_release,omitempty"`
	SensitiveBulkRead    string `json:"sensitive_bulk_read,omitempty"`
	HrProvisioning       string `json:"hr_provisioning,omitempty"`
	ScenarioPattern      string `json:"scenario_pattern,omitempty"`
	ActorRole            string `json:"actor_role,omitempty"`

	// FindingMetadata is the legacy nested-object input. Retained as a
	// back-compatibility shim. New callers should use the flat fields above.
	FindingMetadata map[string]string `json:"finding_metadata,omitempty"`
}

LookupRulesInput is the input for LookupRules.

FindingID identifies the finding being investigated (echoed back in output). FindingFamily filters rules by AppliesTo.Family (case-insensitive equality).

The metadata predicate is supplied as a flat set of named string fields, not a nested object. This works around a reliability problem with the prior nested object schema: flat named string properties are the schema shape every reliably-called tool uses. Each known flag corresponds 1:1 with a metadata_match key currently used by operator-decisions.yaml.

FindingMetadata is retained as a back-compatibility shim — when callers supply the legacy nested object, it is merged into the assembled flat map. Direct flat fields take precedence over FindingMetadata entries with the same key.

type LookupRulesOutput

type LookupRulesOutput struct {
	FindingID string         `json:"finding_id"`
	Rules     []OperatorRule `json:"rules"`
}

LookupRulesOutput is the result of LookupRules: the finding id echoed back and the matching rules (empty slice, never nil, on no matches).

func LookupRules

func LookupRules(repoRoot string, input LookupRulesInput) (LookupRulesOutput, error)

LookupRules is the pure lookup-rules read tool.

Given a repo root and an input (finding id + family + metadata predicate), it loads the operator-decisions corpus and returns the rules whose applies_to predicate matches. An empty Rules slice (never nil) means "no rules matched" — that is a valid result, never an error. LookupRules returns an error only for a malformed input (missing finding_id / finding_family) or a corpus that cannot be loaded (read error or sha256 mismatch when enforcement is on).

type OperatorRule

type OperatorRule struct {
	ID                string        `yaml:"id" json:"id"`
	AppliesTo         RuleAppliesTo `yaml:"applies_to" json:"applies_to"`
	OperatorDirective string        `yaml:"operator_directive" json:"operator_directive"`
}

OperatorRule is a single rule loaded from operator-decisions.yaml.

func FindRuleByID

func FindRuleByID(rules []OperatorRule, ruleID string) (OperatorRule, bool)

FindRuleByID looks up a rule by its ID (case-insensitive). Returns the rule and true on hit, or zero value and false on miss.

func LoadOperatorRules

func LoadOperatorRules(repoRoot string) ([]OperatorRule, error)

LoadOperatorRules reads operator-decisions.yaml from repoRoot and returns the parsed rules. Returns an empty slice (not an error) when the file does not exist — a missing file means "no pre-seeded rules" rather than an error.

This is the explicit-root entry point (eval + the toolrun explicit-root fallback). It treats the supplied repoRoot as resolved (rootErr == nil), so a missing file under that root degrades to the embed (or, with the embed disabled, to an empty result). For the production scan path — where the root resolver itself may fail — use LoadOperatorRulesResolved with the resolver's error so a resolution failure also routes to the embed.

func LoadOperatorRulesResolved

func LoadOperatorRulesResolved(repoRoot string, rootErr error) ([]OperatorRule, error)

LoadOperatorRulesResolved loads the operator-decisions rules using the filesystem-first / embed-last precedence in corpusBytes. rootErr carries any error from the caller's repo-root resolver (findConfigRoot / resolveRepoRoot): when non-nil, no on-disk read is attempted and the loader falls straight to the embedded corpus. This is what lets the standalone /tmp binary self-resolve the corpus instead of dropping the §3.8 rule fold.

type RuleAppliesTo

type RuleAppliesTo struct {
	Family        string            `yaml:"family" json:"family"`
	MetadataMatch map[string]string `yaml:"metadata_match,omitempty" json:"metadata_match,omitempty"`
}

RuleAppliesTo is the matching predicate for a rule.

Family matches finding.detector (string equality, case-insensitive). MetadataMatch is a conjunctive flat map: every (key, value) pair must appear in the supplied finding metadata for the rule to match. Values are compared case-insensitively as strings.

type SearchEventsEnvelope

type SearchEventsEnvelope struct {
	Events        []EventView    `json:"events"`
	MatchedRules  []OperatorRule `json:"matched_rules"`
	FilterApplied FilterApplied  `json:"filter_applied"`
	Notes         string         `json:"notes"`
}

SearchEventsEnvelope is the canonical wrapped output of the search-events tool. The same shape is returned on every call:

• Events is the matched event set (empty slice, never nil, on no matches).
• MatchedRules is the operator-decisions rules that match the returned events' finding family + metadata (empty slice, never nil). Folding the rule lookup into search-events is the §3.8 fix: the model reliably calls search-events and reliably ignores a standalone lookup-rules, so the rule data rides along on the tool the model already drives.
• FilterApplied echoes the effective filter back for transparency, so the consumer can see what actually constrained the result (including a date-fallback that disabled the time window).
• Notes is free-form: empty string on the happy path, an explanation when something non-obvious happened (date filter excluded everything and was dropped; finding family unknown so no rules matched; rule corpus unreadable so matching was skipped). Never the channel for an error.

Every field carries a json tag and is always populated — JSON marshalling never omits a key.

func SearchEventsWrapped

func SearchEventsWrapped(s *store.Store, in SearchEventsInput, findingFamily string, findingMetadata map[string]string) (SearchEventsEnvelope, error)

SearchEventsWrapped is the agent-facing search-events tool. It returns the canonical SearchEventsEnvelope on EVERY call — same shape, every key present — and folds the operator-decisions rule lookup INTO the response (§3.8): the matched_rules field carries the rules whose applies_to predicate matches the returned events' finding family + metadata. The model reliably calls search-events and reliably ignores a standalone lookup-rules, so the rule data rides along on the tool the model already drives. (LookupRules remains callable for callers that want it directly; search-events is the path that reaches the model.)

Contract guarantees:

• §3.3 one shape always: every field is populated on every call (empty slices, empty strings) — never an omitted key, never a conditional shape.
• §3.4 empty-is-data: no matches returns the wrapped envelope with empty Events / MatchedRules and a Notes explanation. An ERROR is returned ONLY for a genuine schema violation (nil store, unreadable store, malformed record JSON) — never for "the world is empty".
• §3.5 self-resolving config: the rule corpus is located by walking up from the binary, not from CWD or a required env var.
• §3.6 date-hallucination fallback: a time window that excludes every candidate is DROPPED (the unfiltered set is returned) with a Notes line and FilterApplied.Effective = "dropped".

findingFamily selects which operator rules can match (matches finding.detector case-insensitively); pass "" to skip rule matching (matched_rules stays empty, the envelope shape is unchanged). findingMetadata is the flat predicate the rules' metadata_match is evaluated against (event metadata the agent observed: maintenance_window, scheduled, location_change, etc.).

type SearchEventsInput

type SearchEventsInput struct {
	Actor  string    `json:"actor,omitempty"`
	Source string    `json:"source,omitempty"`
	Type   string    `json:"type,omitempty"`
	Since  time.Time `json:"since,omitempty"`
	Until  time.Time `json:"until,omitempty"`
}

SearchEventsInput is the filter for SearchEvents. Every field is optional; an empty filter returns every event in the stream.

Actor / Source / Type are case-insensitive equality filters on the matching event field. Since / Until bound the event timestamp (inclusive). A zero time means "unbounded on that side".

type SearchFindingsInput

type SearchFindingsInput struct {
	Actor  string    `json:"actor,omitempty"`
	Source string    `json:"source,omitempty"`
	Since  time.Time `json:"since,omitempty"`
}

SearchFindingsInput is the filter for SearchFindings. Every field is optional; an empty filter returns every finding in the stream.

Actor / Source are case-insensitive equality filters. Since bounds the finding timestamp (inclusive lower bound); a zero time means "unbounded".