Documentation
¶
Overview ¶
Package runtime orchestrates the collect→ingest→build→emit pipeline. Unlike AWSHound's multiprocess builder, graph construction is a single in-process pass over the shared store; goroutine parallelism can be layered on per phase without any cross-process state sharing.
Index ¶
- Constants
- Variables
- func AccountIDFromGAAD(g *model.AuthDetails) string
- func BoundedParallel(max, n int, fn func(i int))
- func BuildGraph(s *store.Store) ([]*graph.Node, []*graph.Edge, error)
- func BuildGraphWithProgress(s *store.Store, p BuildProgress) ([]*graph.Node, []*graph.Edge, error)
- func CollectUnits(names []string, regions int) int
- func IngestData(s *store.Store, accountID, service string, data any) error
- func OpenJSONArtifact(path string) (io.ReadCloser, error)
- func StreamBuildToBuilt(s *store.Store, w io.Writer, p BuildProgress, timer EdgeFamilyTimer) (int, int, error)
- func WriteZIP(path, member string, write func(io.Writer) error) error
- func ZIPOutputPath(path, defaultName string) string
- type BuildProgress
- type CollectError
- type CollectErrorLog
- type CollectResult
- type EdgeFamilyTimer
- type FetchResult
Constants ¶
const ( BuiltArtifactName = "built.json" GraphArtifactName = "graph.json" )
Variables ¶
var DefaultCollectors = []string{
"iam", "kms", "ssm", "ec2", "lambda", "cloudformation", "eks", "organizations",
}
DefaultCollectors is the set collected when no --collectors is given, in a stable order. s3 is deliberately excluded: its per-bucket control-plane + object-listing walk runs serially per account and dominates run time on bucket-heavy accounts (see internal/collect/s3.go), so it is opt-in via `--collectors ...,s3` rather than part of the default sweep. Every collector in allServices remains selectable by name.
Functions ¶
func AccountIDFromGAAD ¶
func AccountIDFromGAAD(g *model.AuthDetails) string
AccountIDFromGAAD derives the account id from the first principal ARN in a GAAD response (arn:aws:iam::<account>:...). Returns "" if none is found.
func BoundedParallel ¶
BoundedParallel runs fn(0), …, fn(n-1) concurrently, but with at most max in flight at once. A max <= 0 means unbounded. It is the account-level throttle that bounds how many accounts' heavy data are resident simultaneously (§9 of docs/authorization-tensor.md), replacing the collect fan-out's spawn-all wg.Add(len(boots)).
func BuildGraph ¶
BuildGraph turns a populated store into the OpenGraph node and edge sets: it constructs the policy evaluator, runs every node parser (merging same-id nodes across accounts), and runs every edge family (deduped).
func BuildGraphWithProgress ¶
BuildGraphWithProgress is BuildGraph instrumented for progress: it reports a "Build nodes" stage advancing per account and a "Build edges" stage advancing per edge family. A nil p makes it behave exactly like BuildGraph.
func CollectUnits ¶
CollectUnits returns the number of leaf collection units a single account will produce for the selected services given its discovered region count: each region-scoped service contributes one unit per region (one fanRegions worker), and each global service contributes one. Callers sum it across accounts to seed the Collect stage's progress total once bootstrap has resolved each account's regions.
It is the initial denominator, not the final one: the global collectors (iam, s3, organizations) grow the total at runtime via Session.addWork as they discover their sub-work (pages/buckets/policies) and emit one matching tick per sub-item, so a long single-unit global collector advances visibly instead of appearing hung on its lone completion tick. Each added unit is matched by a later tick, so the stage still finishes at 100%.
func IngestData ¶
IngestData ingests a single raw fetch payload (as returned by FetchAll's FetchResult.Data) into the store under accountID, dispatching by service name to the same typed ingest the live collect path uses. It lets a caller keep collected data in memory while also persisting the raw JSON, avoiding a read-back from disk. Returns an error for an unknown service or a payload whose type doesn't match the service.
func OpenJSONArtifact ¶
func OpenJSONArtifact(path string) (io.ReadCloser, error)
OpenJSONArtifact opens a raw JSON file or the first JSON member in a ZIP. The returned reader owns all underlying descriptors and must be closed.
func StreamBuildToBuilt ¶
func StreamBuildToBuilt(s *store.Store, w io.Writer, p BuildProgress, timer EdgeFamilyTimer) (int, int, error)
func WriteZIP ¶
WriteZIP streams one JSON member into a ZIP archive without materializing the uncompressed artifact on disk. Best-speed DEFLATE substantially reduces large repetitive graph JSON while keeping the long-running build CPU-oriented.
func ZIPOutputPath ¶
ZIPOutputPath converts a requested output file into a ZIP archive path. A directory request receives defaultName; a non-.zip file request gets .zip appended so existing -o graph.json usage produces graph.json.zip.
Types ¶
type BuildProgress ¶
BuildProgress receives stage/advance signals from BuildGraphWithProgress so a caller can drive a progress display through the build. It is satisfied by *progress.Tracker (and progress.Reporter); a nil BuildProgress disables reporting. Edge progress advances once per completed family; an optional SetStageLabel(string) method lets a display name the family currently running without resetting the counter.
type CollectError ¶
type CollectError struct {
AccountID string `json:"account_id"`
Profile string `json:"profile"`
Service string `json:"service"`
Region string `json:"region,omitempty"`
Message string `json:"message"`
}
CollectError is one bootstrap or service-level collection failure. Region is currently always "" (errors are service-scoped); the field exists so a later per-region/per-resource capture can populate it without changing the schema.
type CollectErrorLog ¶
type CollectErrorLog struct {
// contains filtered or unexported fields
}
CollectErrorLog accumulates bootstrap and service-level collection errors across every account and service in a run. It is safe for concurrent use.
func NewCollectErrorLog ¶
func NewCollectErrorLog() *CollectErrorLog
NewCollectErrorLog returns an empty error log ready for concurrent Record calls.
func (*CollectErrorLog) CountsByService ¶
func (l *CollectErrorLog) CountsByService() map[string]int
CountsByService returns the number of errors recorded per service.
func (*CollectErrorLog) Len ¶
func (l *CollectErrorLog) Len() int
Len reports how many errors have been recorded. A nil receiver reports 0.
func (*CollectErrorLog) Record ¶
func (l *CollectErrorLog) Record(e CollectError)
Record appends one collection error. Safe for concurrent use; a nil receiver is a no-op.
func (*CollectErrorLog) SummaryByService ¶
func (l *CollectErrorLog) SummaryByService() string
SummaryByService renders the per-service counts as "svc:count, svc:count", ordered by service name, for a one-line summary. Returns "" when empty.
func (*CollectErrorLog) WriteJSON ¶
func (l *CollectErrorLog) WriteJSON(path string) error
WriteJSON writes the accumulated errors to path as a structured report (total, per-service counts, and the full error list). It is a no-op returning nil when no errors were recorded, so callers can invoke it unconditionally without leaving an empty file behind on a clean run.
type CollectResult ¶
CollectResult reports one service's collection outcome and how long its Fetch+Ingest took (for run metrics).
func Collect ¶
func Collect(ctx context.Context, s *store.Store, sess *collect.Session, names []string) ([]CollectResult, error)
Collect runs the named collectors against one session concurrently (one goroutine per service — the goroutine model that replaces AWSHound's ThreadPoolExecutor fan-out), ingesting each into the shared thread-safe store. Best-effort: a failing collector is reported in its CollectResult rather than aborting the batch, mirroring the Python; the caller decides how to surface it (awshound routes it through the progress display). Per-unit progress is reported through the session's progress callback (see Session.SetProgress).
type EdgeFamilyTimer ¶
StreamBuildToBuilt builds the graph and writes the built.json artifact to w while bounding node memory: nodes are streamed straight to w through build.StreamNodes (the volume-scaling node set never accumulates), then edges are appended. The bytes are identical to BuildGraph + graph.WriteBuilt. Returns the node and edge counts. A nil p disables progress reporting. EdgeFamilyTimer receives a per-edge-family wall-clock timing (the --metrics per-processor breakdown). *metrics.RunMetrics satisfies it; a nil value disables the breakdown.
type FetchResult ¶
FetchResult is one service's raw fetched payload, for offline raw output. Data is the concrete *model.<Svc>Data (or *model.AuthDetails for iam) the collector returned, ready to json.Marshal to a file the `collect` subcommand writes and the `process`/ingest path later reads back.
func FetchAll ¶
FetchAll runs the named collectors concurrently against one session and returns each service's raw fetched data without ingesting it. It mirrors Collect's fan-out, shared limiter, and best-effort semantics (a failing collector is reported in its FetchResult), but stops at Fetch so the caller can serialize the raw payloads — the `collect` subcommand's path. Per-unit progress still flows through the session's progress callback, so the Collect stage advances the same way as a live-into-store run.
func FetchAllWithProgress ¶
func FetchAllWithProgress(ctx context.Context, sess *collect.Session, names []string, onStart func(service string)) ([]FetchResult, error)
FetchAllWithProgress is FetchAll with an optional callback that fires as each service collector starts. It lets verbose callers identify current work while normal callers retain FetchAll's quiet behavior.
Source Files