Documentation
¶
Overview ¶
Package audit implements the `local-review audit` subcommand — the deep-analysis mode introduced in v0.10.0-c.
Unlike the `review` subcommand which inspects a git diff, `audit` walks the whole committed source tree, groups files by directory, and runs each directory's source through the LLM with a topic- specific system prompt (security, tech-debt, …). The output is a markdown report listing findings per package — accumulated debt / vulnerabilities / smells that no individual diff would have surfaced.
Design constraints (from the v0.10.0 planning discussion):
- Topic-driven, not open-ended ("find all bugs" produces shallow LLM output; "find security gaps" produces actionable output).
- Chunked by directory so the LLM gets a coherent unit of scope, and so a single 100k-LOC monorepo doesn't blow context windows.
- Single-LLM by default ("audit" cost is per-package × per-topic, multiplying by 3 LLMs would put a real-codebase audit in $20+ territory). Multi-LLM is a later enhancement once we see real usage patterns.
- Cost transparency: pre-flight `--dry-run` shows the plan without invoking; the runner prints a per-chunk progress line.
Index ¶
Constants ¶
const MaxAuditParallelism = 16
MaxAuditParallelism is the hard ceiling on the worker pool size, regardless of what the user passes via --parallel. Picked to match the workflow concurrency cap elsewhere in the codebase:
- Spawning more workers than chunks is pointless (nothing for the extras to do); we clamp to len(chunks) too.
- Beyond ~16 concurrent inference calls, the bottleneck is almost always GPU memory / vendor rate limits on the backend side, not the runner; letting the user spawn 1000 goroutines just degrades reliability without buying throughput.
- The 16 ceiling matches min(16, cpu_cores-2) used by the workflow harness — same reasoning, same shape.
A user with a beefier setup who genuinely wants >16 can either raise this constant locally or, more reasonably, run multiple audits in parallel from a shell.
Variables ¶
This section is empty.
Functions ¶
func FormatBytes ¶
FormatBytes renders a byte count in human-readable units (B / KiB / MiB). Exported so the cmd-layer dry-run preview can render chunk sizes in the same units the runner's progress output uses during a real audit — single source of truth, no drift between preview and run. Inputs above ~10 MiB stay in MiB rather than escalating to GiB (audit chunks shouldn't reach a GiB; if they do, the soft-cap warning is the bigger problem).
func WriteJSON ¶
WriteJSON emits the full Report as indented JSON. Useful for downstream tooling that wants to diff audit deltas between commits.
func WriteMarkdown ¶
WriteMarkdown renders a committable audit report — same shape as the bench RESULTS.md leaderboard but for audit data. Default sink for the --out flag when the path ends in .md.
Sections:
# Audit — <topic> _Generated <ts>_ · _LLM: <llm>_ · _Packages: N_ · _Findings: N_ ## Summary | Severity | Count | | ... | ... | ## <package> (N files) - [severity] path:line — body - ...
Clean packages are folded into one trailing "Clean packages" line to keep the report scannable.
Types ¶
type Chunk ¶
type Chunk struct {
// Package is the repo-relative directory path. Files at the
// repo root are bucketed under ".".
Package string
// Files lists the repo-relative paths included in this chunk,
// sorted for deterministic ordering across audit runs.
Files []string
// Body is the LLM-ready concatenation: each file preceded by a
// `// === FILE: <path> ===` marker. The audit pack tells the LLM
// to expect that delimiter shape.
Body string
// SizeBytes is the total size of Body — used by the runner to
// warn / split when a package would overflow a typical LLM
// context window. v1 just emits a warning; the soft split
// strategy is deferred.
SizeBytes int
}
Chunk is one unit of audit work — all the source from one directory, ready to feed into the LLM. The walker produces these; the runner consumes them.
func Walk ¶
func Walk(opts WalkOptions) ([]Chunk, error)
Walk returns one Chunk per directory containing audit-eligible files. Directories with no eligible files are skipped silently (no empty chunks in the output).
Eligibility:
- File is tracked by git (we use TrackedFiles to enumerate).
- File extension maps to a known language pack via internal/lang.Detect, OR matches isAuditable's small built-in allowlist of common config / build / script shapes (Bash, YAML, SQL, Dockerfile, Terraform). Files whose extension lang.Detect classifies as `default` are SKIPPED (binary / image / archive / unknown text) unless they're on the allowlist — keeping audit input focused on source the LLM can usefully reason about.
- File survives the Include/Exclude filters.
Output is sorted by Package (alphabetical) for deterministic audit runs.
type Finding ¶
type Finding struct {
// Path is repo-relative — the path the LLM returned in the
// finding header. Always non-empty for findings produced by
// the v1 parser: findingHeaderRE requires a non-empty path
// token, so a header without one doesn't match and the
// would-be finding is dropped. (Earlier drafts had a chunk-
// package fallback but the regex made it unreachable — see
// PR #73 review for the cleanup.)
Path string `json:"path,omitempty"`
// Line is the (start) line number the LLM cited (best-effort;
// LLMs vary in line-accuracy across audit mode, where they're
// reading more context than in diff mode). Zero = unlocated.
Line int `json:"line,omitempty"`
// LineEnd is the end line of a `LINE-RANGE` citation
// (`file.go:12-18`). Zero when the LLM gave a single line
// (the common case) or when the line was elided entirely.
// Audit packs document the range shape; the v1 renderer
// surfaces it inline as "file:start-end" when present.
LineEnd int `json:"line_end,omitempty"`
// Severity is one of "critical", "major", "warning", "info".
// Audit packs deliberately skip "nit" — whole-codebase reading
// produces enough signal that nits dilute the report.
Severity string `json:"severity"`
// Body is the finding text the LLM produced, lightly cleaned up
// (trimmed, severity prefix stripped). Renderer formats it.
Body string `json:"body"`
}
Finding is one issue surfaced by the audit. Mirrors the review path's finding shape so consumers that already render review JSON can reuse most of the pipeline.
type Options ¶
type Options struct {
// Topic is the audit pack id ("security", "tech-debt", …).
// Required; the runner refuses an empty value (rather than
// falling back to a default) because the choice of topic is
// the whole point of audit mode.
Topic string
// LLM is the agent the runner invokes per chunk. Single-LLM by
// design in v1 — multi-LLM audit would multiply the per-chunk
// cost without obvious quality return; deferred until we see
// real usage. Caller is responsible for picking an authenticated
// LLM via the existing cli.DetectAll / config flow.
LLM cli.LLM
// Timeout per chunk. Zero falls back to LLM.TimeoutSec, then
// to 300 seconds (5 min — audit chunks can be larger than
// review diffs so we give them more headroom).
Timeout time.Duration
// Progress, when non-nil, receives a one-line message after
// each chunk completes. Hooked up by the CLI to print live
// progress to stderr so an audit on a large repo doesn't look
// hung. Pass nil from tests.
Progress io.Writer
// Parallelism caps the number of chunks dispatched to the LLM
// concurrently. Default (zero or 1) preserves the strict
// sequential ordering audit shipped with in v0.10-v0.15.0.
// Setting >1 fans out N chunks at a time to the same agent,
// which is the right knob for Ollama with `OLLAMA_NUM_PARALLEL`
// configured (or any backend that serves concurrent requests).
//
// Returned PackageReports stay in chunk order regardless of
// completion order — internal write to a pre-sized slice by
// index, not append.
//
// Constraints:
// - Cloud LLM rate limits: claude / codex tier limits may
// rate-limit at >1; users on cloud should leave Parallelism=1.
// - Local model VRAM: a 7B model on 12GB Apple Silicon can
// sustain ~2 concurrent; a 32B on 24GB ~3. The runner
// doesn't introspect; if you OOM the server, lower the
// flag.
//
// v0.15.1: added per the audit-perf patch after user-reported
// 22-minute runs on a 37-chunk repo via Ollama qwen 7B.
Parallelism int
// Invoker is an unexported test seam: when non-nil, Run uses it
// directly instead of `cli.NewInvoker(opts.LLM)`. Production
// callers leave it nil; tests inject a fake so parallel-
// dispatch behaviour (concurrent calls observed, results
// stay in chunk order) is verifiable without touching real
// LLM subprocess / HTTP plumbing. Fakes need only satisfy
// cli.Invoker.Review — RunPrompt isn't exercised by audit.
Invoker cli.Invoker
}
Options configure one audit Run.
type PackageReport ¶
type PackageReport struct {
// Package is the repo-relative directory path. Top-level files
// (e.g. main.go in repo root) live under "." by convention.
Package string `json:"package"`
// Files lists the repo-relative paths included in this chunk.
// Used by the renderer to show "audited N files in pkg/" lines
// and by the runner to detect "package too big to chunk."
Files []string `json:"files"`
// Findings is the parsed list. Empty when the LLM returned
// clean OR when the LLM errored (Error captures the failure).
Findings []Finding `json:"findings,omitempty"`
// Clean is the canonical "audited and found nothing" signal:
// true when the LLM emitted the `[clean] no findings in this
// package` sentinel and the runner recognised it. Empty
// Findings alone is ambiguous — could be a clean run, a
// parse miss, or an error frame — so consumers checking for
// "this package is clean" should use this field, not
// len(Findings) == 0. Raw is still populated on clean runs
// (the runner stores the LLM output before deciding) so
// future-self / parser-debug paths can see what the LLM
// actually said.
Clean bool `json:"clean,omitempty"`
// Raw is the LLM's untrimmed response. Kept on the report so a
// reviewer with a parser failure can see what the LLM actually
// said. Renderer doesn't surface it by default.
Raw string `json:"raw,omitempty"`
// Timing + status.
DurationMs int64 `json:"duration_ms"`
Error string `json:"error,omitempty"`
// InputTokens / OutputTokens carry the same semantics as the
// review path's cli.TokenUsage (zero = unknown, not zero-spent).
InputTokens int `json:"input_tokens,omitempty"`
OutputTokens int `json:"output_tokens,omitempty"`
}
PackageReport is the per-package output: one chunk, one LLM invocation, one set of findings. The canonical "audited and found nothing" signal is the Clean field, NOT empty Findings (which is ambiguous — could be a clean run, a parser miss, or an error frame). The renderer collapses Clean rows into a single tail line. Raw stays populated on clean runs too, so parser-debug paths can see what the LLM actually said.
type Report ¶
type Report struct {
Topic string `json:"topic"`
Generated time.Time `json:"generated"`
Root string `json:"root"`
LLM string `json:"llm"`
Version string `json:"version,omitempty"`
Packages []PackageReport `json:"packages"`
// Aggregate counts, summed across packages. The renderer uses
// these for the summary line; consumers can re-derive from
// Packages but the counts on the top level cost nothing to
// emit.
TotalFindings int `json:"total_findings"`
FindingsBySeverity map[string]int `json:"findings_by_severity,omitempty"`
PackagesWithFindings int `json:"packages_with_findings"`
PackagesClean int `json:"packages_clean"`
PackagesErrored int `json:"packages_errored,omitempty"`
// Total token usage across all packages.
TotalInputTokens int `json:"total_input_tokens,omitempty"`
TotalOutputTokens int `json:"total_output_tokens,omitempty"`
}
Report is the top-level audit output, suitable for JSON serialization and for the text/markdown renderers in report.go.
func Run ¶
Run executes the audit: for each chunk, invoke the LLM with the topic pack as system prompt and the chunk body as input. Per- chunk errors are recorded on the PackageReport (not propagated) so one transient LLM failure on package N doesn't abort the other packages.
Returns a Report with aggregate counts filled in. The caller renders to text / markdown / JSON via internal/audit/report.go.
func (Report) Filtered ¶ added in v0.16.0
Filtered returns a copy of the report keeping only findings at or above minSeverity (when non-empty) and capping the total number of findings at maxFindings (when > 0, across packages in report order). Aggregates are recomputed on the filtered set. The second return is the number of findings hidden, so the caller can disclose the truncation rather than dropping it silently (CLAUDE.md rule 4). With no floor and no cap the report is returned unchanged and hidden is 0.
type WalkOptions ¶
type WalkOptions struct {
// Root is the working-tree root the audit operates against.
// Empty = current working directory.
Root string
// Include / Exclude are optional path-prefix filters. Both
// match against the repo-relative path. Include wins (when
// non-empty) — only files under one of the Include prefixes
// are considered, then Exclude further removes matches. Used
// by `--include` / `--exclude` CLI flags so users can audit
// just one subdirectory.
Include []string
Exclude []string
// MaxBytesPerChunk caps the LLM input per chunk. Packages over
// this size are auto-split into `pkg [part N/M]` sub-chunks via
// the greedy bin-packer in splitChunk; single files individually
// over the cap surface a warning and pass through as one chunk
// (splitting a source file at an arbitrary line boundary would
// produce semantically broken chunks). Zero = use the package
// default (96 KiB — empirical headroom needed for claude-code's
// own system prompt + tool definitions on top of the audit pack
// body). Negative values are rejected by Walk at load time —
// silently letting a negative cap flow through would make every
// file appear oversized and produce nonsense warnings.
MaxBytesPerChunk int
// Warn, when non-nil, receives a one-line message per
// over-sized chunk so the user sees the warning before paying
// LLM tokens on a chunk that may not survive the context
// window. The CLI wires this to os.Stderr; tests pass nil.
//
// Walk is sequential — writes to Warn happen one at a time
// from a single goroutine, so the writer doesn't need to be
// thread-safe. If Walk ever fans out across goroutines
// (currently it doesn't; per-chunk LLM calls already give
// us enough parallelism upstream in Run), this field's
// contract changes and synchronization moves into the
// caller.
Warn io.Writer
}
WalkOptions configure which tracked files the walker considers. Zero value = "every tracked text file under the working tree."
Source Files