Documentation
¶
Overview ¶
Package discover walks a repository and decides what exists: which files signpost will analyse, what kind of thing each one is, and how much of each is safe to read.
It is the first stage of the pipeline (design §4.0) and every later stage consumes its output, so two properties are load-bearing:
- Deterministic. Files come back in sorted order with no dependence on filesystem enumeration order, because the bundle is committed and any instability here becomes commit churn (design §8.1).
- Bounded. A repository is untrusted input: it can contain a 4 GB generated file, a symlink loop, or a file that is one 200 MB line. None of those may turn a build into an OOM or a hang.
Index ¶
Constants ¶
const ( MaxFullBytes = 2 << 20 // 2 MiB MaxFullLines = 50_000 // HeadTailBytes is how much is read from each end of an oversized file. HeadTailBytes = 32 << 10 // 32 KiB // DefaultMaxTotalBytes caps the whole walk. A repo that exceeds it is still // processed, but remaining files are recorded as skipped rather than read, // so a pathological tree cannot exhaust memory. // // A default rather than the limit, and exported, because no single number is right // for every tree: file contents are held in memory for the whole analysis, so this // is the ceiling on how much of a tree gets read at all. Options.MaxTotalBytes // overrides it in either direction. Exported so the flag that does so can state // the default it is replacing rather than restating the number. // // Raised from 512 MiB, which truncated the case this tool is for. A monorepo of // roughly 275,000 files recorded 170,530 of them as skipped and reported its own // first-party packages as unresolved imports, because the files defining them were // never opened — and a partial map is the failure that looks like success, since // nothing downstream can tell an absent module from one that does not exist. The // flag was the remedy for a truncated walk and it is still the remedy for a larger // one, but a default that leaves a repository this tool is aimed at three-fifths // unread is the wrong default. Six times the budget is chosen against that ratio // rather than against a measured run of the whole tree, so it is a better default // and not a guarantee: a tree large enough still truncates, and still says so. // // It is a memory ceiling and reads as one: the number is what a walk may hold, not // what it will, and every tree small enough to finish under 512 MiB allocates // exactly what it did before. Raising it costs nothing until a tree is large enough // to need it, which is the direction the trade should run. DefaultMaxTotalBytes = 3 << 30 // 3 GiB )
Size caps, mirroring codeatlas (design §4.0). A file within both caps is ingested whole; a file over either is recorded with metadata plus a head/tail slice so it still contributes structure without being fully read.
const Elision = "\n\n<<< signpost: content elided >>>\n\n"
Elision separates the head and tail of a truncated file. It is a comment in no language, so an extractor cannot mistake it for code.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Class ¶
type Class string
Class is what kind of thing a file is, which decides which extractor sees it.
const ( ClassSource Class = "source" // code, dispatched by Lang ClassManifest Class = "manifest" // dependency/build manifest ClassInfra Class = "infra" // containers, compose, workflows, k8s, helm ClassContract Class = "contract" // proto, OpenAPI, GraphQL SDL ClassMigration Class = "migration" ClassDoc Class = "doc" // markdown, rst, adoc ClassOwnership Class = "ownership" // CODEOWNERS, AGENTS.md, CLAUDE.md ClassData Class = "data" // json/yaml/toml that is not a known manifest ClassOther Class = "other" )
type File ¶
type File struct {
// Path is slash-separated and relative to the walk root, on every platform.
// Windows backslashes are normalised here so that every downstream artifact
// — node IDs, OKF links, manifest.json — is byte-identical across platforms.
Path string
Class Class
Lang Lang
Size int64
Lines int
// Content is the file's text. Empty for binaries and for files skipped by the
// total-bytes cap. For oversized files it holds head and tail separated by
// Elision, and Truncated is set.
Content string
Truncated bool
// IsTest marks test files: kept for tested_by edges, never counted as
// production surface.
IsTest bool
// Vendored marks third-party code committed into the tree. Discovered for
// the record, excluded from analysis.
Vendored bool
// Fixture marks a sample project kept for tests to run against — testdata/
// and friends. Discovered for the record, excluded from analysis: its modules
// and dependencies belong to the sample, not to this repository. See
// isFixture for why this is neither Vendored nor IsTest.
Fixture bool
// Binary marks a file whose content was not read.
Binary bool
}
File is one discovered file.
type Lang ¶
type Lang string
Lang is the source language, empty for non-source files.
const ( LangGo Lang = "go" LangTS Lang = "typescript" LangJS Lang = "javascript" LangPython Lang = "python" LangRust Lang = "rust" LangJava Lang = "java" LangKotlin Lang = "kotlin" LangC Lang = "c" LangCpp Lang = "cpp" LangObjC Lang = "objc" LangRuby Lang = "ruby" LangPHP Lang = "php" LangCSharp Lang = "csharp" LangShell Lang = "shell" // LangPowerShell is separate from LangShell rather than a dialect of it. The two // share `#` as a comment and nothing else — different function syntax, different // import syntax, different scoping, different resolution — so one extractor bent // two ways would be worse than two. That is the inverse of the C-family case // (ADR 0022), where the dialects share a preprocessor and .h serves all three, so // the boundary is not one a filename or an extractor can see. LangPowerShell Lang = "powershell" // LangVue, LangSvelte and LangAstro are single-file component formats rather than // languages, and they are three Langs rather than one because a Lang is what the // bundle *names*. A module page states the language of the directory it describes and // manifest.json scores each extractor per language, so folding these into one // `component` would tell a reader that a Vue repository and a Svelte one are the same // thing — and would attribute one framework's extraction score to the other. // Resolution is shared rather than split: all three go through resolveTS, since a // component's imports are its script's imports and the script is resolved by the same // tsconfig aliases and the same package.json as any `.ts` file beside it. So is // extraction: one SFCExtractor reads all three, because their script blocks are the // same TypeScript. That is the opposite split from shell and PowerShell above, where // one Lang each and one extractor each was right. LangVue Lang = "vue" LangSvelte Lang = "svelte" LangAstro Lang = "astro" LangOther Lang = "other" )
type Options ¶
type Options struct {
// IncludeVendored analyses vendored code instead of only recording it.
IncludeVendored bool
// IncludeFixtures analyses sample projects under testdata/ instead of only
// recording them.
//
// The escape hatch for isFixture guessing wrong about a directory genuinely
// named `fixtures`, and the counterpart to IncludeVendored. Notably *not* what
// the corpus harness uses: it copies testdata/corpus to a root of its own, so
// those files arrive as `go/greeter/...` with no `testdata` segment to match.
// Analysing a fixture in place and analysing it as its own repository are
// different things, and only the second gives it correct module paths.
IncludeFixtures bool
// ExtraIgnores are additional .gitignore-syntax patterns applied at the root.
ExtraIgnores []string
// MaxTotalBytes overrides DefaultMaxTotalBytes for this walk. Zero or negative
// means the default.
//
// Not an "unlimited" option, deliberately: contents are held in memory, so an
// uncapped walk of an arbitrarily large tree is an out-of-memory kill rather than
// a slow success, and the caller is better placed to know how much memory it has
// than this package is. Raising the cap is a number the caller states; removing it
// is not offered.
MaxTotalBytes int64
}
Options configures a walk.
type Result ¶
type Result struct {
Root string
Files []File
// Skipped records paths that were deliberately not read, with a reason.
// Surfaced in manifest.json so the bundle never presents an incomplete walk
// as a complete one (design §4.2: absence of measurement is never a clean
// bill of health).
Skipped []Skip
// IncludeVendored carries Options.IncludeVendored forward, so the consumers
// that filter on File.Vendored can honour the flag. Without it they cannot:
// every one of them holds the walk's result and not the options that produced
// it, which is how -include-vendored spent v0.1.0 reading vendored files that
// nothing downstream would look at.
IncludeVendored bool
}
Result is the outcome of a walk.
func Walk ¶
Walk discovers files under root.
It honours .gitignore at every level, skips binaries, applies the size caps, and returns files sorted by path. Symlinks are recorded but never followed: following them invites both cycles and escapes outside the root, and a symlink target inside the repo is discovered on its own anyway.
Every read goes through an os.Root scoped to the tree. The symlink skip above already prevents an escape, but that is an argument about the code being right, and this is a guarantee from the kernel handle instead. It is worth the difference here because signpost reads a tree it does not control and commits what it found: a path that escaped the root would put content from outside the repository into a file that gets pushed.
func (*Result) Analyses ¶ added in v0.2.0
Analyses reports whether f is content this walk was asked to analyse.
The one place the vendored decision is made, for the reason issue #11 records: it was previously made independently at six call sites, each spelled `!f.Vendored` with no reference to the option, so the flag that exists to overrule it overruled nothing. Consumers keep their own other conditions — a test file, a class, a binary — and defer only this one.
Two of the six decide something; the rest are belt and braces. Sources() gates extraction and manifest.Registry.Run gates the manifest readers, and reverting either one alone is observable in the bundle. The others — ByClass, practice's two counts, semantic's source picker — only ever see a vendored file when one of those two let it through, because with the flag off the walk prunes vendored directories entirely and nothing vendored reaches a consumer at all. They are kept honest anyway: a filter that contradicted this one would be a hole waiting for the day the walk changes.
Deliberately not a question about fixtures as well. A fixture is pruned during the walk and never reaches a consumer, so there is no downstream filter to satisfy and nothing here to ask.
func (*Result) ByClass ¶
ByClass returns files of a given class, excluding binaries and — unless the walk was asked for them — vendored ones.
func (*Result) Sources ¶
Sources returns only the analysable source files: not vendored unless the walk was asked for vendored code, not binary, with content. This is what the language extractors consume, which makes it the method -include-vendored has to reach — extraction is driven from here, so a vendored file excluded at this point is a vendored file no extractor ever sees whatever the flag said.
func (*Result) Unclassified ¶ added in v0.2.0
Unclassified counts the files this walk could not name, keyed by extension and by basename for the extensionless, highest count first being the caller's job.
ClassOther is the one classification that means "signpost does not know what this is": every other class routes to an extractor or a manifest reader, and the two that can still come back empty-handed report it themselves — extract.RunResult and manifest.RunResult both carry an Unhandled map. ClassOther had no such counterpart, so a file landing here left the pipeline with nothing recording that it had. On a repository whose only frontend source was two `.astro` files, that made the coverage report name `.sh` and `.sql` while the pages it did not read went unmentioned — the silence design §4.2 exists to forbid.
Binaries are excluded. A `.png` is not a gap in coverage: it was classified correctly and there is nothing in it to read, and counting it would bury the extensions that are gaps under the ones that never could be.
Source Files