Documentation
¶
Overview ¶
Package extract composes the engine's source, closure, patch, relocation, and rewriting phases into one read-only extraction plan for a single upstream ref.
A plan answers a question rather than producing a release: given this profile and this source commit, exactly which files would the generated module hold, what would be pruned, which patches would apply, what would be rewritten, and what does the result hash to. It stops before module generation, facade generation, history replay, and publication, which are later phases with their own gates.
Three properties bound what a plan may do.
It is read-only outward. The command never pushes, never creates or moves a ref, and never creates a tag; the only writes it performs are inside the cache and work directories it was given and, when explicitly asked, the output tree. The source cache is driven by an anonymous runner, so a credential that exists for publishing cannot travel to the source host.
It is contained. Cache, work, output, and report roots are absolute and checked, the materialized work tree is one this run created below the work root rather than any directory an operator already had, and every read of that tree goes through an os.Root that re-checks containment per operation.
It is deterministic. Two runs over one source commit with different directory layouts produce byte identical reports, manifests, and trees. The report therefore carries no absolute path, no environment value, and no secret: what it records is the profile, the source commit, and the content the two produce.
Index ¶
- Constants
- Variables
- type AnchorReport
- type ClosureReport
- type EmbedReport
- type EngineReport
- type FailureReport
- type GoldenReport
- type Options
- type OutputReport
- type PatchFailure
- type PatchReassert
- type PatchesReport
- type Paths
- type PolicyError
- type Ref
- type RefKind
- type RelocatedFile
- type RelocatedPackage
- type RelocationReport
- type Report
- type Result
- type RewriteReport
- type RewrittenFile
- type SourceReport
- type WorktreeReport
Constants ¶
const ( // GoldenAbsent reports a pinned golden that is not in the profile // directory. It is a notice rather than a failure, because the first run // that establishes a closure has nothing to compare against yet. GoldenAbsent = "absent" // GoldenMatch reports an exact shape identical to the golden's. GoldenMatch = "match" // GoldenDiff reports an exact shape that disagrees with the golden's. GoldenDiff = "diff" )
Golden comparison outcomes.
const ReportSchema = 1
ReportSchema is the version of the report shape.
It is the first field of every report so a consumer can refuse a shape it does not understand before parsing the rest. It changes only when a field is removed or its meaning changes; adding a field does not.
Variables ¶
var ErrCredentialEnvironment = errors.New("a plan must run without publishing credentials")
ErrCredentialEnvironment reports a publishing credential visible to a command that must never use one.
var ErrPathConflict = errors.New("the output tree conflicts with a directory the run owns")
ErrPathConflict reports an output tree that would sit where the run's own state lives.
It is exported so the command line can report it as the usage failure it is. Every directory involved is one the operator named or defaulted, the check reads nothing and creates nothing, and the answer does not depend on a cache, a profile, or a network existing, so it belongs with the other flag problems rather than with the failures a run discovers.
Functions ¶
This section is empty.
Types ¶
type AnchorReport ¶
type AnchorReport struct {
// Commit is the scratch commit's object name.
Commit string `json:"commit"`
// Tree is the pruned tree it records.
Tree string `json:"tree"`
// Parent is the upstream commit it was made on top of.
Parent string `json:"parent"`
// StagedDeletions are the pruned paths staged into it, sorted.
StagedDeletions []string `json:"stagedDeletions"`
}
AnchorReport records the scratch commit made in the detached work tree.
The commit is unreachable from every ref and is never pushed. It exists because three way patch application resolves blobs through the index and the object store, and its rollback restores exactly one committed state, so the pruned tree has to be a commit before a patch may touch it. Its object name is reported because it is deterministic: a fixed identity, a fixed date, a fixed message, and the pruned tree of a known parent leave nothing to vary.
type ClosureReport ¶
type ClosureReport struct {
// Rounds is how many closure builds the plan performed before the package
// set was complete, so one for a profile whose roots already reach
// everything.
Rounds int `json:"rounds"`
// Report is the closure's own exact and observed shape, including the
// pre-prune baseline, the post-prune result, and the external and standard
// boundary imports.
Report closure.ClosureReport `json:"report"`
// RemovedFiles are the profile's prune entries, sorted. Every one of them
// was removed from the materialized tree before the scratch anchor was
// written, so this is the configured prune set rather than a measurement of
// one pass: a reasserting pass over an already pruned tree removes nothing,
// and what each of those passes did is recorded per patch in Patches.
RemovedFiles []string `json:"removedFiles"`
// Golden is the comparison against the profile's checked-in closure record.
Golden GoldenReport `json:"golden"`
}
ClosureReport records the package closure the plan settled on.
type EmbedReport ¶
type EmbedReport struct {
// Path is the destination module relative path of the file holding it.
Path string `json:"path"`
// Line is the one based line the directive sits on.
Line int `json:"line"`
// Patterns are the patterns the directive names, in written order.
Patterns []string `json:"patterns"`
// Matches are the destination paths they resolve to, sorted.
Matches []string `json:"matches"`
}
EmbedReport is one verified go:embed directive.
type EngineReport ¶
type EngineReport struct {
// Version is the engine version.
Version string `json:"version"`
// Toolchain is the Go toolchain the profile pins for deterministic
// formatting.
Toolchain string `json:"toolchain"`
// ProfileHash is the digest of the output affecting subset of the profile,
// which is what a later phase compares to decide whether a control plane
// change started a new epoch. Operational settings are deliberately absent
// from it, so changing a cache location or a limit does not move it.
ProfileHash string `json:"profileHash"`
}
EngineReport identifies what produced the plan.
type FailureReport ¶
type FailureReport struct {
// Stage names the phase that refused, matching PolicyError.Stage.
Stage string `json:"stage"`
// Message is the rendered failure. It carries no absolute path, because the
// stages that can name one render the repository relative path instead.
Message string `json:"message"`
// Patch is the structured conflict record, nil unless a patch failed.
Patch *PatchFailure `json:"patch"`
}
FailureReport records one refused plan.
It is written for a policy failure and for the two runtime failures that still leave a reviewable tree, a cache that moved under the run and an output or report write that did not complete. A runtime failure that stopped the run before it measured anything produces no report at all, because there would be nothing in it but the message already on stderr.
type GoldenReport ¶
type GoldenReport struct {
// Path is the golden's profile relative path, empty when the profile pins
// none. It is profile relative rather than absolute because this report is
// compared between runs over different layouts.
Path string `json:"path"`
// Status is absent, match, or diff. It is empty when the profile pins no
// golden, which is the one case that is not a finding at all.
Status string `json:"status"`
// Differences name the exact fields of the golden's exact shape that
// disagree, sorted. It is empty unless Status is diff.
Differences []string `json:"differences"`
}
GoldenReport records the comparison against the closure golden a profile pins.
A golden is how a maintainer states the closure they reviewed. Without the comparison the limits are the only gate, and limits notice a closure that grew past a number rather than one that changed shape underneath them: a package swapped for another of the same size passes every limit and is exactly the change a reviewer has to see.
type Options ¶
type Options struct {
// Config is the decoded, validated profile.
Config *config.Config
// ProfileDir is the repository directory holding the profile, the patch
// files its patch entries name, and the closure golden it pins.
ProfileDir string
// CacheRoot holds the reusable bare source cache.
CacheRoot string
// WorkRoot holds the scratch work trees this run creates and removes.
WorkRoot string
// OutputRoot is where -materialize writes the relocated tree. It must not
// exist; relocation never merges into or overwrites a tree.
OutputRoot string
// Ref selects the upstream ref to plan.
Ref Ref
// PatchBranch is the tracked branch a patch's branch selector is matched
// against. It is required only when the profile carries patches.
PatchBranch string
// SourceRemote overrides the profile's source repository, which is how a
// test or an air-gapped operator points the run at a local mirror.
SourceRemote string
// Fetch updates the cache before the ref is resolved.
Fetch bool
// Offline refuses every network operation, including the clone an absent
// cache would otherwise trigger and the lazy blob fetch a checkout would
// perform without anything having asked it to.
//
// The refusal is carried by the materialization rather than by the runner
// the caller passes, so a plan cannot be offline in name only: a caller that
// builds Git the ordinary way still gets a run that fails closed on a blob
// the cache does not hold.
Offline bool
// Materialize writes the relocated tree to OutputRoot. Without it the plan
// computes the tree and hashes it without touching a disk.
Materialize bool
// KeepWorktree leaves the materialized source tree in place for inspection.
KeepWorktree bool
// Strict turns every advisory notice into a policy failure.
Strict bool
// Git is the runner the plan drives. It must be anonymous: a plan talks to
// the public source host and to nothing else.
Git *gitcli.Runner
// LookupEnv reads the process environment. A nil value uses os.LookupEnv.
// It is injectable so the credential check is testable without mutating the
// environment of a running test binary.
LookupEnv func(string) (string, bool)
}
Options configures one plan.
Every directory is absolute because a plan must name the same directories no matter where the process was started from, and because the run adopts none of them: the cache and work roots are created if absent and owned by the run thereafter, and the output tree must not exist at all.
func (Options) CheckPaths ¶
CheckPaths refuses an output tree that would sit where the run's own state lives.
It is exported and separate from the rest of validation because it reads nothing and creates nothing: every directory involved is one the caller named or defaulted, so a command line can decide it before it opens a profile, and an operator who typed the wrong flag gets the same answer whether or not a profile, a cache, or a network happens to be there. Plan checks it again, so a caller that skipped it cannot proceed on a layout that cannot work.
Two different relationships are refused, for two different reasons. An output tree inside the materialized source root would be relocated content sitting in the middle of the source the next pass measures, and the closure is handed that tree with permission to remove files from it. An output tree that is, or contains, the cache, the work root, or the profile directory is worse: the destination must not exist when the run starts and is written atomically from scratch, so a directory the run reads from could not survive being the thing the run creates.
Everything else may nest. The documented defaults put the work root below the cache and the output tree below the work root, so that one directory holds all of a run's scratch and an operator can remove all of it in one step.
type OutputReport ¶
type OutputReport struct {
// Module is the destination module path.
Module string `json:"module"`
// Files is how many files the tree holds.
Files int `json:"files"`
// Packages is how many Go packages it holds.
//
// It counts the closure's packages rather than the tree's directories. A
// package that carries embedded data or a matched asset in a subdirectory
// relocates that file into a directory of its own, and counting directories
// would report a module with more packages than any build of it has.
Packages int `json:"packages"`
// ProvenanceFiles are the generated per-package records, sorted.
ProvenanceFiles []string `json:"provenanceFiles"`
// ManifestHash digests the complete tree: every destination path, its mode,
// and its content. Two plans that agree on it produced the same module.
ManifestHash string `json:"manifestHash"`
// Materialized reports that the tree was written to a disk. A plan computes
// the same tree either way, so the hash above does not depend on it.
Materialized bool `json:"materialized"`
}
OutputReport records the tree the plan produced.
type PatchFailure ¶
type PatchFailure struct {
// SourceRef and SourceSHA identify the abandoned ref transaction.
SourceRef string `json:"sourceRef"`
SourceSHA string `json:"sourceSHA"`
// PatchID, PatchIndex, and PatchCount locate the failure in the series.
// PatchIndex is zero based, matching the error it is taken from.
PatchID string `json:"patchID"`
PatchIndex int `json:"patchIndex"`
PatchCount int `json:"patchCount"`
// Stage names the step that failed: apply, prune, or cancel.
Stage string `json:"stage"`
// ConflictedPaths lists the unmerged repository relative paths, sorted.
ConflictedPaths []string `json:"conflictedPaths"`
// Status is the porcelain status captured before the rollback, rendered one
// entry per line in the order Git reported them.
Status []string `json:"status"`
// Diff is the work tree diff captured before the rollback. For a three way
// apply it holds the conflict markers, which is what a maintainer edits the
// patch against.
Diff string `json:"diff"`
}
PatchFailure is the reviewable half of a patch conflict.
It repeats what *patchset.ConflictError carries because the error itself reaches only stderr, and reproducing a conflict from a one line message means rerunning the whole pipeline. Nothing here is a path on this machine: the paths are repository relative and the diff is upstream content.
type PatchReassert ¶
type PatchReassert struct {
// PatchID is the patch that had just applied.
PatchID string `json:"patchID"`
// Files are the repository relative paths this reassertion removed again,
// sorted. It is empty for a patch that reintroduced nothing, which is the
// normal case.
Files []string `json:"files"`
}
PatchReassert is one patch's reassertion of the profile's pruning.
type PatchesReport ¶
type PatchesReport struct {
// Branch is the tracked branch the branch selectors were matched against,
// empty for a profile with no patches.
Branch string `json:"branch"`
// Available is how many patches the profile carries.
Available int `json:"available"`
// Selected are the identifiers the selectors chose, in application order.
Selected []string `json:"selected"`
// Applied are the identifiers that applied cleanly, in application order.
Applied []string `json:"applied"`
// Reasserted is how many times pruning was reasserted, which is once per
// applied patch.
Reasserted int `json:"reasserted"`
// Reassert names what each reassertion removed, in application order.
//
// The count alone cannot answer the question the reassertion exists to
// answer. Pruning is reasserted after every patch precisely so a patch that
// reintroduced a pruned file is caught, and when one does the maintainer's
// first question is which patch, which the count does not carry.
Reassert []PatchReassert `json:"reassert"`
}
PatchesReport records patch selection and application.
type Paths ¶
type Paths struct {
// Cache is the bare source cache directory.
Cache string
// Work is the scratch root the work tree was created below.
Work string
// Worktree is the materialized source tree, empty once it was removed.
Worktree string
// Output is the relocated tree destination, written only with -materialize.
Output string
}
Paths are the absolute directories one plan used.
type PolicyError ¶
type PolicyError struct {
// Stage names the phase that refused, such as anchor, closure, or patch.
Stage string
// Err is the underlying failure.
Err error
}
PolicyError reports a plan that ran correctly and found the profile, its patches, or the content they select unacceptable.
It exists so the command line can separate the three answers an operator acts on differently. A policy failure means the engine worked and the answer is no: a prune target upstream renamed, a denied import that came back, a patch that no longer applies, a closure past its limit, a file the pinned gofmt would reformat. A runtime failure means the engine could not answer at all, and a usage failure means the command line was wrong. Only the first is a finding about the profile, so only the first exits with the check code that CI reads as "review this" rather than "the tool is broken".
func (*PolicyError) Error ¶
func (e *PolicyError) Error() string
Error renders the stage scoped failure.
func (*PolicyError) Unwrap ¶
func (e *PolicyError) Unwrap() error
Unwrap exposes the cause so errors.Is and errors.As reach the typed errors the lower packages return, such as *closure.LimitError and *patchset.ConflictError.
type Ref ¶
type Ref struct {
// Kind names the ref namespace.
Kind RefKind
// Name is the short ref name, such as v1.36.1 or master.
Name string
}
Ref is the single upstream ref a plan covers.
type RefKind ¶
type RefKind string
RefKind names the ref namespace a plan selects from.
const ( // RefTag selects an upstream release tag, which is the normal case. RefTag RefKind = "tag" // RefBranch selects a tracked upstream branch, which is how an operator // inspects what the next release would contain. RefBranch RefKind = "branch" // RefCommit selects an exact object already present in the source cache. It // is engine-only: CLI users select reviewed tags or tracked branches, while // reconciliation uses this kind for commits inside a release-bounded DAG. RefCommit RefKind = "commit" )
The selectable ref kinds. A plan covers exactly one ref, because a plan is a statement about one source commit.
type RelocatedFile ¶
type RelocatedFile struct {
// Source is the upstream repository relative path, empty for a file this
// engine generated, such as a provenance record.
Source string `json:"source"`
// Destination is the module relative path.
Destination string `json:"destination"`
// Mode is the Git octal file mode.
Mode string `json:"mode"`
// Generated records that the upstream file carried a Code generated marker.
Generated bool `json:"generated"`
// SHA256 is the digest of the final bytes, after rewriting.
SHA256 string `json:"sha256"`
}
RelocatedFile is one file of the generated module.
type RelocatedPackage ¶
type RelocatedPackage struct {
// SourcePackage is the upstream package directory.
SourcePackage string `json:"sourcePackage"`
// Package is the destination package directory.
Package string `json:"package"`
// Files are the package's files, sorted by destination path.
Files []RelocatedFile `json:"files"`
}
RelocatedPackage is one package of the generated module.
type RelocationReport ¶
type RelocationReport struct {
// InternalPrefix is the module relative directory every upstream path is
// preserved below.
InternalPrefix string `json:"internalPrefix"`
// Packages are the relocated packages, sorted by destination directory.
Packages []RelocatedPackage `json:"packages"`
}
RelocationReport records the upstream to destination mapping.
type Report ¶
type Report struct {
Schema int `json:"schema"`
Engine EngineReport `json:"engine"`
Source SourceReport `json:"source"`
Worktree WorktreeReport `json:"worktree"`
Patches PatchesReport `json:"patches"`
Closure ClosureReport `json:"closure"`
Relocation RelocationReport `json:"relocation"`
Rewrite RewriteReport `json:"rewrite"`
Output OutputReport `json:"output"`
// Failure records why the plan refused, nil when it did not. A report is
// produced for a policy failure precisely so the refusal is reviewable
// without rerunning the pipeline, and a refusal with no machine readable
// record would leave CI parsing a stderr line.
Failure *FailureReport `json:"failure"`
// Notices are advisory findings, sorted. They never stop a plan on their
// own; -strict is what turns them into a policy failure.
Notices []string `json:"notices"`
}
Report is the deterministic record of one plan.
It carries no absolute path, no environment value, and no secret. That is not tidiness: the report is compared byte for byte between two runs over different directory layouts, it is attached to CI artifacts, and it is the evidence a reviewer reads before approving an outward action. A path from the machine that produced it would break the first use and leak into the second.
Every list is sorted and non-nil, so the encoding depends on the plan alone and never on map iteration order or on whether a list happened to be empty.
type Result ¶
type Result struct {
// Report is the deterministic record of what the plan found.
Report Report
// Files is the final relocated file set, including the rewritten bytes and
// the per-package provenance records. It is what -materialize writes, and it
// is empty for a plan that refused before relocating.
Files relocate.FileSet
// Provenance is the structured form of the per-package records, one entry
// per relocated package, ordered by destination directory.
//
// It carries exactly what the committed SOAPBOX_PROVENANCE.txt beside each
// package states, in the shape the root NOTICE generator consumes. Rendering
// those records and then parsing the text back is the one way this evidence
// could disagree with itself, so the structure the text was rendered from is
// what leaves the plan.
//
// A plan that refused after relocating still carries the records it built,
// because a refusal is when the evidence is most worth reading. It is empty
// for a plan that refused before it got that far.
Provenance []*rewrite.PackageProvenance
// Paths are the absolute directories the run used. They are deliberately
// outside Report, which carries no absolute path.
Paths Paths
}
Result is one completed plan.
A plan that refused still produces one whenever it measured enough to be worth reading. Report.Failure is what tells the two apart, and it is the reason a refusal is reviewable from an artifact rather than from a stderr line.
func Plan ¶
Plan computes one extraction plan.
Nothing outside the cache and work roots is written unless Materialize is set, and no ref is ever created, moved, or deleted. The scratch anchor commit the patch phase needs is recorded in the work tree's detached HEAD, which is why the plan can commit at all without the cache observing it; the run proves that by comparing the cache's refs before and after.
A refusal returns both a result and an error whenever the run measured enough for the report to be worth reading, which is every policy failure and every failure that left the cache or the output in a state an operator has to see. A failure that stopped the run before it measured anything returns the error alone, because a report of nothing is not evidence.
type RewriteReport ¶
type RewriteReport struct {
// Files are the files a transformation changed, sorted by path. A file
// nothing changed is absent, because listing every unchanged file would
// bury the ones a reviewer has to read.
Files []RewrittenFile `json:"files"`
// DirectiveRemovals are the removed generator and toolchain directive
// lines, rendered and sorted. They are listed separately from the per-file
// changes because a removed marker is the transformation most likely to
// change behaviour, so it must be readable without scanning every file.
DirectiveRemovals []string `json:"directiveRemovals"`
// Embeds are the verified go:embed directives, sorted.
Embeds []EmbedReport `json:"embeds"`
// GoFiles is how many Go files the plan transformed and reparsed.
GoFiles int `json:"goFiles"`
// Unparsed are destination paths the pinned parser could not read, sorted.
// It must always be empty.
Unparsed []string `json:"unparsed"`
// Unformatted are destination paths the pinned gofmt would reformat,
// sorted. Relocating an import can move it within its group, so this is
// where that shows up.
Unformatted []string `json:"unformatted"`
}
RewriteReport records what the syntax aware transformations did.
type RewrittenFile ¶
type RewrittenFile struct {
// Path is the destination module relative path.
Path string `json:"path"`
// NoticeInserted reports that the file received the modification notice.
NoticeInserted bool `json:"noticeInserted"`
// Changes are every recorded transformation, rendered and sorted.
Changes []string `json:"changes"`
}
RewrittenFile is one transformed file.
type SourceReport ¶
type SourceReport struct {
// RefKind and RefName are the selected ref.
RefKind string `json:"refKind"`
RefName string `json:"refName"`
// Ref is the fully qualified ref name.
Ref string `json:"ref"`
// Object is what the ref points at, which is the tag object itself for an
// annotated tag.
Object string `json:"object"`
// Commit is the commit the ref resolves to, with annotated tags peeled.
Commit string `json:"commit"`
// Annotated reports a tag object rather than a direct commit reference.
Annotated bool `json:"annotated"`
// AnchorCommit is the recorded transformed anchor, empty when the profile
// has not resolved one yet.
AnchorCommit string `json:"anchorCommit"`
// AnchorVerified reports that the selected commit descends from the anchor.
// It is false when no anchor is configured, because nothing was verified.
AnchorVerified bool `json:"anchorVerified"`
// Fetched reports that this run updated the cache from the remote.
Fetched bool `json:"fetched"`
// CacheCreated reports that this run cloned the cache rather than reusing
// one.
CacheCreated bool `json:"cacheCreated"`
// Offline reports that the run refused every network operation.
Offline bool `json:"offline"`
// RemoteOverridden reports that the run read history from a remote other
// than the profile's, which is how a test or an air-gapped operator points
// at a local mirror.
//
// Only the fact is recorded. The override's value is frequently a path on
// the machine that ran the plan, and this report is compared byte for byte
// between two runs over different layouts, so naming it would break the
// comparison the determinism check depends on. The fact still has to be
// here: a report produced against a mirror describes whatever that mirror
// held, and a reviewer must be able to see that before trusting it.
RemoteOverridden bool `json:"remoteOverridden"`
}
SourceReport records the upstream commit the plan covers and how it was obtained.
The remote is absent on purpose. It is already covered by the profile hash, and an override may name a local mirror, which would put an absolute path into a report that two runs have to agree on byte for byte.
type WorktreeReport ¶
type WorktreeReport struct {
// SparsePatterns is the final pattern set, in the order git applies it.
SparsePatterns []string `json:"sparsePatterns"`
// WidenRounds is how many times the closure discovered a package the
// pattern set did not materialize.
WidenRounds int `json:"widenRounds"`
// WidenedPackages are the repository relative directories widening added,
// sorted. They materialize files; they never seed the closure, so they
// cannot change which packages the closure contains.
WidenedPackages []string `json:"widenedPackages"`
// ScratchAnchor is the local commit the patch phase applies against.
ScratchAnchor AnchorReport `json:"scratchAnchor"`
// CacheRefsMoved reports whether any ref in the shared cache changed while
// the plan ran.
//
// A plan makes exactly one commit and makes it on a detached HEAD in a
// linked work tree, so the value is false for every run that behaved. It is
// carried in the report rather than merely asserted because the cache is
// reused across runs and is what later phases publish from: a run that moved
// a ref has to leave a record a reviewer can find, which means the report
// has to be produced for that failure rather than suppressed by it.
CacheRefsMoved bool `json:"cacheRefsMoved"`
}
WorktreeReport records how the source tree was materialized.