Documentation
¶
Overview ¶
Package build holds the orchestration helpers that turn a parsed recipe into a runnable build: the dependency closure (recipe deps reduced for the target), the ambient base toolchain brewkit provides, a sanitized run environment, and the autotools maintainer-mode defeat. These were distilled from proving bk on real packages (zlib native, zlib→windows cross, wget with openssl).
Index ¶
- Constants
- Variables
- func BaseToolchain() []string
- func BootstrapToolDeps(buildDeps map[string]any, tgt target.Target) []string
- func BuildDeps(recipe *pantry.Recipe) map[string]any
- func CheckToolchainPerl(set *logical.Set, pantryDir string) []error
- func DepSpecs(deps map[string]any, tgt target.Target) []string
- func DepTokens(runtime, buildDeps map[string]any, tgt target.Target, pkgxDir string, ...) ([]moustache.Token, error)
- func EvalDeps(project string, runtime, buildDeps map[string]any, tgt target.Target) []stringdeprecated
- func EvalLinkDeps(runtime map[string]any, tgt target.Target) []string
- func EvalToolDeps(project string, runtime, buildDeps map[string]any, tgt target.Target) []string
- func HostPrefixForSelfDep(project string, provides any, look func(string) (string, error)) []moustache.Token
- func ReduceDeps(deps map[string]any, tgt target.Target) map[string]string
- func SanitizedEnv(home, pkgxDir string) []string
- func SpecProject(spec string) string
- func TouchAutotools(srcDir string) error
- func WithoutSelfDep(project string, buildDeps map[string]any) map[string]any
- func WithoutUnresolvable(buildDeps map[string]any, tgt target.Target, ...) (map[string]any, []string)
- type Result
- type Runner
- type SourceRef
Constants ¶
const ToolchainGawk = "~5.3"
ToolchainGawk is the gawk constraint the base toolchain pins. Stated once, for the same reason as ToolchainPerl: the spelling is the whole content, and a copy of it in a test is a copy that cannot disagree.
const ToolchainPerl = "~5.44"
ToolchainPerl is the perl constraint the base toolchain pins, and the one every XS-bearing tool in it must be built against. Stated once because the only way it can be wrong is by disagreeing with itself.
Variables ¶
var ErrChecksumMismatch = errors.New("build: source checksum mismatch")
ErrChecksumMismatch is returned when a downloaded archive does not match the digest its recipe's `sha:` URL declares.
Functions ¶
func BaseToolchain ¶
func BaseToolchain() []string
BaseToolchain is the ambient build toolset brewkit provides so recipes need only declare their SPECIFIC deps. Without it, autotools packages fail on aclocal/makeinfo not-found (proven with wget).
func BootstrapToolDeps ¶
BootstrapToolDeps is EvalToolDeps WITHOUT the base toolchain: only what the recipe itself declares as a build dependency.
It exists because the base toolchain is a CYCLE with no entry point, and that is invisible until an architecture has no bottles at all. Every build gets the fifteen projects BaseToolchain names injected into its pkgx environment, as BOTTLES -- so on linux/s390x, where neither this registry nor dist.pkgx.dev publishes anything, gnu.org/m4 fails on
pkgx: no version of freedesktop.org/pkg-config satisfies AND is
published for linux/s390x
although m4's recipe declares no build dependencies at all. Building pkg-config asks for the other fourteen in turn. Nothing can be first.
--libc=pkgx does not help and neither does its absence: the build container's distribution supplies a COMPILER fallback, never a declared dependency, so both modes go through pkgx and both demand a bottle.
So one generation has to be made with the host's own tools, into a registry that generation never leaves. This is that escape, and it is deliberately the smallest one that works: the recipe's own build dependencies still resolve as bottles, and the LINK closure is untouched, because what a bottle links against is what it ships -- only the tools that RUN the build come from outside.
func BuildDeps ¶
BuildDeps is a recipe's build-time dependency map, which lives under the free-form `build:` node rather than in a typed field. Exported because a recipe's build deps are as capable of naming an unpublishable version as its runtime ones — `bk depgaps` counts both.
func CheckToolchainPerl ¶
CheckToolchainPerl reports every XS-bearing toolchain recipe whose own `perl.org` constraint disagrees with ToolchainPerl.
It reads the recipes rather than a copy of them. That distinction is the whole point: the unit test that claimed to check this asserted the pin equalled the string "~5.42", which was texinfo's constraint on the day it was written. texinfo moved to ~5.44, the literal did not, and the test kept passing while every man-page-generating build broke.
A recipe that is absent is not a disagreement — a pantry need not be complete, and refusing to build because a file is missing would be a worse failure than the one this prevents.
func DepSpecs ¶
DepSpecs reduces a recipe dependency map into pkgx pkgspecs ("project@constraint") for the target, sorted for determinism.
func DepTokens ¶
func DepTokens(runtime, buildDeps map[string]any, tgt target.Target, pkgxDir string, resolve func(project, constraint string) (string, error)) ([]moustache.Token, error)
DepTokens resolves each recipe dependency (runtime + build) to a version and install prefix ($PKGX_DIR/<project>/v<version>) and returns the `{{deps.<project>.prefix}}` / `{{deps.<project>.version.*}}` moustache tokens. resolve maps a project + constraint to a concrete version.
func EvalDeps
deprecated
Deprecated: the build now composes two closures — EvalLinkDeps and EvalToolDeps. This is kept as the CONTROL for that split: a test asserts that it still puts a link constraint and a build tool in ONE list, which is what forced qt.io's unicode.org ^71 and nodejs.org's ^73 through a single intersection. Delete it once that history stops being worth pinning.
func EvalLinkDeps ¶
EvalDeps is the full `pkgx +…` set for a build: the recipe's own runtime + build deps, then the base toolchain filling whatever they did not ask for, de-duplicated by project and sorted.
The ORDER is the point. The base toolchain is a floor — "these tools must be present" — not a pin, so a recipe that constrains one of those projects must win. It used to lose: the base was added first and the dedup dropped the recipe's spec, so gnu.org/texinfo, which declares `perl.org: ~5.42` because its XS modules are compiled against that perl, silently got the base's unconstrained perl.org — 5.44 — and every recipe that runs makeinfo died with
Perl API version v5.42.0 of …/TreeElementXS.c does not match v5.44.0
The same silent override applied to every base project (gawk, make, bison…), so a recipe could never correct one. EvalLinkDeps is the set a build LINKS against: the recipe's own runtime dependencies, and nothing else.
These are the constraints that have to agree with what a CONSUMER will resolve, because they are the ones that end up as @rpath references in the artefact. Nothing a build merely runs belongs here.
func EvalToolDeps ¶
EvalToolDeps is the set a build RUNS: the recipe's build dependencies plus the base toolchain. Nothing here is linked into the artefact, so its version constraints are nobody else's business.
Keeping these apart from the link set is what lets qt.io build at all. qt declares unicode.org ^71 as a LINK dependency and nodejs.org as a BUILD one, and nodejs needs unicode.org ^73 to start. ICU bumps its major — and its soname — every release, so the two can never intersect:
pkgx: no version of unicode.org satisfies "^71" AND "^73" (available: 3);
asked for by ^71 (requested), ^73 (nodejs.org)
They did not need to. nodejs runs as a build step and exits; its ICU is never in the same process as qt's. One closure asked a question that has no answer.
The project itself is filtered out of the base toolchain for the reason EvalDeps gives below: its own published bottle must not shadow the thing being built.
func HostPrefixForSelfDep ¶
func HostPrefixForSelfDep(project string, provides any, look func(string) (string, error)) []moustache.Token
HostPrefixForSelfDep gives {{deps.<project>.prefix}} a value when bootstrap dropped a recipe's dependency on ITSELF and the machine demonstrably provides what the recipe says it provides.
gnu.org/sed build-depends on gnu.org/sed for one line:
export PATH="{{deps.gnu.org/sed.prefix}}/bin:$PATH"
— putting a known-good sed in front of whatever the system has. On an architecture with no sed bottle the system's IS the only sed, and the unresolved token stopped the build with the guard Runner.Build added for exactly this shape.
The prefix is not invented. The recipe says it provides `bin/sed`, the host says `sed` is at /usr/bin/sed, and the prefix that makes both true is /usr. Where the two do not agree — the binary is somewhere that is not <prefix>/bin, or the host does not have it at all — no token is produced and the build still refuses. A guessed path would send a compiler somewhere nobody chose.
The SELF dependency only, and that restriction is the whole of its justification. `provides:` belongs to the recipe being built; it says what THIS project puts on the machine and nothing about any other. A first version took a list of every dropped dependency and read this recipe's provides for each of them, which would have answered "cmake.org's prefix is /usr" because the recipe in hand ships a bin/ of its own. A dropped dependency that is not the self-edge still leaves its token unresolved, and the build still refuses — loudly, which is the correct outcome for a question this cannot answer.
func ReduceDeps ¶
reduceDepMap flattens a recipe dependency map (project → constraint, with optional platform-keyed sub-maps) to project → constraint for the target, dropping non-matching platform keys and merging matching ones. ReduceDeps flattens a recipe dependency map to project → constraint for the target, as the RECIPE spells the constraint.
Exported because a caller that needs the constraint must not derive it from DepSpecs. That renders a pkgx WIRE form — `gnu.org/m4@1`, `cmake.org^3` — and trimming the project off it yields "@1", a string nothing can resolve. One caller already did that and dropped a bottle it had just built.
So there is one reduction, and everything that asks a question about a dependency asks it the same way.
func SanitizedEnv ¶
SanitizedEnv is the clean environment the build script runs within (brewkit's clearEnv): a fixed PATH, PKGX_DIR and HOME, plus a small allow-list passed through from the caller's environment. It prevents a stray toolchain (eg. a system Homebrew) from leaking into the build (proven: wget linked the wrong openssl under an unsanitized env).
func SpecProject ¶
SpecProject extracts the bare project name from a rendered pkgspec, stopping at the first version delimiter (@ or a range operator) so dedup keys match regardless of the constraint form. depSpec renders a caret/tilde/range constraint with NO separator (`invisible-island.net/ncurses^6`), so a consumer that splits on "@" alone keeps the operator inside the project name — which is how a constrained dependency silently drops out of a closure.
func TouchAutotools ¶
TouchAutotools defeats autotools maintainer-mode: an extracted tarball's file timestamps make `make` think configure.ac changed and rerun aclocal/automake (which fail without the exact tool versions). Touching the generated files newer than their sources, in ascending tiers, marks the build system fresh.
func WithoutSelfDep ¶
WithoutSelfDep removes a recipe's build dependency on ITSELF.
Four recipes in the bootstrap set declare one: gnu.org/gcc, gnu.org/sed, gnu.org/grep and rust-lang.org/cargo. It is the right declaration almost always -- gcc's own comment says ">=14 lets the newest gcc WE have build the older one, so the seed comes from our own chain instead of an upstream binary" -- and it is exactly wrong when the chain does not exist yet. On an architecture with no bottles, a project that needs itself can never be first, and nothing downstream of it can either: gnu.org/glibc build-depends on gcc 14, so the whole toolchain sits behind that one edge.
Only in bootstrap mode, and only the self-edge. Every other build dependency stays a bottle, because every other one CAN be built in order.
func WithoutUnresolvable ¶
func WithoutUnresolvable(buildDeps map[string]any, tgt target.Target, resolve func(project, constraint string) (string, error), log func(string)) (map[string]any, []string)
WithoutUnresolvable drops the build dependencies this registry cannot provide for the target, asking `resolve` about each one.
Only in bootstrap mode, and the reason it is safe is written a few functions above: EvalToolDeps is "the set a build RUNS", and "Nothing here is linked into the artefact." A build dependency is a TOOL. The link closure -- EvalLinkDeps, what the bottle actually carries -- is untouched.
It is what the remaining s390x seed failures all were, each differently:
- curl.se/ca-certs declares curl.se so its script can run `curl -k https://curl.se/ca/… -o cert.pem`. curl.se in turn needs ca-certs, so neither can be first. The host has a curl.
- perl.org declares `llvm.org: <19` on linux: a compiler, named by bottle. The host has a compiler; that is the whole premise of --bootstrap.
`resolve` is asked whether the tool can be INSTALLED, not merely whether a version of it exists. The two differ, and the difference stopped the seed: github.com/besser82/libxcrypt build-depends on perl.org, perl.org WAS in the seed registry, and the build still died —
bk: the tool environment failed: pkgx +freedesktop.org/pkg-config~0.29 +perl.org pkgx: GET …/github.com/besser82/libxcrypt/linux/s390x/versions.txt: Not Found
because perl's closure needs github.com/besser82/libxcrypt — the project being built. OUR OVERLAY declares that edge for linux (upstream's perl.org does not), so it is an ordinary recipe dependency and no soname map is involved; an earlier version of this comment said otherwise. A tool you cannot install is not a tool you have, so the question has to be about the closure.
A drop is reported, not silent: a seed bottle built without a tool somebody declared is a fact that outlives the run.
If the recipe needed that dependency's PATH rather than its binaries, the build refuses instead — see the unresolved-{{deps.…}} check in Runner.Build. That guard is why this can be a blunt rule without being a reckless one.
Types ¶
type Result ¶
type Result struct {
Version string
Install string
ScriptPath string
BottlePath string
// Source is empty for a recipe with no distributable (a few build entirely
// from their dependencies).
Source SourceRef
}
Result reports what a build produced.
type Runner ¶
type Runner struct {
PickVersion func(project, constraint string) (string, error)
// ResolveVersion resolves the MAIN project version from the recipe's own
// `versions:` spec, so it matches the distributable URL's {{version.raw}}
// rather than dist's already-built (possibly normalised) bottle list. It
// also returns the raw upstream git tag the version was resolved from, for
// the {{version.tag}} moustache used in GitHub release download URLs.
ResolveVersion func(spec any, constraint string) (version, tag string, err error)
// Fetch and FetchGit report WHAT they got, not merely that they got it: an
// archive's SHA-256, a clone's commit. See SourceRef.
Fetch func(url, dest string, strip int) (sha256 string, err error)
FetchGit func(repo, ref, dest string) (commit string, err error)
// FetchSHA reads the checksum a recipe's `sha:` URL declares for an archive.
// Nil disables verification — what a test that does not care about it wants;
// the real runner always sets it.
FetchSHA func(shaURL, archiveName string) (string, error)
Touch func(dir string) error
Run func(scriptPath string, env []string) error
FixUp func(fixup.Options) error
WriteBottle func(installDir, project, version, osn, arch, outDir string) (string, error)
// ResolveDep, when set, resolves each dependency to a version so the build
// gets {{deps.<project>.prefix}}/{{deps.<project>.version}} tokens.
ResolveDep func(project, constraint string) (string, error)
// ToolInstallable answers whether a BUILD dependency can actually be
// installed here — its whole closure, not just a version of itself. Only
// bootstrap asks, and only to decide what to leave to the host; nil falls
// back to ResolveDep, which is the weaker question.
//
// The two differ. perl.org was in the s390x seed registry, so ResolveDep
// said yes, and the build died installing it: our overlay declares
// github.com/besser82/libxcrypt as a linux dependency of perl, and that is
// the project being built.
ToolInstallable func(project, constraint string) (string, error)
Concurrency int
PkgxBin string
BashPath string
// RecipeDir is the pantry project directory holding the recipe (package.yml)
// and its sibling files. It is copied into the build tree as `props/` so
// recipes can reference those files as `props/foo` or via the {{props}}
// moustache (pkgx's convention). Empty disables the behaviour.
RecipeDir string
// LibcMode selects the C library the build links against. "pkgx" targets the
// gnu.org/glibc bottle (sovereign FROM-scratch, linux only, C recipes);
// anything else (default "") keeps the build container's system glibc.
LibcMode string
// Glibc pins the exact pkgx glibc version to link against in LibcMode "pkgx"
// (a chosen HPC floor, e.g. "2.27.0"); empty = newest.
Glibc string
// Bootstrap runs the build WITHOUT the base toolchain in its environment,
// taking those tools from the host instead. It is for the first generation
// on an architecture no registry has: see BootstrapToolDeps for why nothing
// can otherwise be built first. It must not be set for an ordinary build --
// a bottle made this way was driven by tools nobody pinned.
Bootstrap bool
// LookPath finds a host binary, for the one case bootstrap has to ask the
// machine what it already has. nil means exec.LookPath. It is a field so a
// test can answer without depending on what the machine running it happens
// to have installed — /usr/bin/sed exists on every developer's laptop and
// on none of the scratch images.
LookPath func(string) (string, error)
}
Runner drives a full build. Its side-effecting steps are fields so the whole orchestration is testable with stubs; NewRunner wires the real bk packages.
type SourceRef ¶
type SourceRef struct {
URI string // the candidate that actually answered
SHA256 string // lowercase hex, for an archive
Commit string // for a git source
}
SourceRef identifies the bytes a build was actually made from: which of the recipe's candidate URLs answered, and the digest of what came back.
A recipe names a URL. A URL is not a source: the list form falls through to a mirror when the first host is down, a git tag can be re-cut, and more than half of this pantry fetches a tarball GitHub generates on request rather than one it stores. Recording the URL alone attests where we knocked, not what we were handed.
Source Files