buildscript

Documentation

Overview

Package buildscript turns a recipe's build/test node into the shell script brewkit runs, a faithful port of libpkgx's usePantry.getScript: moustache expansion, `if:` guards evaluated against the build TARGET, platform-reduced env, working-directory wrapping and prop/fixture here-docs.

Index

• Constants
• func Generate(node any, opts Options) (string, error)
• func Wrap(o WrapOptions) string
• func WrapTest(o TestWrapOptions) string
• func WriteLibexec(dir string) error
• func WriteLibexecFor(dir string, libcPkgx bool, triple, platform, arch string) error
• type Options
• type TestWrapOptions
• type WrapOptions

Constants

const EnvFailExit = 69

EnvFailExit is the status the test script exits with when `pkgx +…` could not assemble the environment — as opposed to the test itself failing.

The two were one for a while, and the s390x seed's first sweep shows what that costs: of nine reported failures, FIVE never ran a line of their test block. invisible-island.net/ncurses wants github.com/tmux/tmux ^3 as a test dependency and no s390x tmux exists anywhere; gnu.org/readline's closure needs libCNS.so, which pkgx's soname map does not name. Neither is a bottle that does not work, and calling them failures would file bugs against packages for an incomplete registry.

69 is sysexits' EX_UNAVAILABLE, which is what happened. A test that exits 69 on its own would be read as this — and would be saying very nearly the same thing, which is why the collision is tolerable where an arbitrary number would not be.

const SysrootCC = baseSysrootFlags + ` --unwindlib=none`

SysrootCC is the driver flag set for compiling C against a pkgx-supplied glibc. It is the sysroot and nothing else: no recipe CFLAGS, no rpath, no CMAKE_PREFIX_PATH. A test that compiles must be able to produce a binary at all; it must NOT be handed the flags that made the artefact link, or a bottle whose own headers are wrong would still pass.

const SysrootCXX = `${BK_LIBCXX_PREFIX:+-stdlib=libc++ -isystem "${BK_LIBCXX_PREFIX}include/c++/v1"} ` + baseSysrootFlags + ` ${BK_LIBCXX_PREFIX:+--unwindlib=libunwind}`

SysrootCXX is SysrootCC plus libc++.

Functions

func Generate

func Generate(node any, opts Options) (string, error)

Generate renders node (a recipe's build or test value) into a shell script.

func Wrap

func Wrap(o WrapOptions) string

Wrap assembles the complete, runnable build script: a sanitized env, the dependency environment (`eval "$(pkgx +…)"`), target-specific compiler/linker FLAGS, a per-host TMPDIR, then the user script run from SRCROOT.

func WrapTest

func WrapTest(o TestWrapOptions) string

WrapTest renders the runnable script for a recipe's test: block.

NO `set -x`, and the first version of this had it with a reason that was right about diagnosis and wrong about correctness.

The trace goes to stderr, and a test that captures stderr captures it:

out=$("{{prefix}}/bin/iconv" --version 2>&1 | head -1)
→ out='+ /…/bin/iconv --version'

gnu.org/glibc reported a broken iconv on nothing but that. Measured on pantry 2df061b: 31 of the 1897 recipes with a test block capture stderr into a substitution, and every one of them would read its own trace.

BASH_XTRACEFD would have kept both — the trace on a descriptor `2>&1` cannot reach — and mvdan.cc/sh ignores it, measured rather than assumed.

So the trace is dropped. A missing trace makes a failure slower to read; a contaminated capture makes a WORKING package report FAIL, and a wrong answer is worse than a slow one. The generated script stays on disk beside the sandbox, which is where the commands can still be read.

func WriteLibexec

func WriteLibexec(dir string) error

func WriteLibexecFor

func WriteLibexecFor(dir string, libcPkgx bool, triple, platform, arch string) error

WriteLibexecFor is WriteLibexec plus the compiler shims when libcPkgx is set.

Types

type Options

type Options struct {
	Target     target.Target     // the build-for platform/arch (guards key on this)
	PkgVersion string            // used to evaluate semver `if:` guards
	Tokens     []moustache.Token // prefix/version/deps/hw/pkgx/srcroot/props, pre-assembled
}

Options carries everything script generation needs beyond the node itself.

type TestWrapOptions

type TestWrapOptions struct {
	UserScript string // the test node, already rendered by Generate
	// Package is the pkgspec of the package UNDER TEST, and it must name an
	// exact version. `bk test gnu.org/gawk` after building 5.4.1 with 5.3.0
	// still published would otherwise test 5.3.0 and report on the wrong
	// bytes — a green result about a build that never happened.
	Package string
	// Deps are the recipe's test.dependencies, already reduced for the target
	// and rendered as pkgspecs. They are a SEPARATE map from build and runtime
	// dependencies: a test may need a fixture generator or a diff tool that
	// the package itself must not carry.
	Deps []string
	// Compiler asks for one in the test environment.
	//
	// The first version of this file said a test may not have a compiler,
	// because "a consumer installing this package gets none". Measured
	// against pantry 2df061b with `bk tools --scope test --all`, that is
	// wrong: 260 recipes' tests call a compiler (cc 193, c++ 38, gcc 13,
	// g++ 7, clang 6, clang++ 6 — 260 distinct, not the sum of 263, because
	// three call two of them), and only 6 declare llvm.org. A recipe's test
	// compiling a five-line program against the headers it just shipped IS
	// the convention here — zlib.net's is `cc test.c -lz` — and a sandbox
	// that refuses would have reported 260 false failures.
	//
	// Asked per test rather than always, from the rendered script's own
	// command set, so the 1500-odd tests that compile nothing do not install
	// a compiler to not use it.
	Compiler bool
	Home     string        // a fresh HOME, created by the script
	Sandbox  string        // the empty directory the test runs in
	PkgxDir  string        // $PKGX_DIR, so the eval resolves where the build published
	PkgxBin  string        // path to the pkgx binary
	BashPath string        // shebang interpreter (default /bin/bash)
	Host     target.Target // where we run — drives TMPDIR
	// ShimDir holds bk's compiler shims (cc, gcc, c++, g++ and their
	// triple-prefixed spellings), materialised by the caller with
	// WriteLibexecFor. It is prepended to PATH in a scratch rootfs so a test
	// that calls `cc` by name — which 194 of the pantry's do — gets the
	// sovereign driver rather than a bare clang with no libc.
	//
	// Empty is allowed and means "no shims": the preamble then falls back to
	// putting the flags in $CC, which is what it used to do and what reaches
	// almost nothing.
	ShimDir string
}

TestWrapOptions carries what a recipe's test: block needs around it.

The schema calls the test "a post-build sanity test run in a fresh sandbox", and the sandbox is the whole point: it answers "does what we PUBLISHED work", which is a different question from "did the build succeed". So this deliberately does NOT reuse WrapOptions.

What a build script has and a test script must not:

CFLAGS/LDFLAGS/rpath  a test may COMPILE — 260 recipes' do — but against
                      the package as shipped. A flag set that made the
                      artefact link would mask a bottle whose own headers
                      or rpath are wrong, which is the thing worth
                      catching. The compiler itself is supplied; the
                      build's flags are not.
CMAKE_PREFIX_PATH     the whole store (go-pkgx/bk#164). A test that found
                      a header there would be finding it by accident.
$SRCROOT / the build  the sources are gone by then, and a test that reads
                      tree                  them is testing the tree and
                      not the package.

What it keeps is close to what a user has: the package itself, whatever the recipe declares under test.dependencies, the recipe's own fixture files, a fresh HOME and an empty directory to work in — plus a compiler when the test's own script calls one.

type WrapOptions

type WrapOptions struct {
	UserScript string   // the pantry script, already rendered by Generate
	Deps       []string // LINK dependencies: what the artefact is built against
	// ToolDeps are what the build RUNS — build dependencies and the base
	// toolchain — resolved in their OWN `pkgx +…` closure.
	//
	// Two closures, not one, because a build tool's runtime requirement and the
	// product's link requirement answer different questions and do not have to
	// agree. qt.io could not build at all while they shared one: it links
	// unicode.org ^71 and its build runs nodejs.org, which needs ^73 to start,
	// and ICU changes its major — and its soname — every release.
	//
	// The tool eval is emitted FIRST so that the link eval's directories land
	// ahead of it in every search path: pkgx writes each variable as
	// VAR="new${VAR:+:$VAR}", so the last eval wins. Nothing is dropped, so a
	// recipe that names a header-bearing dependency under build.dependencies
	// still finds it — one place later than before.
	ToolDeps    []string
	Target      target.Target // what we build FOR — drives FLAGS
	Host        target.Target // where we run — drives TMPDIR
	Home        string        // a fresh HOME for the build
	SrcRoot     string        // the build directory (also SRCROOT / cd target)
	PkgxDir     string        // $PKGX_DIR — the rpath root and CMAKE_PREFIX_PATH
	Install     string        // the final install prefix ($PKGX_DIR/project/vX.Y.Z)
	Project     string        // the project being built, so we never preset a tool IT provides
	BrewkitPath string        // dir prepended to PATH for the build shims (optional)
	PkgxBin     string        // path to the pkgx binary (for the deps eval + $PKGX)
	BashPath    string        // shebang interpreter (default /bin/bash)
	HasCompiler bool          // a compiler (llvm.org / gnu.org/gcc) is already a dep
	// Bootstrap takes the compiler from the HOST instead of adding the llvm.org
	// bottle, for the first generation on an architecture no registry has. It
	// travels with build.Runner.Bootstrap, which drops the base toolchain for
	// the same reason: a compiler is a tool that RUNS the build, not something
	// the artefact carries.
	//
	// Without it --bootstrap stops one step further along than it used to and
	// no further:
	//
	//	pkgx: GET https://dist.pkgx.dev/llvm.org/linux/s390x/versions.txt: Not Found
	//	bk: the dependency environment failed: pkgx +llvm.org
	//
	// which reads as a network fault and is not one. bottle falls back to the
	// upstream dist for a project our registry does not carry -- right for
	// llvm.org on x86-64, where upstream HAS it -- and upstream carries no
	// s390x at all.
	Bootstrap   bool
	HasBinutils bool // gnu.org/binutils is a dep (darwin AR/RANLIB workaround)
	// LibcPkgx targets the pkgx gnu.org/glibc bottle instead of the build
	// container's system glibc (linux only): the output links its crt objects,
	// libc and dynamic linker from the bottle, so the bottle owes nothing to the
	// debian build container — the sovereign FROM-scratch end state. Opt-in
	// (default off); the caller also adds gnu.org/glibc to Deps so the bottle is
	// present in the eval. C-only for now: the pkgx llvm.org bottle ships no
	// libc++/libunwind, so C++ recipes still need the system libstdc++.
	LibcPkgx bool
	// Glibc, when set (with LibcPkgx), pins the exact pkgx gnu.org/glibc version
	// used as the sysroot — so the output links THAT glibc's symbols and runs on
	// any host whose kernel satisfies it (a chosen HPC floor, e.g. "2.27.0").
	// Empty = newest available. wrapFlags then resolves that sole installed
	// version dir, so no change is needed there.
	Glibc string
}

WrapOptions carries the environment the porcelain wrapper sets up around the user script (the output of Generate). It is a port of libpkgx brewkit's make_build_script.