fetch

Documentation

Overview

Package fetch downloads and extracts pkgx package sources.

It supports the archive formats used by pantry distributables (.tar.gz/.tgz, .tar.xz, .tar.bz2/.tbz2, .tar, .zip) plus shallow git clones via the system git binary. Extraction is pure Go, preserves file modes, recreates symlinks, and rejects archive entries whose paths are absolute or would escape the destination directory.

Index

• Variables
• func DeclaredSHA256(url, archiveName string) (string, error)
• func ExtractTarGzFile(src, destDir string, strip int) error
• func Fetch(url, destDir string, stripComponents int) (string, error)
• func FetchGit(repoURL, ref, destDir string) (string, error)

Variables

var ErrSourcePinMismatch = errors.New("fetch: this URL has served different bytes before")

Mirror, when set, is handed every archive Fetch downloads: its path on disk, its sha256 in lowercase hex, and the URL it came from.

It exists because 57% of this pantry is built from tarballs GitHub GENERATES on request rather than stores — of 165 such URLs probed, not one advertises a Content-Length, because the object does not exist until someone asks. For those sources a mirror is not a copy of anything: it is the first stored artefact they have ever had, and this is the only place in the build where the bytes and their digest are both in hand.

It returns nothing on purpose. A build holding the bytes it needs must not fail because a registry was unreachable — the mirror serves the NEXT rebuild. ErrSourcePinMismatch is returned when a URL serves bytes other than the ones it served before. It is wrapped with both digests and the URL.

var Mirror func(archivePath, sha256hex, url string)

An implementation that wants its failures seen logs them itself.

var Pin func(url, sha256hex string) error

Pin, when set, is consulted with a download's URL and the digest it actually has, BEFORE Mirror is given the chance to store it. Returning an error fails the fetch.

This is trust on first use, and it is the only control there is. Exactly one recipe of 904 in the pantry carries a `sha:`, and that one points at a `.sha256` served by the same host as the tarball — it catches corruption, not substitution. For everything else TLS is the whole of it, across 233 distinct upstream hosts.

It does not make an upstream trustworthy. It makes a CHANGE visible, which is the part nobody had. A version bump is not a change here: the version is in the URL, so a new release asks a question that has never been asked and is recorded rather than refused. What fires is the same URL serving different bytes — a re-cut tarball, a compromised mirror, a hijacked domain — and those are exactly the cases nothing else would notice.

Separate from Mirror because the contracts differ. Mirror is best-effort and swallows its own failures: a build that already holds the bytes must not die because a registry was unreachable. A refusal must not be swallowed.

Functions

func DeclaredSHA256

func DeclaredSHA256(url, archiveName string) (string, error)

DeclaredSHA256 fetches a checksum file and returns the SHA-256 it declares for archiveName, in lowercase hex.

The pantry format already had this: `sha: ${{url}}.sha256` points at what the upstream publishes NEXT to the tarball, so the expected digest moves with the version instead of being re-written into the recipe for each one. bk simply never read it.

Three shapes are in the wild and all three appear in this ecosystem:

b6a5f44b…                          bare hex (openssl.org)
b6a5f44b…  openssl-3.6.0.tar.gz    coreutils `sha256sum` output
SHA256 (openssl-3.6.0.tar.gz) = b6a5f44b…   BSD `sha256 -r` output

A file listing MANY archives is matched on the basename, because taking the first line of a SHA256SUMS covering a whole release would verify the download against some other file's digest and call it a pass.

func ExtractTarGzFile

func ExtractTarGzFile(src, destDir string, strip int) error

ExtractTarGzFile extracts the local gzip-compressed tar at src into destDir, stripping strip leading path components (like `tar --strip-components=N`). It reuses the same hardened extractor as Fetch, so absolute or dest-escaping entries are rejected rather than written. bkpyvenv's poetry seal uses it to unpack a freshly built sdist into the in-project venv's site-packages.

func Fetch

func Fetch(url, destDir string, stripComponents int) (string, error)

Fetch downloads url and extracts the archive into destDir, stripping stripComponents leading path components from every entry (like `tar --strip-components=N`). The archive format is detected from the URL's extension: .tar.gz/.tgz, .tar.xz, .tar.bz2/.tbz2, .tar, .zip.

It returns the SHA-256 of the bytes that arrived, in lowercase hex. The factory signs and attests the bottle it produces; until this existed nothing recorded what the bottle was made FROM, so a source tarball that changed upstream — 57% of this pantry is built from tarballs GitHub generates on request rather than stores — produced a different bottle, correctly signed, whose provenance named a URL and not the bytes that came back from it. The digest is returned even when extraction then fails: what arrived is worth knowing precisely when it was not what was expected.

func FetchGit

func FetchGit(repoURL, ref, destDir string) (string, error)

FetchGit shallow-clones ref of repoURL into destDir with go-git's pure-Go clone — NO `git` binary dependency. `git clone --branch <ref>` accepts either a tag or a branch, so try the ref as a tag first, then as a branch (cleaning the destination between attempts, since PlainClone needs an empty target). FetchGit shallow-clones repoURL at ref into destDir and returns the commit hash it landed on. A tag is a moving target — it can be deleted and re-cut at different content, and nothing in a clone says it was — so the commit is the only thing about a git source worth attesting.