fetch

Documentation

Overview

Package fetch downloads and extracts pkgx package sources.

It supports the archive formats used by pantry distributables (.tar.gz/.tgz, .tar.xz, .tar.bz2/.tbz2, .tar, .zip) plus shallow git clones via the system git binary. Extraction is pure Go, preserves file modes, recreates symlinks, and rejects archive entries whose paths are absolute or would escape the destination directory.

Index

• Variables
• func DeclaredSHA256(url, archiveName string) (string, error)
• func ExtractTarGzFile(src, destDir string, strip int) error
• func Fetch(url, destDir string, stripComponents int) (string, error)
• func FetchGit(repoURL, ref, destDir string) (string, error)

Variables

var Mirror func(archivePath, sha256hex, url string)

Mirror, when set, is handed every archive Fetch downloads: its path on disk, its sha256 in lowercase hex, and the URL it came from.

It exists because 57% of this pantry is built from tarballs GitHub GENERATES on request rather than stores — of 165 such URLs probed, not one advertises a Content-Length, because the object does not exist until someone asks. For those sources a mirror is not a copy of anything: it is the first stored artefact they have ever had, and this is the only place in the build where the bytes and their digest are both in hand.

It returns nothing on purpose. A build holding the bytes it needs must not fail because a registry was unreachable — the mirror serves the NEXT rebuild. An implementation that wants its failures seen logs them itself.

Functions

func DeclaredSHA256

func DeclaredSHA256(url, archiveName string) (string, error)

DeclaredSHA256 fetches a checksum file and returns the SHA-256 it declares for archiveName, in lowercase hex.

The pantry format already had this: `sha: ${{url}}.sha256` points at what the upstream publishes NEXT to the tarball, so the expected digest moves with the version instead of being re-written into the recipe for each one. bk simply never read it.

Three shapes are in the wild and all three appear in this ecosystem:

b6a5f44b…                          bare hex (openssl.org)
b6a5f44b…  openssl-3.6.0.tar.gz    coreutils `sha256sum` output
SHA256 (openssl-3.6.0.tar.gz) = b6a5f44b…   BSD `sha256 -r` output

A file listing MANY archives is matched on the basename, because taking the first line of a SHA256SUMS covering a whole release would verify the download against some other file's digest and call it a pass.

func ExtractTarGzFile

func ExtractTarGzFile(src, destDir string, strip int) error

ExtractTarGzFile extracts the local gzip-compressed tar at src into destDir, stripping strip leading path components (like `tar --strip-components=N`). It reuses the same hardened extractor as Fetch, so absolute or dest-escaping entries are rejected rather than written. bkpyvenv's poetry seal uses it to unpack a freshly built sdist into the in-project venv's site-packages.

func Fetch

func Fetch(url, destDir string, stripComponents int) (string, error)

Fetch downloads url and extracts the archive into destDir, stripping stripComponents leading path components from every entry (like `tar --strip-components=N`). The archive format is detected from the URL's extension: .tar.gz/.tgz, .tar.xz, .tar.bz2/.tbz2, .tar, .zip.

It returns the SHA-256 of the bytes that arrived, in lowercase hex. The factory signs and attests the bottle it produces; until this existed nothing recorded what the bottle was made FROM, so a source tarball that changed upstream — 57% of this pantry is built from tarballs GitHub generates on request rather than stores — produced a different bottle, correctly signed, whose provenance named a URL and not the bytes that came back from it. The digest is returned even when extraction then fails: what arrived is worth knowing precisely when it was not what was expected.

func FetchGit

func FetchGit(repoURL, ref, destDir string) (string, error)

FetchGit shallow-clones ref of repoURL into destDir with go-git's pure-Go clone — NO `git` binary dependency. `git clone --branch <ref>` accepts either a tag or a branch, so try the ref as a tag first, then as a branch (cleaning the destination between attempts, since PlainClone needs an empty target). FetchGit shallow-clones repoURL at ref into destDir and returns the commit hash it landed on. A tag is a moving target — it can be deleted and re-cut at different content, and nothing in a clone says it was — so the commit is the only thing about a git source worth attesting.