fetch

README

fetch

A generic zip archive fetcher: race several download hosts, verify the archive's SHA-256 before extraction, and land the extracted tree in one atomic rename.

fetch is a Satellite module of wand, whose launcher uses it to download managed browsers. It knows nothing about browsers: no URL templates, platform names or revisions. Those belong to the caller.

Usage

import "github.com/headlesslab/fetch"

err := fetch.Zip(ctx, "/home/me/.cache/app/tool-1.2.3", []string{
	"https://origin.example.com/tool/1.2.3/tool-linux64.zip",
	"https://mirror.example.net/tool/1.2.3/tool-linux64.zip",
},
	fetch.WithSHA256("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"),
	fetch.WithStripFirstDir(), // the archive nests everything in one top-level folder
	fetch.WithLogger(log.Default()),
)

fetch.Zip(ctx, dir, urls, opts...) fetches one archive into the directory dir:

• Every URL is probed concurrently; the first host to answer with the first bytes of the archive wins. The others stay in reserve, in the order they answered: when a download fails, or its hash does not match, the next responder is tried. When no URL yields a verified archive, the error names every attempt.
• The archive is hashed while it downloads and compared with WithSHA256 before it is opened. A mismatch fails with a *fetch.HashMismatchError naming the expected and the actual digest. Without WithSHA256 the archive is extracted unverified, and the logger says so.
• A file lock (<dir>.lock, via gofrs/flock) is held around download and extraction, so concurrent fetchers of the same directory, in one process or across processes, serialize; the ones that wait find the directory in place and return without downloading. The lock file stays on disk: removing it would race other waiters.
• Extraction goes into a temporary sibling directory that is renamed to dir at the end, so dir either does not exist or is complete. Any failure removes the temporary directory. Entries that would escape the destination (absolute paths, .., a symbolic link whose target leads outside once every earlier link is followed, an entry placed under a symbolic link) fail the fetch. File modes and symbolic links are preserved; on Windows, recreating a symbolic link needs a privilege or Developer Mode, so an archive with one fails there.
• WithStripFirstDir expects exactly one top-level folder in the archive and makes that folder's content dir, as browser archives nest one folder. Anything else at the top level fails the fetch rather than being dropped.
• When dir already exists, Zip returns without a request. Remove dir to fetch again.
• The context cancels probing, download and extraction. WithClient supplies the *http.Client (default http.DefaultClient); WithLogger receives one line per event: the host chosen, progress, verification, extraction.

Zip is the only archive format. Requires Go 1.21 or later.

Development

The suite runs against ephemeral HTTP servers and needs no network:

go test -race ./...

CI runs through the reusable workflow in headlesslab/.github on Linux (Go floor and stable), Windows and macOS, with golangci-lint and govulncheck.

License

MIT.

Documentation

Overview

Package fetch downloads one zip archive from the first of several hosts to respond, verifies its SHA-256 before extraction, and lands the extracted tree in the destination directory with one atomic rename.

The package knows nothing about what the archive holds: URL templates, platform names and version pins belong to the caller.

Index

• func Zip(ctx context.Context, dir string, urls []string, opts ...Option) error
• type HashMismatchError
  • func (e *HashMismatchError) Error() string
• type Logger
• type Option
  • func WithClient(client *http.Client) Option
  • func WithLogger(logger Logger) Option
  • func WithSHA256(hexDigest string) Option
  • func WithStripFirstDir() Option

Functions

func Zip

func Zip(ctx context.Context, dir string, urls []string, opts ...Option) error

Zip fetches the archive served at any of urls and extracts it into dir.

Every URL is probed concurrently and the first host to answer with the start of the archive is downloaded from; the others are fallbacks, in the order they answered, for a download that fails or does not match the expected digest. Extraction goes into a temporary sibling of dir that is renamed into place at the end, so dir either does not exist or is complete. A file lock next to dir serializes concurrent fetchers of the same directory; when dir already exists, Zip returns without a request.

Types

type HashMismatchError

type HashMismatchError struct {
	URL      string
	Expected string
	Actual   string
}

HashMismatchError reports an archive whose SHA-256 differs from the one given with WithSHA256. Nothing was extracted from it.

func (*HashMismatchError) Error

func (e *HashMismatchError) Error() string

type Logger

type Logger interface {
	Println(v ...any)
}

Logger receives one line per event: the host chosen, download progress, verification and extraction. A *log.Logger satisfies it.

type Option

type Option func(*fetcher)

Option configures Zip.

func WithClient

func WithClient(client *http.Client) Option

WithClient sets the HTTP client. The default is http.DefaultClient.

func WithLogger

func WithLogger(logger Logger) Option

WithLogger sets where event lines go. The default discards them.

func WithSHA256

func WithSHA256(hexDigest string) Option

WithSHA256 sets the expected hex digest of the archive. The download is hashed as it streams and compared before the archive is opened; a mismatch fails with a *HashMismatchError. An empty digest means no verification, which the logger reports.

func WithStripFirstDir

func WithStripFirstDir() Option

WithStripFirstDir expects the archive to hold exactly one top-level directory and makes that directory's content the destination, the way browser archives nest one folder.