fetch

Documentation

Overview

Package fetch downloads a file from an untrusted source and verifies it before it is installed: the generic, security-first transport for pulling any file (model weights, a plugin, a dataset) onto the local machine. It is download-and-verify ONLY: it writes a verified file to disk and never parses, loads, or executes it. What to do with the file, and any content policy, is the caller's concern; this package guarantees that the bytes on disk are exactly what was asked for, or that nothing is written at all.

It is distinct from the integration request transport, which makes API calls: this one streams a large body, caps and hashes it as it arrives, verifies a digest, and installs atomically.

Every layer is defensive. The transport refuses anything but https and refuses to connect to a non-public address (anti-SSRF, re-checked on every redirect hop so DNS rebinding cannot slip through, with no environment proxy to route around it). The download is capped on the stream so a hostile server cannot exhaust the disk, hashed as it is written, and verified against a caller-pinned digest so a compromised source cannot substitute a file. The install is atomic, so a partial, oversized, or mismatched download never appears as a usable file.

Index

• type Downloader
  • func New(opts ...Option) *Downloader
  • func (d *Downloader) Fetch(ctx context.Context, req Request) (Result, error)
• type Option
  • func WithHTTPClient(c *http.Client) Option
• type Request
• type Result

Types

type Downloader

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader performs verified downloads over a hardened HTTP client.

func New

func New(opts ...Option) *Downloader

New builds a Downloader. With no client injected it uses SafeClient, the anti-SSRF, https-only transport.

func (*Downloader) Fetch

func (d *Downloader) Fetch(ctx context.Context, req Request) (Result, error)

Fetch downloads req.URL, verifies it, and installs it atomically at req.Dest, returning what was verified. It never executes the file. Any failure leaves no file at Dest: a partial, oversized, or digest-mismatched download is discarded.

type Option

type Option func(*Downloader)

Option configures a Downloader.

func WithHTTPClient

func WithHTTPClient(c *http.Client) Option

WithHTTPClient injects the HTTP client, so a test can supply one that reaches a local server. Production uses the default hardened client (see SafeClient).

type Request

type Request struct {
	// URL is the https source of the weights.
	URL string
	// Dest is the final install path; the file appears here only after it verifies.
	Dest string
	// ExpectSHA256 is the pinned digest the download is checked against (an optional
	// "sha256:" prefix is accepted). Empty means pin-on-fetch: the computed digest is
	// returned but nothing pre-pinned was verified, which is lower trust.
	ExpectSHA256 string
	// MaxBytes is the hard cap on the download size; 0 uses a safe default ceiling.
	MaxBytes int64
}

Request describes one verified download.

type Result

type Result struct {
	Path   string
	Bytes  int64
	SHA256 string
	// Pinned is true when the bytes were verified against a caller-pinned digest, the
	// strong-trust case; false when the digest was only computed (pin-on-fetch).
	Pinned bool
}

Result reports a completed, verified download.