hostwire

Documentation

Overview

Package hostwire holds the pieces of the orchestrator-to-host-agent wire that both the firecracker and qemu providers speak. Today that is the raw HTTP/1.1 upgrade used to open an attach stream to a guest; the JSON request path still lives in each provider.

Index

• Constants
• Variables
• func AttachQuery(spec orchestrator.AttachSpec) url.Values
• func Dial(ctx context.Context, baseURL, token, path, proto string) (net.Conn, error)
• func MintArtifactGrant(servingToken, digest string, ttl time.Duration) (string, error)
• func NewArtifactClient() *http.Client
• func NewClient() *http.Client
• func ParseAttachQuery(q url.Values) orchestrator.AttachSpec
• func ReadErrorBody(r io.Reader) string
• func ValidateAgentURL(raw string) error
• func VerifyArtifactGrant(servingToken, wantDigest, raw string) error
• type ArtifactFetchOptions
• type ArtifactFetchResult
  • func FetchArtifact(ctx context.Context, client *http.Client, opts ArtifactFetchOptions) (ArtifactFetchResult, error)
• type ArtifactGrant
  • func ParseArtifactGrant(raw string) (ArtifactGrant, error)
• type ArtifactPullRequest
• type ArtifactPullResponse
  • func PullArtifact(ctx context.Context, client *http.Client, baseURL, token string, ...) (ArtifactPullResponse, error)

Constants

const (
	// ArtifactGrantHeader carries the grant on the peer-to-peer fetch. It is a
	// distinct header from Authorization on purpose: a grant is not a bearer
	// token for the agent API, and mixing them invites a verifier that
	// accidentally accepts one where it meant the other.
	ArtifactGrantHeader = "X-Fuse-Artifact-Grant"
)

Host-to-host artifact grants.

A host pulling an artifact from another host is a new trust edge: until now every agent only ever trusted `Bearer $FC_AGENT_TOKEN` presented by the orchestrator. Handing the pulling host the serving host's token would over-grant catastrophically, because that token is full control of the serving host: create VMs, exec in guests, delete snapshots.

Instead the orchestrator mints a capability that the serving host can verify entirely on its own. The key is the serving host's own agent token, which needs no new key distribution: the orchestrator already stores every host's token in order to talk to it at all.

grant = "v1." + digest + "." + expiry_unix + "." + nonce + "." + mac
mac   = HMAC-SHA256(
            key = <FC_AGENT_TOKEN of the SERVING host>,
            msg = "fuse-artifact-grant/v1\n" + digest + "\n"
                  + expiry_unix + "\n" + nonce)

What this buys, and why each part matters:

• the pulling host never learns the serving host's token. HMAC is one way, so possession of (message, mac) does not yield the key.
• the grant authorizes exactly one digest on exactly one endpoint. The digest is inside the signed preimage and the verifier also checks it against the digest in the request path, so a grant for blob X cannot be replayed against blob Y, and it is not a general credential for the serving agent's API.
• the grant is worthless after expiry. Nothing has to revoke it.
• the serving host never has to call the orchestrator to verify. Pulls stay possible while the control plane is busy, restarting, or unreachable, and verification costs one HMAC rather than a network round trip.

What it deliberately does NOT defend against, so nobody assumes more:

• within its TTL the grant is replayable by anyone who holds it. There is no per-grant server-side state, no nonce ledger, no binding to the pulling host's identity or address. That is acceptable because the only thing a replay achieves is re-reading one blob whose contents the holder was already authorized to read. It grants no write, no exec, and no access to any other artifact.
• it says nothing about who the puller is. It is a bearer capability, not an authentication of the peer.
• it does not protect the blob in transit. Confidentiality on the wire is TLS's job, exactly as it is for every other orchestrator-to-agent call.

The live verifier is the Python host agent, which recomputes the same mac and compares it with hmac.compare_digest. The Go verifier below is not on that path; it exists so the scheme is testable in this repo and so the wire format is pinned by a second independent implementation. If the two ever disagree, one of them is a bug and the tests here are what catch it.

const AttachProto = "fuse-attach/1"

AttachProto is the value of the Upgrade header that opens an attach stream. It is spoken on both hops — client to orchestrator, and orchestrator to host agent — so the orchestrator can relay bytes without reframing them.

const DefaultArtifactGrantTTL = 5 * time.Minute

DefaultArtifactGrantTTL is how long a minted grant stays valid. Five minutes is long enough to cover scheduling the pull and the peer opening the connection, and short enough that a leaked grant is stale before anyone can do much with it. Note that the TTL bounds when the transfer may START, not how long it may run: the serving agent checks the grant once, at request time, so a multi-hour 40 GiB transfer is not killed by a five minute grant.

const (

	// DefaultArtifactStallTimeout is the maximum time the transfer may make no
	// progress at all before it is abandoned. A healthy link, however slow,
	// still delivers something inside this window; two minutes of complete
	// silence means the peer or the path is gone.
	DefaultArtifactStallTimeout = 2 * time.Minute
)

Timeouts for a blob transfer.

There is deliberately NO overall request timeout. NewClient's 10 minute ceiling is right for control-plane calls, whose duration is bounded by what the agent has to do; it is wrong here, because transfer time is a function of artifact size and link speed, neither of which this side knows. A total deadline would kill a perfectly healthy 40 GiB pull over a slow link, and the only way to pick one that never does is to pick one so large it stops detecting anything. Any fixed number is either too small for the biggest legitimate transfer or too large to be a useful failure detector.

So instead of bounding the whole thing, bound the parts whose duration does NOT scale with size (connect, TLS, and time to first byte), and then require PROGRESS on the body via a stall timeout. A transfer that is moving is allowed to take as long as it takes; a transfer that has stopped moving fails quickly, which is the condition anyone actually wanted to detect.

const DefaultMaxArtifactBytes int64 = 128 << 30

DefaultMaxArtifactBytes caps how much a peer may stream before the pull is abandoned. Without a cap, a hostile or broken peer can serve an endless body and fill the puller's disk, and the digest check is no defence because it only fails once the stream ends.

128 GiB is well above any plausible rootfs artifact (the largest environment disks in the fleet are an order of magnitude smaller) while still being a number a disk can survive. Callers with tighter knowledge of the artifact, for instance from a size recorded at snapshot time, should pass a smaller value.

const MaxErrorBodyBytes = 64 << 10

MaxErrorBodyBytes bounds how much of a non-2xx response body is read into an error message. A host agent that answers an error with an unbounded stream would otherwise be read to completion into memory, and the interesting part of any real agent error is in the first line anyway.

Variables

var (
	ErrArtifactTooLarge       = errors.New("artifact exceeds the maximum allowed size")
	ErrArtifactDigestMismatch = errors.New("artifact digest does not match")
	ErrArtifactStalled        = errors.New("artifact transfer stalled")
)

Blob transfer failures. These are separate sentinels because they mean genuinely different things to a caller: a digest mismatch is corruption or an attack and must never be retried against the same peer blindly, a size overrun is a policy stop, and a stall is a transport problem worth retrying.

var (
	ErrArtifactGrantInvalid        = errors.New("artifact grant is not valid")
	ErrArtifactGrantMalformed      = fmt.Errorf("%w: malformed", ErrArtifactGrantInvalid)
	ErrArtifactGrantExpired        = fmt.Errorf("%w: expired", ErrArtifactGrantInvalid)
	ErrArtifactGrantDigestMismatch = fmt.Errorf("%w: digest mismatch", ErrArtifactGrantInvalid)
	ErrArtifactGrantSignature      = fmt.Errorf("%w: signature mismatch", ErrArtifactGrantInvalid)
)

Grant verification failures. Every specific reason wraps ErrArtifactGrantInvalid so a caller can reject with one check, which is what the HTTP layer wants: the response is a bare 403 that says nothing about which check failed, because telling an attacker whether the mac or the expiry was wrong is free help. The specific sentinels exist for tests and logs on the trusted side.

var ErrRedirectNotAllowed = errors.New("host agent redirects are not followed")

ErrRedirectNotAllowed is returned when a host agent answers with a redirect.

Functions

func AttachQuery

func AttachQuery(spec orchestrator.AttachSpec) url.Values

AttachQuery encodes an AttachSpec as the query string of an attach request. The spec rides in the URL rather than a body because the upgrade is a GET: there is no body to put it in, and inventing a pre-upgrade handshake frame would buy nothing.

func Dial

func Dial(ctx context.Context, baseURL, token, path, proto string) (net.Conn, error)

Dial opens a raw duplex connection to a host-agent endpoint by performing an HTTP/1.1 Upgrade by hand, and returns the socket once the host agent has answered 101.

It deliberately bypasses http.Client. Two reasons, both structural:

• net/http gives a client no way to reclaim the underlying connection after a response; only servers get Hijack. An upgrade is exactly the case where the caller needs the socket back.
• An http.Client speaking TLS may negotiate HTTP/2, and HTTP/2 has no connection upgrade at all. Writing the request onto a raw conn pins us to HTTP/1.1, where the upgrade is well defined.

The returned conn is positioned immediately after the 101 response, so the first byte read from it is the first byte of the stream proper.

func MintArtifactGrant

func MintArtifactGrant(servingToken, digest string, ttl time.Duration) (string, error)

MintArtifactGrant returns a grant authorizing the holder to fetch exactly digest from the host whose agent token is servingToken.

servingToken is the SERVING host's token, never the pulling host's: the point of the scheme is that the serving host can verify with a key it already has. Passing the wrong host's token produces a grant that simply fails verification, which is a confusing failure, so the orchestrator should take the token from the same host record it takes the peer URL from.

A ttl of zero or less means DefaultArtifactGrantTTL.

func NewArtifactClient

func NewArtifactClient() *http.Client

NewArtifactClient returns the HTTP client used for the blob transfer hop.

It is a separate client from NewClient rather than a tuning of it because the two want opposite things: NewClient exists to guarantee no call runs long, and this one exists to allow exactly that for a single endpoint while still failing fast on a peer that has stopped talking. See the timeout constants above for why a total deadline is the wrong tool.

Redirects are refused for the same reason NewClient refuses them, with one addition: this request carries a grant that authorizes reading one blob from one host, and following a redirect would present that grant to a host the orchestrator never chose.

func NewClient

func NewClient() *http.Client

NewClient returns the HTTP client the providers use to reach a host agent.

It exists because http.DefaultClient has no timeout at all: a host that accepts the connection and never answers would pin a provider call, and through it a fleet operation, forever.

Redirects are refused rather than followed. Requests to a host agent carry that host's bearer token in a header, and Go's default policy forwards headers on same-host redirects while a redirect to a new origin would still take the orchestrator somewhere the operator never registered. Neither is something a host agent has any reason to ask for.

func ParseAttachQuery

func ParseAttachQuery(q url.Values) orchestrator.AttachSpec

ParseAttachQuery is the inverse of AttachQuery, used by the orchestrator to read a client's attach request before relaying it onward.

func ReadErrorBody

func ReadErrorBody(r io.Reader) string

ReadErrorBody reads at most MaxErrorBodyBytes of an error response body and returns it trimmed, for use in an error message.

func ValidateAgentURL

func ValidateAgentURL(raw string) error

ValidateAgentURL reports whether raw is an acceptable host-agent base URL.

The URL is caller-supplied at host registration and the orchestrator then makes authenticated requests to it, so it is a destination policy decision, not a formatting one. The policy:

• http or https only. Any other scheme (file, gopher, unix, ...) is not a host agent, and some of them read local state.
• a host must be present, and must not be an empty or malformed authority.
• no embedded credentials. A userinfo section would be sent to the destination on every request and would end up in stored config.
• no query string and no fragment. Paths are concatenated onto this base URL, so anything after the path is either ignored or silently changes the request that gets built.

A path prefix is allowed: a host agent may legitimately sit behind a reverse proxy at /agent.

func VerifyArtifactGrant

func VerifyArtifactGrant(servingToken, wantDigest, raw string) error

VerifyArtifactGrant checks that raw is a live grant, signed with servingToken, for wantDigest.

wantDigest is the digest from the request path, not from the grant: checking the grant against itself would prove nothing. The checks run in the order the design fixes (shape, digest, expiry, mac) and any failure is a single opaque rejection to the peer.

Types

type ArtifactFetchOptions

type ArtifactFetchOptions struct {
	// PeerBaseURL is the serving host agent's base URL, in the same form as
	// any other host agent URL (see ValidateAgentURL).
	PeerBaseURL string

	// Digest is the expected hex sha256 of the artifact. It is both the thing
	// being asked for and the thing being checked: content addressing means
	// the request and the integrity check are the same value.
	Digest string

	// Grant is the orchestrator-minted capability, sent in ArtifactGrantHeader.
	Grant string

	// DestPath is where a VERIFIED artifact lands. Nothing is ever written to
	// this path directly; see FetchArtifact.
	DestPath string

	// MaxBytes overrides DefaultMaxArtifactBytes when positive.
	MaxBytes int64

	// StallTimeout overrides DefaultArtifactStallTimeout when positive.
	StallTimeout time.Duration
}

ArtifactFetchOptions describes one blob transfer.

type ArtifactFetchResult

type ArtifactFetchResult struct {
	Path      string
	SizeBytes int64
}

ArtifactFetchResult describes a transfer that completed AND verified.

func FetchArtifact

func FetchArtifact(ctx context.Context, client *http.Client, opts ArtifactFetchOptions) (ArtifactFetchResult, error)

FetchArtifact streams one artifact from a peer host agent, verifying it as the bytes arrive, and only then places it at DestPath.

The API takes a destination path rather than an io.Writer on purpose. A writer would hand the caller every byte as it arrived, and the caller would then be responsible for not using those bytes until this function returned, which is precisely the mistake that turns "we verify artifacts" into "we verify artifacts, usually". Here, unverified bytes only ever exist in a temp file this function owns, and the single moment the data becomes visible under DestPath is after the digest matched. A caller cannot use the artifact early because until success there is nothing at DestPath to use.

The commit is a fresh temp inode plus rename, matching the pattern the firecracker host agent uses on restore. That pattern exists there because writing into a path something else may already hold open produced corrupt (NUL-filled) images; the same hazard applies to an artifact another pull or a running VM may be reading.

On any failure the temp file is removed, so a failed pull leaves nothing behind and nothing partial is ever reachable at DestPath.

type ArtifactGrant

type ArtifactGrant struct {
	Digest string
	Expiry time.Time
	Nonce  string // hex
	MAC    string // hex
}

ArtifactGrant is a parsed grant. The fields are exactly the signed preimage plus the mac, so a parsed grant can be re-verified without the original string.

func ParseArtifactGrant

func ParseArtifactGrant(raw string) (ArtifactGrant, error)

ParseArtifactGrant splits a grant into its fields without verifying it.

Parsing is separate from verification because the serving agent has to read the digest out of the grant before it can compare it to the one in the URL path, and because a malformed grant should be rejected without ever touching the token. A parsed grant is NOT a trusted grant; nothing may act on one until VerifyArtifactGrant has returned nil.

type ArtifactPullRequest

type ArtifactPullRequest struct {
	// Digest travels in the URL path, not the body, so the agent can route and
	// authorize on it without parsing a body. It is here so callers pass one
	// value object.
	Digest string `json:"-"`

	// PeerURL is the base URL of the host agent that already has the artifact.
	PeerURL string `json:"peer_url"`

	// Grant authorizes exactly this digest on exactly that peer. It is minted
	// with the PEER's agent token; see MintArtifactGrant.
	Grant string `json:"grant"`

	// SnapshotID is the local id the pulled artifact is stored under. It is
	// optional: the agent derives one from the digest when it is empty. The
	// orchestrator should set it, because the id is what its own snapshot
	// records key on and letting the agent invent one means the control plane
	// has to read it back to know what it now has.
	SnapshotID string `json:"snapshot_id,omitempty"`
}

ArtifactPullRequest is the body of POST /v1/artifacts/{digest}/pull, the call the orchestrator makes to tell a host to fetch an artifact from a peer.

Keeping the shape in one struct means renaming a field is one edit here and one in the agent, rather than a search across every call site.

type ArtifactPullResponse

type ArtifactPullResponse struct {
	SnapshotID string `json:"snapshot_id"`
	Digest     string `json:"digest"`
	SizeBytes  int64  `json:"bytes"`
	CreatedAt  string `json:"created_at,omitempty"`
	// SourcePeer is the peer the bytes came from. Provenance matters here in
	// a way it does not for a locally built snapshot: there is no origin VM on
	// this host to point at.
	SourcePeer string `json:"source_peer,omitempty"`
}

ArtifactPullResponse is what the pulling agent answers once the artifact is verified and committed. It is the agent's snapshot record, so the field names follow the agent's meta.json rather than being invented here.

func PullArtifact

func PullArtifact(ctx context.Context, client *http.Client, baseURL, token string, req ArtifactPullRequest) (ArtifactPullResponse, error)

PullArtifact asks the host agent at baseURL to pull an artifact from a peer.

This is an ordinary control-plane call: small body, the TARGET host's own bearer token, and NewClient's timeouts. It is not the transfer, and must not be given the artifact client: the request returns when the agent has done the work it is going to do, and if that work is a long blocking transfer then the ceiling belongs on the agent side, not on an orchestrator call that would otherwise leak a goroutine per pull with no bound at all.