vmimage

package
v0.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: MIT Imports: 33 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MediaTypeOCIIndex    = "application/vnd.oci.image.index.v1+json"
	MediaTypeOCIManifest = "application/vnd.oci.image.manifest.v1+json"
	MediaTypeOCIConfig   = "application/vnd.oci.image.config.v1+json"
	MediaTypeOCILayer    = "application/vnd.oci.image.layer.v1.tar+gzip"

	// MediaTypeShedKernel and MediaTypeShedInitrd are shed-specific blob
	// types carried alongside the standard rootfs layer. Foreign OCI
	// tools see them as generic blobs and skip them; shed reads them as
	// the kernel/initrd to boot the VM.
	MediaTypeShedKernel = "application/vnd.shed.kernel"
	MediaTypeShedInitrd = "application/vnd.shed.initrd"
)

OCI media types used by shed. We standardize on the OCI variants (application/vnd.oci.*) rather than the legacy Docker variants — both reference implementations interoperate, but OCI is the spec.

View Source 
const (
	// AnnotationVariant marks the variant of a shed image
	// (e.g. "base", "extensions", "full"). Display-only.
	AnnotationVariant = "io.shed.variant"

	// AnnotationKernelDigest names the blob digest of the kernel
	// associated with this image, when shed extracted one.
	AnnotationKernelDigest = "io.shed.kernel.digest"

	// AnnotationInitrdDigest names the blob digest of the initrd
	// associated with this image, when shed extracted one.
	AnnotationInitrdDigest = "io.shed.initrd.digest"

	// AnnotationRootfsErofsDigest names the blob digest of the
	// flattened rootfs erofs that shed minted at image-build time
	// (using a pinned mkfs.erofs from the shed-build-tools image).
	// When present, the host skips local mkfs.erofs entirely and
	// mounts this blob directly as the read-only lower; when absent,
	// the host rejects the image (no on-host fallback in the v0.5.2+
	// layout — see internal/vmimage/manager.go for the reject path).
	AnnotationRootfsErofsDigest = "io.shed.rootfs.erofs.digest"

	// AnnotationSchemaVersion records shed's own manifest schema version
	// alongside OCI's schemaVersion (always 2 for OCI manifests). Lets
	// us bump shed semantics independently of OCI.
	AnnotationSchemaVersion = "io.shed.schema-version"

	// AnnotationSourceRef preserves the registry ref the image was
	// pulled from or built against. Useful for cache freshness checks.
	AnnotationSourceRef = "io.shed.source-ref"

	// AnnotationRootfsLogicalSize records the logical (sparse) byte
	// size of the derived rootfs ext4. Display-only.
	AnnotationRootfsLogicalSize = "io.shed.rootfs.logical-size"
)

Shed-specific manifest annotations. These live alongside standard OCI annotations (org.opencontainers.image.*) in the manifest's annotations map and on per-descriptor annotations.

View Source 
const (
	BlobStatusDownloading = "downloading" // a byte tick or the initial 0-byte start
	BlobStatusExists      = "exists"      // blob already present in the store
	BlobStatusDone        = "done"        // fully fetched + verified (Current == Total)
)

Blob status values for ProgressEvent.Status when Kind=="blob". These string values cross the wire (the backend forwards Status verbatim), so they are the single source of truth for both sides.

View Source 
const CacheLowerExt = ".erofs"

CacheLowerExt is the file extension used by v0.5.1's derived lower-image cache. Kept so `PruneImages` can recognize and sweep stale entries during the post-upgrade window.

View Source 
const CurrentOCILayoutVersion = "1.0.0"

CurrentOCILayoutVersion is the OCI image-layout version shed writes.

View Source 
const DefaultPlatform = "linux/arm64"

DefaultPlatform is the Docker platform used for VZ images (Apple Silicon).

View Source 
const DigestPrefix = "sha256:"

DigestPrefix is the algorithm prefix for image digests. Always "sha256:".

View Source 
const FirecrackerPlatform = "linux/amd64"

FirecrackerPlatform is the Docker platform used for Firecracker images (x86_64 Linux).

View Source 
const MaxLayers = 16

MaxLayers caps the number of layers a shed image manifest may list. Bounds both the Firecracker drive count (1 upper + N lowers ≤ 17, well under FC's 26-drive limit) and overlayfs lowerdir= argument length (each layer adds a path; PAGE_SIZE ~4096 caps the total).

View Source 
const ShedSchemaVersion = "1"

ShedSchemaVersion is the current shed-specific manifest schema version. Bumped when the annotation contract changes.

Variables

View Source 
var (
	// ErrBlobNotFound is returned when a digest is not present in the store.
	ErrBlobNotFound = errors.New("blob not found")

	// ErrTagNotFound is returned when a tag does not exist.
	ErrTagNotFound = errors.New("tag not found")

	// ErrInvalidDigest is returned when a digest string is malformed.
	ErrInvalidDigest = errors.New("invalid digest")

	// ErrInvalidTag is returned when a tag name is unsafe.
	ErrInvalidTag = errors.New("invalid tag name")

	// ErrLegacyBundledBlob is returned when a blob path is occupied by a
	// v0.4.x bundled directory layout (a directory holding manifest.json,
	// kernel, initrd, rootfs.ext4) instead of the v0.5.0+ flat-file OCI
	// blob. The caller should surface the wrapped message so the user
	// knows to wipe the legacy store. See ocilayout.go ReadBlob for the
	// detection step.
	ErrLegacyBundledBlob = errors.New("legacy v0.4.x bundled blob layout")
)

Sentinel errors for blob/tag operations.

View Source 
var (
	// ErrImageNotFound is returned when a tag does not exist.
	ErrImageNotFound = errors.New("image not found")

	// ErrImageInUse is returned when trying to delete an image referenced
	// by config or an existing shed.
	ErrImageInUse = errors.New("image is in use")

	// ErrLayersMissing is returned when an operation needs layer blobs that
	// aren't present locally — e.g. pushing an image that was pulled
	// boot-only. Re-pull with --with-layers to recover.
	ErrLayersMissing = errors.New("image layers not present locally")
)

Sentinel errors for image operations. Backend wrappers map these to config sentinel errors (e.g., config.ErrImageNotFoundSentinel).

View Source 
var ErrPullDisabled = errors.New("image not present locally and pull_policy=never")

ErrPullDisabled is returned when pull_policy=never and the configured ref is not present in the local store. Callers should surface this as a client-actionable error (4xx), not an internal failure.

Functions

func BlobExists 

func BlobExists(imagesDir, digest string) bool

BlobExists reports whether a blob is present in the store.

func BlobPath 

func BlobPath(imagesDir, digest string) (string, error)

BlobPath returns the on-disk path of a blob keyed by digest. digest must be of the form "sha256:<hex>".

func BlobSize 

func BlobSize(imagesDir, digest string) int64

BlobSize returns the on-disk size of a blob, or 0 if missing.

func BlobsRoot 

func BlobsRoot(imagesDir string) string

BlobsRoot returns {imagesDir}/blobs.

func CacheLowerPath 

func CacheLowerPath(imagesDir, manifestDigest string) (string, error)

CacheLowerPath returns the on-disk path a v0.5.1 install would have used for a manifest's flattened lower. manifestDigest must be of the form "sha256:<hex>". v0.5.2+ never writes to this path.

func CacheLowerSize 

func CacheLowerSize(imagesDir, manifestDigest string) int64

CacheLowerSize returns the on-disk size of the legacy cache file for a manifest, or 0 if absent (the common case post-v0.5.2). Reports actual allocated blocks (st_blocks × 512), not the sparse file logical length, so `shed image ls` SIZE reads true on-disk usage during the upgrade window.

func DeleteBlob 

func DeleteBlob(imagesDir, digest string) error

DeleteBlob removes a blob from the store. Caller is responsible for refcount checks. Returns ErrBlobNotFound if missing.

func DeleteTag 

func DeleteTag(imagesDir, tag string) error

DeleteTag removes a tag file. Returns ErrTagNotFound if missing.

func DeriveTagFromRef added in v0.6.0 

func DeriveTagFromRef(ref string) string

DeriveTagFromRef reduces a Docker ref to a short, filesystem-safe cosmetic name (e.g. ghcr.io/charliek/shed-vz-full:v0.5.9 -> "full"). It is NOT a unique identity — multiple ref versions collapse to the same name — so it is only used as a lock-adjacent label. Resolution identity is the full ref via the ref-index.

func DigestBytes 

func DigestBytes(data []byte) string

DigestBytes computes the sha256 digest of an in-memory byte slice, formatted as "sha256:<hex>".

func DigestReader 

func DigestReader(r io.Reader) (string, int64, error)

DigestReader streams a reader through sha256, returning the digest of the consumed bytes. Reader is fully drained.

func EnsureOCILayout 

func EnsureOCILayout(imagesDir string) error

EnsureOCILayout creates the OCI image-layout markers at imagesDir if they don't already exist. Idempotent.

func FindDigestBySourceRef added in v0.6.0 

func FindDigestBySourceRef(imagesDir, ref string) (digest string, ok bool)

FindDigestBySourceRef scans installed manifests for one whose io.shed.source-ref equals ref, returning its digest. This is a READ-ONLY, O(N-manifests) fallback for the cold paths (rm, prune protection) where a ref may not have a sidecar entry yet (e.g. images present from before the ref-index existed). It deliberately does NOT write the sidecar: the create hot path resolves sidecar-only, so a manifest left behind by `shed image rm` (blob persists until prune, Docker model) must not be silently re-cached for create — `rm` then `create` should re-pull.

func HashFile 

func HashFile(path string) (string, error)

HashFile computes the sha256 digest of a file's contents. Returns a digest string of the form "sha256:<hex>".

func HexDigest 

func HexDigest(digest string) string

HexDigest returns the hex portion of a digest, panicking on malformed input. For test use only.

func IndexManifestDigests 

func IndexManifestDigests(imagesDir string) (map[string]bool, error)

IndexManifestDigests reads index.json and returns the set of manifest digests recorded there. This is the cheap way to discover which blobs are manifests without probe-reading every blob.

Returns an empty set if index.json is missing or unreadable — callers should treat that as "no known manifests" and may fall back to a more expensive scan if they require completeness.

func IndexRemoveByDigest 

func IndexRemoveByDigest(imagesDir, digest string) error

IndexRemoveByDigest drops a manifest entry from index.json. Called by PruneImages after deleting a manifest blob so foreign tools don't see a stale descriptor pointing at a missing blob.

func InstallSyntheticImage 

func InstallSyntheticImage(imagesDir, tagName, sourceRef string, rootfsContent, kernelContent, initrdContent []byte) (string, error)

InstallSyntheticImage writes a faux OCI image into imagesDir using the provided rootfsContent bytes as the layer content. Tags the resulting manifest at tagName.

The synthetic image consists of:

  • one layer blob (the raw rootfsContent — not actually gzipped, but stored at a sha256-keyed blob path so the OCI shape is correct)
  • one config blob (minimal OCI image config)
  • one manifest blob with the standard annotations
  • optional kernel + initrd blobs referenced by manifest annotations
  • a tag pointing at the manifest digest

Returns the manifest digest. Used by tests that previously called InstallBlob with synthetic content.

func IsDockerRef 

func IsDockerRef(s string) bool

IsDockerRef returns true if s is a Docker image reference rather than a filesystem path.

func ListBlobs 

func ListBlobs(imagesDir string) ([]string, error)

ListBlobs returns all installed blob digests.

func ListTags 

func ListTags(imagesDir string) ([]string, error)

ListTags returns all tag names currently present in the store.

func MergeLayers 

func MergeLayers(ctx context.Context, imagesDir string, layerDigests []string, outTarPath string) error

MergeLayers is the manifest-less variant of MergeLayersFromManifest used during image build, where we have the layer digests in hand but the shed-annotated manifest hasn't been minted yet. layerDigests must be in OCI order (lowest at index 0).

func MergeLayersFromManifest 

func MergeLayersFromManifest(ctx context.Context, imagesDir, manifestDigest, outTarPath string) error

MergeLayersFromManifest writes a flattened tar of the manifest's layers to outTarPath. Returns an error if the manifest is missing, has zero layers, or any layer blob fails to decode.

func MintRootfsErofs 

func MintRootfsErofs(ctx context.Context, opts MintErofsOptions) (string, error)

MintRootfsErofs flattens layerDigests, runs mkfs.erofs inside the build-tools container, and installs the resulting erofs as a content-addressed blob in imagesDir. Returns the blob digest (sha256:...) — caller stamps this into the manifest's io.shed.rootfs.erofs.digest annotation.

The mkfs.erofs invocation pins:

  • `-b 4096`: erofs block size. Host page sizes vary (4 KiB on Linux/amd64, 16 KiB on Apple Silicon); fixing the block size keeps the same erofs mountable in both contexts. The guest kernel expects 4 KiB.
  • `-z lz4`: lz4 compression. Random-read friendly, ~50% on-disk reduction vs raw, decompression is cheap enough that the boot path doesn't measurably slow down.
  • `-E force-inode-compact`: 32-byte inodes (vs the default 64-byte extended layout). Saves disk for image rootfs's thousands of files. The 1.7.x writer bug that motivated this whole change was an interaction between this flag and big pcluster headers; mkfs.erofs 1.8+ (which ships in shed-build-tools) fixes it.
  • `-T 0`: zeroes the per-inode mtime field. Without this the erofs digest would vary by clock skew on the build host, breaking content addressing for byte-for-byte identical rootfs content.

Changing any of these flags changes the produced digest. The shed-build-tools image is versioned in lockstep with shed-server so a flag change rides a known release.

func OpenBlob 

func OpenBlob(imagesDir, digest string) (*os.File, error)

OpenBlob opens a blob for streaming reads. Returns ErrBlobNotFound if missing.

func PlatformOCI 

func PlatformOCI(s string) (*v1.Platform, error)

PlatformOCI parses a Docker-style platform string ("linux/arm64") into a go-containerregistry v1.Platform. Returns an error for malformed input (missing os/arch, extra components, or empty tokens).

func PushFromOCILayout 

func PushFromOCILayout(ctx context.Context, opts PushOptions) error

PushFromOCILayout uploads the on-disk manifest + config + layers to the destination registry. Layer bytes are streamed straight from the OCI store so the registry digests are preserved end-to-end (the byte-perfect push guarantee). Shed-specific kernel/initrd blobs are also uploaded when the manifest carries the corresponding annotations, so other shed instances can pull them back without re-extracting from rootfs.

func ReadBlob 

func ReadBlob(imagesDir, digest string) ([]byte, error)

ReadBlob reads a blob's bytes. Returns ErrBlobNotFound if missing.

func RefIndexDeleteByDigest added in v0.6.0 

func RefIndexDeleteByDigest(imagesDir, digest string)

RefIndexDeleteByDigest removes every ref-index entry pointing at digest. Called by prune/rm after a manifest blob is deleted so a later resolve can't hit a dangling entry. Best-effort: scan errors are logged, not fatal.

func RefIndexGet added in v0.6.0 

func RefIndexGet(imagesDir, ref string) (digest string, ok bool)

RefIndexGet returns the cached manifest digest for ref as a VALIDATED hit: the sidecar entry exists, its stored ref matches the lookup ref (guarding against a sha256 key collision), and the manifest blob is still present. On any validation failure it removes the stale entry and reports a miss (ok=false), so the caller's pull_policy then governs.

Identity here is the ref the image was PULLED BY (the index key), NOT the manifest's io.shed.source-ref annotation — those legitimately differ when the configured ref is a mutable tag (:latest), a digest pin, or a mirror host, while the annotation records the immutable publish ref. Validating against the annotation would delete the entry EnsureImage just wrote.

Blob-completeness beyond the manifest digest (erofs/kernel/initrd) is validated by resolveManifestLower at the call site.

func RefIndexPut added in v0.6.0 

func RefIndexPut(imagesDir, ref, digest string) error

RefIndexPut records ref -> manifest digest in the sidecar ref-index. It is the final commit step of a successful pull/build: callers MUST invoke it only after the manifest, config, layers, kernel/initrd/erofs, and index.json are durable, so a crash can never leave the index pointing at an incomplete digest. The write itself is atomic (temp+rename).

func RefIndexReverse added in v0.6.0 

func RefIndexReverse(imagesDir string) map[string]string

RefIndexReverse returns a digest -> ref map built from all sidecar entries (first entry wins when several refs share a digest). Read-only; used by `ls`/`inspect` so an image is displayed by the ref it was pulled by, not the manifest's publish-time io.shed.source-ref (which differs for mutable tags, digest pins, and mirrors).

func Resolve 

func Resolve(imagesDir, tag, expectedRef string) string

Resolve looks up a tag and returns the path to the prebuilt rootfs erofs blob referenced by the manifest's io.shed.rootfs.erofs.digest annotation, when the tag exists, the manifest is installed, and (when expectedRef is non-empty) the source-ref annotation matches expectedRef. Returns "" otherwise (cache miss, manifest missing annotation, or expectedRef mismatch).

func ResolveTag 

func ResolveTag(imagesDir, tag string) (digest, rootfsPath string, err error)

ResolveTag looks up a tag and returns its manifest digest plus the path to the manifest's prebuilt rootfs erofs blob. Returns ErrTagNotFound or ErrBlobNotFound on miss; returns a clear error when the manifest lacks the v0.5.2+ io.shed.rootfs.erofs.digest annotation.

func SetTag 

func SetTag(imagesDir, tag, digest string) error

SetTag atomically points <tag> at <digest>. The digest does not need to exist (caller's responsibility — useful for record-only flows like recording the original ref of a snapshot whose blob has been pruned).

func ShortDigest 

func ShortDigest(digest string) string

ShortDigest returns the first 12 hex chars of a digest for display.

func SyntheticDigestFromBytes 

func SyntheticDigestFromBytes(content []byte) string

SyntheticDigestFromBytes returns a sha256 digest of the given content formatted as "sha256:<hex>". Used by tests that need a deterministic digest without writing to disk.

func SyntheticDigestFromString 

func SyntheticDigestFromString(s string) string

SyntheticDigestFromString is a convenience wrapper for tests using string literals to generate stable digests.

func TagPath 

func TagPath(imagesDir, tag string) (string, error)

TagPath returns the path to tags/<tag>.json.

func TagsRoot 

func TagsRoot(imagesDir string) string

TagsRoot returns {imagesDir}/tags.

func TryAcquireFileLock 

func TryAcquireFileLock(path string) (release func(), held bool, err error)

TryAcquireFileLock attempts a non-blocking exclusive flock on path. Returns held=true with a no-op release if the lock file does not exist.

func TryAcquireFileLockBlocking 

func TryAcquireFileLockBlocking(path string) (func(), error)

TryAcquireFileLockBlocking takes an exclusive flock on path and blocks until it's available. Use this from tests that need to simulate a live conversion holding the lock.

func ValidateImageName 

func ValidateImageName(name string) error

ValidateImageName validates that an image name is safe for filesystem operations.

func WriteBlob 

func WriteBlob(imagesDir, expectedDigest string, data []byte) (string, error)

WriteBlob installs `data` as a blob in the store. The provided expectedDigest is verified against sha256(data). Returns the blob path. Idempotent: if the blob is already present, returns success without rewriting.

func WriteBlobFromFile 

func WriteBlobFromFile(imagesDir, srcPath string, consume bool) (digest, blobPath string, err error)

WriteBlobFromFile installs the contents of srcPath as a blob in the store. The STAGED bytes are hashed (not the source) so a concurrent writer to srcPath can't end up with the staged content stored under a stale digest. If consume is true, srcPath is renamed into the staging path (saving a copy); otherwise it is copied and srcPath is preserved.

func WriteIndex 

func WriteIndex(imagesDir string, idx *OCIIndex) error

WriteIndex atomically writes the top-level OCI index.

Types

type ConvertOptions 

type ConvertOptions struct {
	// OCIArchivePath is the path to an OCI image-layout tar produced
	// by `docker buildx build --output type=oci,dest=...`. Required.
	OCIArchivePath string

	// DockerRef is the image reference being converted (e.g.
	// ghcr.io/.../shed-vz-full:v1.0.0). Recorded verbatim in the
	// io.shed.source-ref manifest annotation so the server's
	// resolveImage cache-hit check (compares the annotation to the
	// configured ref) matches on subsequent pulls. Not used to fetch
	// content — that's the OCI archive's job.
	DockerRef string

	// Name is the variant name (e.g. "full"); recorded as io.shed.variant.
	Name string

	// ImagesDir is the OCI image-layout root; blobs are written under
	// {ImagesDir}/blobs/sha256/.
	ImagesDir string

	// Platform is the Docker platform (e.g., "linux/arm64"). Defaults to DefaultPlatform.
	Platform string

	// ExtractKernel controls whether the kernel should be extracted and
	// written as a shed-typed blob referenced by the manifest annotation.
	ExtractKernel bool

	// NeedsInitrd controls whether an initrd should be extracted alongside
	// the kernel. Only consulted when ExtractKernel is true. When set,
	// either InitrdSourcePath must be provided OR the Ubuntu-style
	// /boot/initrd.img-* will be extracted from the image (a fallback
	// useful for ad-hoc Dockerfiles; not appropriate for shed images
	// that need the shed-overlay initramfs to assemble overlayfs at boot).
	NeedsInitrd bool

	// InitrdSourcePath, when set, points at a pre-built initrd file (on
	// the host) that shed should install as the image's boot initrd
	// instead of extracting one from the rootfs. The shed build flow
	// builds the shed-overlay initramfs separately via build-initramfs.sh
	// and passes the path here so the resulting image boots through
	// shed's overlay assembly path rather than Ubuntu's regular initrd.
	InitrdSourcePath string

	// BuildToolsRef, when set, triggers MintRootfsErofs after the
	// layer blobs are installed: a docker container running mkfs.erofs
	// from this image flattens the layers and produces the
	// io.shed.rootfs.erofs.digest blob. The shed publish workflow pins
	// this to ghcr.io/charliek/shed-build-tools:<current shed tag>;
	// local `shed image build` runs default to the same pin via
	// cmd/shed/image.go's BuildToolsRef flag (see --build-tools-version).
	// When empty, no erofs is minted — the resulting image will be
	// rejected by v0.5.2+ servers at EnsureImage. Empty is only
	// appropriate for tests or for images that will be re-processed
	// by a later mint pass.
	BuildToolsRef string
}

ConvertOptions configures an OCI-archive ingestion.

OCIArchivePath is the only input mode — Convert reads the OCI image-layout tar, copies its layer + config blobs into the local OCI image-layout, extracts kernel/initrd, mints the rootfs erofs when BuildToolsRef is set, and writes a shed-annotated manifest. The docker image's layer structure is preserved across variants (extensions and full share base's layers in the on-disk store).

type ConvertResult 

type ConvertResult struct {
	// ManifestDigest is the OCI image manifest digest. Used as the
	// image identity in the tag store.
	ManifestDigest string

	// ConfigDigest is the OCI image config digest.
	ConfigDigest string

	// LayerDigests lists the OCI layer descriptor digests (one per
	// layer). Phase 1 always produces exactly one entry; layered
	// conversions land in later phases.
	LayerDigests []string

	// KernelDigest names the shed-typed kernel blob (if extracted).
	KernelDigest string

	// InitrdDigest names the shed-typed initrd blob (if extracted).
	InitrdDigest string

	// RootfsErofsDigest names the prebuilt rootfs erofs blob (only
	// populated when ConvertOptions.BuildToolsRef was non-empty).
	RootfsErofsDigest string

	// RootfsLogicalSize records the uncompressed tar size in bytes.
	RootfsLogicalSize int64
}

ConvertResult holds the digests produced by a successful conversion.

func Convert 

func Convert(ctx context.Context, opts ConvertOptions) (*ConvertResult, error)

Convert dispatches between two input modes:

  • OCIArchivePath: ingest a buildx OCI tar, preserving its layer structure. Shared layers (extensions FROM base) end up referenced by both manifests with identical digests in the store, so disk usage scales with unique-deltas rather than full-rootfs-per-tag.
  • DockerRef: flatten the named local docker daemon image to a single layer via docker create + docker export. Kept for the pull-fallback case where a ref is in the daemon but not on a registry.

The caller advances the tag (SetTag) after Convert returns; this keeps Convert pure with respect to tag indirection.

type Descriptor 

type Descriptor struct {
	MediaType   string            `json:"mediaType"`
	Digest      string            `json:"digest"`
	Size        int64             `json:"size"`
	Annotations map[string]string `json:"annotations,omitempty"`
}

Descriptor is the OCI content-descriptor pointing at a blob by digest.

type EnsureResult 

type EnsureResult struct {
	Path   string // path to the cached lower image (manifest-digest-keyed erofs)
	Digest string // OCI manifest digest
}

EnsureResult is what EnsureImage returns: the path to the content-addressed lower image for the resolved manifest, and the manifest digest itself.

type ImageConfig 

type ImageConfig interface {
	GetDefaultImage() string            // ref (or path) for new sheds when no --image
	GetImageAliases() map[string]string // alias -> ref convenience map
	GetPullPolicy() string              // "missing" (default) | "always" | "never"
	GetImagesDir() string
	GetPlatform() string     // "linux/arm64" or "linux/amd64"
	GetExtractKernel() bool  // true for both VZ and Firecracker
	GetNeedsInitrd() bool    // true for VZ, false for Firecracker
	GetPullConcurrency() int // max concurrent blob downloads per pull (>=1)
}

ImageConfig provides the configuration data needed by image management operations. Both VZConfig and FirecrackerConfig implement this interface.

type ImageInfo 

type ImageInfo struct {
	Name        string // tag name (or "<dangling>" for unreferenced blobs)
	Digest      string // "sha256:..." digest of the underlying OCI manifest
	Tag         string // tag name (same as Name for tagged images, empty for dangling)
	Path        string // path to the first layer's cached ext4 (single-layer compat)
	DockerRef   string // OCI manifest annotation io.shed.source-ref
	SizeBytes   int64  // sum of layer descriptor sizes + cached ext4 bytes
	UniqueBytes int64  // bytes attributable to layers only this manifest references
	SharedBytes int64  // bytes attributable to layers also referenced by other manifests
	Source      string // "config", "user", or "dangling"
	Cached      bool   // manifest blob is installed
	InUse       bool   // protected by a shed or snapshot reference
	Alias       string // friendly image_aliases key (empty for user/dangling)
	IsDefault   bool   // this image's ref is the configured default_image
	BootOnly    bool   // pulled without layer tarballs (boots, can't push without --with-layers)
}

ImageInfo describes an image known to the blob store, addressed by tag. Keep in sync with config.ImageInfo (field-by-field copy in backend wrappers).

type LayerInfo 

type LayerInfo struct {
	Index       int               `json:"index"`
	Digest      string            `json:"digest"`
	Size        int64             `json:"size"`
	MediaType   string            `json:"media_type"`
	CreatedAt   time.Time         `json:"created_at,omitempty"`
	CreatedBy   string            `json:"created_by,omitempty"`
	Comment     string            `json:"comment,omitempty"`
	Variant     string            `json:"variant,omitempty"`
	EmptyLayer  bool              `json:"empty_layer,omitempty"`
	Annotations map[string]string `json:"annotations,omitempty"`
}

LayerInfo describes one layer in an image's history.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager handles image lifecycle: ensure, list, delete, prune. All file locking happens inside the Manager.

func NewManager 

func NewManager(cfg ImageConfig, scanner RefScanner) *Manager

NewManager creates a new image Manager with the given configuration.

func (*Manager) DeleteImage 

func (m *Manager) DeleteImage(ident string) error

DeleteImage removes an image's addressability — every cosmetic tag and the ref-index entry pointing at the resolved manifest. Following the Docker model, the underlying manifest blob is NOT removed; call PruneImages to garbage-collect once nothing references it. ident may be a Docker ref, a digest, or a cosmetic tag label.

Hard-blocks only when the manifest is pinned by a live shed or snapshot. (Warning when the target is the configured default_image is a CLI concern.)

func (*Manager) EnsureImage 

func (m *Manager) EnsureImage(ctx context.Context, ref ResolvedRef, progress ProgressFunc) (EnsureResult, error)

EnsureImage ensures an image is available locally. For Docker refs, pulls + converts + writes OCI blobs + materializes ext4 cache, then advances the tag. For local-path refs, returns the path directly (legacy escape hatch).

func (*Manager) ImageHistory 

func (m *Manager) ImageHistory(tagOrDigest string) ([]LayerInfo, error)

ImageHistory loads the manifest + config for tagOrDigest and returns per-layer history in display order (layer[0] = base, layer[N-1] = top).

func (*Manager) InspectImage 

func (m *Manager) InspectImage(tagOrDigest string) (*ImageInfo, *OCIManifest, error)

InspectImage returns full details for a tag or digest.

func (*Manager) ListImages 

func (m *Manager) ListImages() ([]ImageInfo, error)

ListImages returns ImageInfo entries for every known tag plus dangling blobs (installed manifests not referenced by any tag). UniqueBytes and SharedBytes are computed by attributing each layer's on-disk cost to its referencing manifests.

func (*Manager) LoadImage 

func (m *Manager) LoadImage(in io.Reader) ([]string, error)

LoadImage reads a tar produced by SaveImage (or any compatible OCI image-layout-tar) and ingests it into the local store. Returns the list of manifest digests added.

Behavior:

  • Every blob in the tar is content-verified against its filename digest before being written.
  • Manifests listed in the input index.json are recorded in the local index. ref-name annotations are translated to local tags (if not already taken).
  • Layer ext4 files are NOT materialized — call EnsureImage afterwards to populate the cache.

func (*Manager) PruneImages 

func (m *Manager) PruneImages(dryRun bool) ([]ImageInfo, error)

PruneImages removes blobs unreferenced by any shed/snapshot.

Reachability: a blob is "live" iff it is a manifest pinned by a shed or snapshot (via metadata.LowerDigest, which is the OCI manifest digest), or it is reachable from a live manifest (its config, layers, kernel, initrd, and any cached ext4 for those layers).

All other blobs are candidates for prune. Cached ext4 files for orphaned layers are also evicted. Dangling tag files (pointing at a no-longer-present manifest) are dropped.

func (*Manager) PullImage 

func (m *Manager) PullImage(ctx context.Context, dockerRef, tag, platform string, withLayers bool, progress ProgressFunc) (string, error)

PullImage pulls a registry reference straight to the OCI layout (no Docker daemon required) and advances the named tag. Defaults to the backend's native platform when platform is empty.

func (*Manager) PushImage 

func (m *Manager) PushImage(ctx context.Context, tagOrDigest, dstRef string, progress ProgressFunc) error

PushImage uploads the manifest currently held by tagOrDigest to a destination registry ref. Byte-perfect: the on-disk tar.gz layer blobs are streamed straight from the OCI store.

func (*Manager) ResolveImageBlobs 

func (m *Manager) ResolveImageBlobs(manifestDigest string) (manifest *OCIManifest, kernelPath, initrdPath string, err error)

ResolveImageBlobs returns the manifest and config for an installed image, plus its kernel/initrd blob paths if the manifest advertises them via shed annotations.

func (*Manager) ResolveManifestLower 

func (m *Manager) ResolveManifestLower(ctx context.Context, manifestDigest string) (string, error)

ResolveManifestLower returns the path of the single content-addressed lower-image (flattened, all-layers-merged erofs) for the manifest, materializing it from the layer blobs if it isn't already cached. Used by VM start to attach one read-only block device.

func (*Manager) SaveImage 

func (m *Manager) SaveImage(tagOrDigest string, out io.Writer) error

SaveImage writes the OCI subtree reachable from tagOrDigest as a tar stream to out. The stream contains: 

oci-layout
index.json          (one entry pointing at the saved manifest)
blobs/sha256/<hex>  for manifest, config, every layer, and kernel/initrd

Foreign tools that accept "OCI image layout in a tar" (crane, skopeo, oras) can consume the stream directly.

func (*Manager) TagImage 

func (m *Manager) TagImage(srcTagOrDigest, newTag string) error

TagImage points a new tag at the manifest digest currently held by srcTagOrDigest. Overwrites newTag if it already exists.

type MintErofsOptions 

type MintErofsOptions struct {
	// ImagesDir is the OCI store the layer blobs are already
	// installed in and where the new erofs blob will be written.
	ImagesDir string

	// LayerDigests are the OCI manifest's layer digests in OCI order
	// (lowest at index 0). MergeLayers flattens them into a single
	// tarball before feeding mkfs.erofs.
	LayerDigests []string

	// BuildToolsRef is the OCI image carrying mkfs.erofs that will
	// run inside `docker run --rm ...`. Pin to the same tag as the
	// shed-server version being released — see
	// docs/reference/build-tools.md for the versioning model.
	BuildToolsRef string

	// DockerBinary defaults to "docker"; tests / unusual hosts
	// override (e.g. "podman" with a compat wrapper).
	DockerBinary string
}

MintErofsOptions controls what MintRootfsErofs invokes on the publishing host.

type OCIConfig 

type OCIConfig struct {
	Architecture string          `json:"architecture"` // "arm64" or "amd64"
	OS           string          `json:"os"`           // "linux"
	Created      string          `json:"created,omitempty"`
	Author       string          `json:"author,omitempty"`
	RootFS       OCIRootFS       `json:"rootfs"`
	History      []OCIHistory    `json:"history,omitempty"`
	Config       OCIConfigConfig `json:"config,omitempty"`
}

OCIConfig is the OCI image config. Shed populates only the fields it uses; the rest are omitted via omitempty so foreign tools see a well-formed config with no surprises.

func LoadConfigByDigest 

func LoadConfigByDigest(imagesDir, configDigest string) (*OCIConfig, error)

LoadConfigByDigest reads + parses an OCI image config blob.

func ParseConfig 

func ParseConfig(data []byte) (*OCIConfig, error)

ParseConfig decodes an OCI image config blob.

func (*OCIConfig) MarshalIndent 

func (c *OCIConfig) MarshalIndent() ([]byte, error)

MarshalIndent emits OCI-compliant JSON with stable indentation.

type OCIConfigConfig 

type OCIConfigConfig struct {
	Env        []string `json:"Env,omitempty"`
	Entrypoint []string `json:"Entrypoint,omitempty"`
	Cmd        []string `json:"Cmd,omitempty"`
	WorkingDir string   `json:"WorkingDir,omitempty"`
}

OCIConfigConfig is the runtime config block. Shed doesn't run images like a container, so we leave most fields empty.

type OCIHistory 

type OCIHistory struct {
	Created    string `json:"created,omitempty"`
	CreatedBy  string `json:"created_by,omitempty"`
	Comment    string `json:"comment,omitempty"`
	EmptyLayer bool   `json:"empty_layer,omitempty"`
}

OCIHistory is the per-layer history entry, optional but conventional.

type OCIIndex 

type OCIIndex struct {
	SchemaVersion int               `json:"schemaVersion"` // always 2
	MediaType     string            `json:"mediaType,omitempty"`
	Manifests     []Descriptor      `json:"manifests"`
	Annotations   map[string]string `json:"annotations,omitempty"`
}

OCIIndex is the top-level image index stored at {imagesDir}/index.json. Shed lists every installed manifest here with a ref-name annotation (org.opencontainers.image.ref.name) so foreign tools can enumerate the store with `crane catalog dir:<imagesDir>`.

func ParseIndex 

func ParseIndex(data []byte) (*OCIIndex, error)

ParseIndex decodes an OCI index blob.

func ReadIndex 

func ReadIndex(imagesDir string) (*OCIIndex, error)

ReadIndex returns the parsed top-level OCI index.

func (*OCIIndex) MarshalIndent 

func (i *OCIIndex) MarshalIndent() ([]byte, error)

MarshalIndent emits OCI-compliant JSON with stable indentation.

type OCILayoutMarker 

type OCILayoutMarker struct {
	ImageLayoutVersion string `json:"imageLayoutVersion"`
}

OCILayoutMarker is the content of the {imagesDir}/oci-layout file.

type OCIManifest 

type OCIManifest struct {
	SchemaVersion int               `json:"schemaVersion"` // always 2
	MediaType     string            `json:"mediaType,omitempty"`
	Config        Descriptor        `json:"config"`
	Layers        []Descriptor      `json:"layers"`
	Annotations   map[string]string `json:"annotations,omitempty"`
}

OCIManifest is an OCI image manifest. The shed-on-disk shape is the standard OCI shape; consumers can inspect it via `crane manifest`.

func LoadManifestByDigest 

func LoadManifestByDigest(imagesDir, manifestDigest string) (*OCIManifest, error)

LoadManifestByDigest reads + parses an OCI manifest blob.

func ParseManifest 

func ParseManifest(data []byte) (*OCIManifest, error)

ParseManifest decodes an OCI manifest blob.

func (*OCIManifest) MarshalIndent 

func (m *OCIManifest) MarshalIndent() ([]byte, error)

MarshalIndent emits OCI-compliant JSON with stable indentation.

func (*OCIManifest) ShedInitrdDigest 

func (m *OCIManifest) ShedInitrdDigest() string

ShedInitrdDigest returns the initrd blob digest carried in annotations, or "" if no initrd is associated with this image.

func (*OCIManifest) ShedKernelDigest 

func (m *OCIManifest) ShedKernelDigest() string

ShedKernelDigest returns the kernel blob digest carried in annotations, or "" if no kernel is associated with this image.

func (*OCIManifest) ShedRootfsErofsDigest 

func (m *OCIManifest) ShedRootfsErofsDigest() string

ShedRootfsErofsDigest returns the prebuilt rootfs erofs blob digest carried in annotations, or "" if the image was built before v0.5.2 (when this annotation was introduced). Callers that depend on the blob being present should fail with a clear "image too old" error rather than fall back to local mkfs.erofs — see manager.go.

func (*OCIManifest) ShedSourceRef 

func (m *OCIManifest) ShedSourceRef() string

ShedSourceRef returns the manifest's recorded source ref annotation, or "" if unset.

func (*OCIManifest) ShedVariant 

func (m *OCIManifest) ShedVariant() string

ShedVariant returns the manifest's recorded variant annotation, or "" if unset.

type OCIRootFS 

type OCIRootFS struct {
	Type    string   `json:"type"`     // "layers"
	DiffIDs []string `json:"diff_ids"` // sha256 digests of uncompressed layer tars
}

OCIRootFS is the rootfs section of the OCI image config.

type ProgressEvent added in v0.6.2 

type ProgressEvent struct {
	Stage   string // phase/stage label (e.g. "image"); advances the PhaseTimer
	Message string // human-readable status line; always set for emitted events

	// Structured per-blob byte progress. Zero on plain status events.
	Kind    string // "" plain status; "blob" per-blob byte progress
	ID      string // full digest (blob events); renderer keys on this
	Status  string // BlobStatus* (blob events)
	Current int64  // bytes fetched so far (blob events)
	Total   int64  // total bytes — compressed descriptor size (blob events)
}

ProgressEvent is a progress update from a long-running image operation (pull / push / ensure). Stage + Message are the plain status fields used since the beginning. The Kind=="blob" fields carry structured per-blob byte progress for the `shed` CLI's live renderer; line-mode and older clients ignore them and render Message.

Backends translate this to backend.ProgressEvent at the API boundary (vmimage must not import backend), so the wire shape is owned by the backend package — this struct is the in-process carrier.

func (ProgressEvent) IsBlob added in v0.6.2 

func (e ProgressEvent) IsBlob() bool

IsBlob reports whether e carries structured per-blob byte progress. Plain line-oriented consumers (logs, the bulk pull-images command) use this to skip byte-tick events they would otherwise spam.

type ProgressFunc 

type ProgressFunc func(ProgressEvent)

ProgressFunc receives progress events during long-running operations.

type PullOptions 

type PullOptions struct {
	// Ref is the registry reference (e.g. "ghcr.io/charliek/shed-vz-full:v1").
	Ref string

	// ImagesDir is the OCI image-layout root that receives the blobs.
	ImagesDir string

	// TagName is the shed tag to advance to the new manifest digest.
	// If empty, no tag is set — useful for previewing pulls.
	TagName string

	// Platform selects the per-platform manifest from a multi-arch
	// image index. Defaults to the backend's native platform via
	// the caller; pass an empty string to use go-containerregistry's
	// default-platform resolver.
	Platform string

	// Insecure permits plain-HTTP transport (typical for local test
	// registries like `registry:2` on localhost:5000).
	Insecure bool

	// AuthKeychain is the credential helper chain. nil falls back to
	// authn.DefaultKeychain which reads ~/.docker/config.json.
	AuthKeychain authn.Keychain

	// ExtractKernel, when true, asks the puller to extract a kernel
	// from the layer rootfs if the manifest has no
	// io.shed.kernel.digest annotation. ExtractKernel + a manifest
	// without the annotation triggers an in-Go tar walk that fishes
	// out /boot/vmlinuz-* or /boot/vmlinux.
	ExtractKernel bool

	// NeedsInitrd is the initrd analog of ExtractKernel.
	NeedsInitrd bool

	// Progress receives stage messages during the pull.
	Progress ProgressFunc

	// Concurrency caps how many blobs download in parallel. <=1 means
	// serial. Callers should pass cfg.GetPullConcurrency().
	Concurrency int

	// SkipLayers pulls "boot-only": config + kernel + initrd + erofs +
	// manifest, but NOT the layer tarballs (the host boots from the erofs
	// and never reads layers). Requires a v0.5.2+ image whose kernel/initrd/
	// erofs come from manifest annotations; rejected otherwise. The skipped
	// layers are re-fetchable later via a full pull (`--with-layers`), which
	// is required before `shed image push` of a boot-only image.
	SkipLayers bool
}

PullOptions configures a registry-direct image pull.

type PullPolicy added in v0.6.0 

type PullPolicy string

PullPolicy controls how EnsureImage reconciles a configured Docker ref against the local content-addressed store at create time. It mirrors the Docker/Podman vocabulary so existing operator intuition transfers. 

const (
	// PullMissing uses the cached ref if present, pulling only when absent.
	// This is the default and keeps the create hot path O(1) and offline.
	PullMissing PullPolicy = "missing"
	// PullAlways always contacts the registry and pulls, bypassing the cache.
	PullAlways PullPolicy = "always"
	// PullNever uses the cached ref if present and errors on a miss; it never
	// contacts the registry.
	PullNever PullPolicy = "never"
)

func ParsePullPolicy added in v0.6.0 

func ParsePullPolicy(s string) (PullPolicy, error)

ParsePullPolicy validates a configured pull_policy string. An empty value defaults to PullMissing.

type PullResult 

type PullResult struct {
	ManifestDigest string
	ConfigDigest   string
	LayerDigests   []string
	KernelDigest   string
	InitrdDigest   string
}

PullResult mirrors ConvertResult so callers can use either flow.

func PullToOCILayout 

func PullToOCILayout(ctx context.Context, opts PullOptions) (*PullResult, error)

PullToOCILayout fetches an image from a registry into the OCI image layout under opts.ImagesDir. The pulled manifest's digest is recorded; each layer + config blob is written under blobs/sha256/<digest>. If the manifest carries shed annotations for kernel/initrd, those blobs are also fetched. Otherwise — and when opts.ExtractKernel is set — the kernel/initrd are pulled out of the rootfs layer in Go.

After all blobs are written, the named tag (if any) is advanced and each layer is materialized into the derived ext4 cache.

type PushOptions 

type PushOptions struct {
	// Ref is the destination registry reference.
	Ref string

	// ImagesDir is the source OCI layout.
	ImagesDir string

	// ManifestDigest is the source manifest digest to push (typically
	// resolved from a tag by the caller).
	ManifestDigest string

	// Insecure permits plain-HTTP transport. Auto-detected for
	// loopback hosts when left at its zero value (caller can also
	// set explicitly).
	Insecure bool

	// AuthKeychain controls credential lookup. nil → DefaultKeychain.
	AuthKeychain authn.Keychain

	// Progress is invoked with stage updates.
	Progress ProgressFunc
}

PushOptions configures a registry-direct image push.

type RefKind 

type RefKind string

RefKind classifies a reference protecting a blob from prune. 

const (
	// RefKindShed indicates a reference held by an existing shed instance.
	RefKindShed RefKind = "shed"

	// RefKindSnapshot indicates a reference held by a snapshot.
	RefKindSnapshot RefKind = "snapshot"

	// RefKindTag indicates a reference held by a tag. Tags ARE
	// protective from prune (changed in v0.5.8 — pre-v0.5.8 the prune
	// walker followed Docker's "tags don't protect" model, which made
	// `shed image pull X && shed image prune` delete the manifest
	// blob just pulled if no shed pinned it yet, leaving operators
	// with a tag pointing at a missing blob or, worse, silently
	// reverting to a stale locally-cached manifest). To delete a
	// blob a tag points at, the workflow is `shed image rm <tag>`
	// followed by `shed image prune`. See
	// docs/upgrades/v0.5.7-to-v0.5.8.md.
	RefKindTag RefKind = "tag"

	// RefKindPending indicates a reference held by an in-flight
	// `shed create`. The backend writes a `.creating` marker into the
	// instance directory between EnsureImage and meta.Save; the
	// refscanner emits a Pending ref for each fresh marker so prune
	// can't delete the blob the create is about to depend on. Fresh
	// for 1 h (see systemprune.InstanceCreatingMaxAge); stale markers
	// produce no reference and the underlying blob becomes prunable.
	RefKindPending RefKind = "pending-create"
)

type RefScanner 

type RefScanner interface {
	ScanRefs(strict bool) ([]Reference, error)
}

RefScanner scans the on-disk layout for references that point at blobs. Implementations live in the backend packages where the appropriate metadata directories are reachable.

Implementations MUST return shed and snapshot references (the protective refs that block prune). Tag refs are scanned by the Manager itself.

The `strict` parameter selects how malformed instance metadata is handled:

  • strict=false (read paths — ListImages, InspectImage, DiskUsage): skip the broken instance with a warning, return the rest. A `shed list` or `shed system df` shouldn't fail because one instance has a corrupt JSON file somewhere.
  • strict=true (PruneImages): fail closed. Returning a partial ref set from a destructive caller risks deleting a blob the broken-but-recoverable shed still pinned. The caller surfaces the error to the operator and tells them to fix or `rm -rf` the broken instance before pruning.

Sentinels for "the listing itself succeeded but a known-broken instance was skipped" are intentionally not part of the interface — implementations log a warning. Callers that want to inspect the skipped set should walk instances/* themselves.

type Reference 

type Reference struct {
	// Digest is the blob digest the reference points at.
	Digest string

	// Kind is the kind of reference (shed, snapshot, tag).
	Kind RefKind

	// Name is the name of the referencing object — shed name, snapshot
	// name, or tag name. Used for human-readable error messages.
	Name string
}

Reference describes one thing that points at a blob digest.

func ProtectiveRefs 

func ProtectiveRefs(refs []Reference, digest string) []Reference

ProtectiveRefs reports whether a digest has any protective reference — Shed, Snapshot, Pending (in-flight create), or Tag.

As of v0.5.8 tag references are protective: see RefKindTag's doc for the rationale. RefScanner implementations still don't emit tag refs (tags live in the central store the Manager owns, not in per-backend metadata), so the Manager merges them into the ref list at the caller sites that need protection (notably PruneImages).

type ResolvedRef 

type ResolvedRef struct {
	Path      string // set when the ext4 image already exists on disk
	DockerRef string // set when the image needs to be pulled and converted
	Name      string // cosmetic label derived from the ref; not an identity key
	Digest    string // set when Path came from a tag in the blob store; preserved through to EnsureResult.Digest

	// Policy governs cache-vs-pull for a DockerRef. Empty means PullMissing.
	// Ignored for local-path refs.
	Policy PullPolicy
}

ResolvedRef describes an image to ensure: either a local path or a Docker ref to pull.

type Tag 

type Tag struct {
	Digest    string    `json:"digest"`
	UpdatedAt time.Time `json:"updated_at"`
}

Tag represents a named pointer to an OCI manifest digest. Stored at tags/<name>.json.

func GetTag 

func GetTag(imagesDir, tag string) (Tag, error)

GetTag returns the digest a tag currently points at. Returns ErrTagNotFound if the tag file is missing.

Source Files

View all Source files

Directories

Path Synopsis
Package clone implements the fast-path file-copy strategy chain used by CopyRootfs.
 Package clone implements the fast-path file-copy strategy chain used by CopyRootfs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.