images

package
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2026 License: MIT Imports: 30 Imported by: 0

README

Image Manager

Converts OCI images to bootable erofs disks for Cloud Hypervisor VMs.

Architecture

OCI Registry → go-containerregistry → OCI Layout → umoci → rootfs/ → mkfs.erofs → disk.erofs

Design Decisions

Why go-containerregistry? (oci.go)

What: Pull OCI images from any registry (Docker Hub, ghcr.io, etc.)

Why:

  • Lightweight library from Google (used by ko, crane, etc.)
  • Works directly with registries (no daemon required)
  • Can propagate errors from registry (like 429)
  • Supports all registry authentication methods

Alternative: containers/image - has automatic retry logic that delays error reporting, can't fail fast for registry rate limits. Heavier, supporting more use cases in comparison to go-containerregistry.

Why umoci? (oci.go)

What: Unpack OCI image layers in userspace

Why:

  • Purpose-built for rootless OCI manipulation (official OpenContainers project)
  • Handles OCI layer semantics (whiteouts, layer ordering) correctly
  • Designed to work without root privileges

Alternative: With Docker API, the daemon (running as root) mounts image layers using overlayfs, then exports the merged filesystem. Users get the result without needing root themselves but it still has the dependency on Docker and does actually mount the overlays to get the merged filesystem. With umoci, layers are merged in userspace by extracting each tar layer sequentially and applying changes (including whiteouts for deletions). No kernel mount needed, fully rootless. Umoci was chosen because it's purpose-built for this use case and embeddable with the go program.

Why erofs? (disk.go)

What: erofs (Enhanced Read-Only File System) with LZ4 compression

Why:

  • Purpose-built for read-only overlay lowerdir
  • Fast compression (~20-25% space savings)
  • Fast decompression at VM boot
  • Lower memory footprint than ext4
  • No journal/inode overhead

Options:

  • -zlz4 - Fast compression

Alternative: ext4 without journal works but erofs is optimized for this exact use case

Filesystem Layout (storage.go, oci.go)

Content-addressable storage with tag symlinks (similar to Docker/Unikraft):

/var/lib/hypeman/
  images/
    docker.io/library/alpine/
      abc123def456.../      # Digest (sha256:abc123def456...)
        metadata.json       # Status, entrypoint, cmd, env
        rootfs.erofs        # Compressed read-only disk
      def456abc123.../      # Another version (digest)
        metadata.json
        rootfs.erofs
      latest -> abc123def456...   # Tag symlink to digest
      3.18 -> def456abc123...     # Another tag
  system/
    oci-cache/              # Shared OCI layout for all images
      index.json            # Manifest index with digest-based tags
      blobs/sha256/
        2d35eb...           # Layer blobs (shared across all images!)
        44cf07...           # Another layer
        706db5...           # Config blob for alpine
        abc123def456...     # Manifest for alpine:latest

Benefits:

  • Content-addressable: Digests are immutable, same content stored once
  • Tag mutability: Tags (symlinks) can point to different digests over time
  • Deduplication: Multiple tags can point to same digest
  • Natural hierarchy: All versions of an image grouped under repository
  • Easy inspection: Clear which digest belongs to which image
  • Layer caching: All images share the same blob storage, layers deduplicated automatically

Design:

  • Images stored by manifest digest (content hash)
  • Tags are filesystem symlinks pointing to digest directories
  • Manifest always inspected upfront to discover digest (validates existence)
  • Pulling same tag twice updates the symlink if digest changed
  • OCI cache uses digest hex as layout tag for true content-addressable caching
  • Shared blob storage enables automatic layer deduplication across all images
  • Orphaned digests are automatically deleted when the last tag referencing them is removed
  • Symlinks only created after successful build (status: ready)

Reference Handling (reference.go)

Two types for type-safe image reference handling:

NormalizedRef - Validated format (parsing only):

normalized, err := ParseNormalizedRef("alpine")
// Normalizes to "docker.io/library/alpine:latest"

ResolvedRef - Normalized + manifest digest (network call):

resolved, err := normalized.Resolve(ctx, ociClient)
// Now has digest from registry inspection

resolved.Repository()  // "docker.io/library/alpine"
resolved.Tag()         // "latest"
resolved.Digest()      // "sha256:abc123..." (always present)

Validation via github.com/distribution/reference:

  • alpine → docker.io/library/alpine:latest
  • alpine:3.18 → docker.io/library/alpine:3.18
  • alpine@sha256:abc123... → digest validated against registry
  • Rejects invalid formats (returns 400)

Build Tags

Requires -tags containers_image_openpgp for umoci dependency compatibility.

Registry Authentication

go-containerregistry automatically uses ~/.docker/config.json via authn.DefaultKeychain.

# Login to Docker Hub (avoid rate limits)
docker login

# Works for any registry
docker login ghcr.io

No code changes needed - credentials are automatically discovered.

Documentation

Index

Constants

View Source 
const (
	StatusPending    = "pending"
	StatusPulling    = "pulling"
	StatusConverting = "converting"
	StatusReady      = "ready"
	StatusFailed     = "failed"
)

Variables

View Source 
var (
	ErrNotFound    = errors.New("image not found")
	ErrInvalidName = errors.New("invalid image name")
)
View Source 
var DefaultImageFormat = func() ExportFormat {
	if runtime.GOOS == "darwin" {
		return FormatExt4
	}
	return FormatErofs
}()

DefaultImageFormat is the default export format for OCI images. On Linux, we use erofs (compressed, read-only) for smaller images. On Darwin, we use ext4 because the VZ kernel doesn't have erofs support.

Functions

func CreateEmptyExt4Disk 

func CreateEmptyExt4Disk(diskPath string, sizeBytes int64) error

CreateEmptyExt4Disk creates a sparse disk file and formats it as ext4. Used for volumes and instance overlays that need empty writable filesystems.

func ExportRootfs 

func ExportRootfs(rootfsDir, outputPath string, format ExportFormat) (int64, error)

ExportRootfs exports rootfs directory in specified format (public for system manager)

func GetDiskPath 

func GetDiskPath(p *paths.Paths, imageName string, digest string) (string, error)

GetDiskPath returns the filesystem path to an image's rootfs disk file (public for instances manager)

func IsSystemdImage 

func IsSystemdImage(entrypoint, cmd []string) bool

IsSystemdImage checks if the image's CMD indicates it wants systemd as init. Detection is based on the effective command (entrypoint + cmd), not whether systemd is installed in the image.

Returns true if the image's command is:

  • /sbin/init
  • /lib/systemd/systemd
  • /usr/lib/systemd/systemd

Types

type BuildQueue 

type BuildQueue struct {
	// contains filtered or unexported fields
}

BuildQueue manages concurrent image builds with a configurable limit

func NewBuildQueue 

func NewBuildQueue(maxConcurrent int) *BuildQueue

func (*BuildQueue) ActiveCount 

func (q *BuildQueue) ActiveCount() int

ActiveCount returns number of actively building images

func (*BuildQueue) Enqueue 

func (q *BuildQueue) Enqueue(imageName string, req CreateImageRequest, startFn func()) int

Enqueue adds a build to the queue. Returns queue position (0 if started immediately, >0 if queued). If the image is already building or queued, returns its current position without re-enqueueing.

func (*BuildQueue) GetPosition 

func (q *BuildQueue) GetPosition(imageName string) *int

func (*BuildQueue) MarkComplete 

func (q *BuildQueue) MarkComplete(imageName string)

func (*BuildQueue) PendingCount 

func (q *BuildQueue) PendingCount() int

PendingCount returns number of queued builds

func (*BuildQueue) QueueLength 

func (q *BuildQueue) QueueLength() int

QueueLength returns the total number of builds (active + pending)

type CreateImageRequest 

type CreateImageRequest struct {
	Name string
}

CreateImageRequest represents a request to create an image

type ExportFormat 

type ExportFormat string

ExportFormat defines supported rootfs export formats 

const (
	FormatExt4  ExportFormat = "ext4"  // Read-only ext4 (legacy, used on Darwin)
	FormatErofs ExportFormat = "erofs" // Read-only compressed with LZ4 (default on Linux)
	FormatCpio  ExportFormat = "cpio"  // Uncompressed archive (initrd, fast boot)
)

type Image 

type Image struct {
	Name          string // Normalized ref (e.g., docker.io/library/alpine:latest)
	Digest        string // Resolved manifest digest (sha256:...)
	Status        string
	QueuePosition *int
	Error         *string
	SizeBytes     *int64
	Entrypoint    []string
	Cmd           []string
	Env           map[string]string
	WorkingDir    string
	CreatedAt     time.Time
}

Image represents a container image converted to bootable disk

type Manager 

type Manager interface {
	ListImages(ctx context.Context) ([]Image, error)
	CreateImage(ctx context.Context, req CreateImageRequest) (*Image, error)
	// ImportLocalImage imports an image that was pushed to the local OCI cache.
	// Unlike CreateImage, it does not resolve from a remote registry.
	ImportLocalImage(ctx context.Context, repo, reference, digest string) (*Image, error)
	GetImage(ctx context.Context, name string) (*Image, error)
	DeleteImage(ctx context.Context, name string) error
	RecoverInterruptedBuilds()
	// TotalImageBytes returns the total size of all ready images on disk.
	// Used by the resource manager for disk capacity tracking.
	TotalImageBytes(ctx context.Context) (int64, error)
	// TotalOCICacheBytes returns the total size of the OCI layer cache.
	// Used by the resource manager for disk capacity tracking.
	TotalOCICacheBytes(ctx context.Context) (int64, error)
	// WaitForReady blocks until the image identified by name reaches a terminal
	// state (ready or failed) or the context is cancelled.
	WaitForReady(ctx context.Context, name string) error
}

func NewManager 

func NewManager(p *paths.Paths, maxConcurrentBuilds int, meter metric.Meter) (Manager, error)

NewManager creates a new image manager. If meter is nil, metrics are disabled.

type ManifestInspector 

type ManifestInspector interface {
	// contains filtered or unexported methods
}

Resolve inspects the manifest to get the digest and returns a ResolvedRef. This requires an ociClient interface for manifest inspection.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics holds the metrics instruments for image operations.

type MirrorRequest added in v0.0.6 

type MirrorRequest struct {
	// SourceImage is the full image reference to pull from (e.g., "docker.io/onkernel/nodejs22-base:0.1.1")
	SourceImage string
}

MirrorRequest contains the parameters for mirroring a base image

type MirrorResult added in v0.0.6 

type MirrorResult struct {
	// SourceImage is the original image reference
	SourceImage string `json:"source_image"`
	// LocalRef is the local registry reference (e.g., "onkernel/nodejs22-base:0.1.1")
	LocalRef string `json:"local_ref"`
	// Digest is the image digest
	Digest string `json:"digest"`
}

MirrorResult contains the result of a mirror operation

func MirrorBaseImage added in v0.0.6 

func MirrorBaseImage(ctx context.Context, registryURL string, req MirrorRequest, authConfig *authn.AuthConfig) (*MirrorResult, error)

MirrorBaseImage pulls an image from an external registry and pushes it to the local registry with the same normalized name. This enables Dockerfile FROM rewriting to use locally mirrored base images instead of pulling from Docker Hub.

For example, mirroring "docker.io/onkernel/nodejs22-base:0.1.1" will create "onkernel/nodejs22-base:0.1.1" in the local registry.

type NormalizedRef 

type NormalizedRef struct {
	// contains filtered or unexported fields
}

NormalizedRef is a validated and normalized OCI image reference. It can be either a tagged reference (e.g., "docker.io/library/alpine:latest") or a digest reference (e.g., "docker.io/library/alpine@sha256:abc123...").

func ParseNormalizedRef 

func ParseNormalizedRef(s string) (*NormalizedRef, error)

ParseNormalizedRef validates and normalizes a user-provided image reference. Examples:

  • "alpine" -> "docker.io/library/alpine:latest"
  • "alpine:3.18" -> "docker.io/library/alpine:3.18"
  • "alpine@sha256:abc..." -> "docker.io/library/alpine@sha256:abc..."

func (*NormalizedRef) Digest 

func (r *NormalizedRef) Digest() string

Digest returns the digest if present (e.g., "sha256:abc123..."). Returns empty string if this is a tagged reference.

func (*NormalizedRef) DigestHex 

func (r *NormalizedRef) DigestHex() string

DigestHex returns just the hex portion of the digest (without "sha256:" prefix). Returns empty string if this is a tagged reference.

func (*NormalizedRef) IsDigest 

func (r *NormalizedRef) IsDigest() bool

IsDigest returns true if this reference contains a digest (@sha256:...).

func (*NormalizedRef) Repository 

func (r *NormalizedRef) Repository() string

Repository returns the repository path without tag or digest. Example: "docker.io/library/alpine"

func (*NormalizedRef) Resolve 

func (r *NormalizedRef) Resolve(ctx context.Context, inspector ManifestInspector) (*ResolvedRef, error)

Resolve returns a ResolvedRef by inspecting the manifest to get the authoritative digest.

func (*NormalizedRef) String 

func (r *NormalizedRef) String() string

String returns the full normalized reference.

func (*NormalizedRef) Tag 

func (r *NormalizedRef) Tag() string

Tag returns the tag if this is a tagged reference (e.g., "latest"). Returns empty string if this is a digest reference.

type OCIClient 

type OCIClient struct {
	// contains filtered or unexported fields
}

OCIClient is a public wrapper for system manager to use OCI operations

func NewOCIClient 

func NewOCIClient(cacheDir string) (*OCIClient, error)

NewOCIClient creates a new OCI client (public for system manager)

func (*OCIClient) InspectManifest 

func (c *OCIClient) InspectManifest(ctx context.Context, imageRef string) (string, error)

InspectManifest inspects a remote image to get its digest (public for system manager). Always targets Linux platform since hypeman VMs are Linux guests.

func (*OCIClient) InspectManifestForLinux added in v0.0.6 

func (c *OCIClient) InspectManifestForLinux(ctx context.Context, imageRef string) (string, error)

InspectManifestForLinux is an alias for InspectManifest (all images target Linux)

func (*OCIClient) PullAndUnpack 

func (c *OCIClient) PullAndUnpack(ctx context.Context, imageRef, digest, exportDir string) error

PullAndUnpack pulls an OCI image and unpacks it to a directory (public for system manager). Always targets Linux platform since hypeman VMs are Linux guests.

func (*OCIClient) PullAndUnpackForLinux added in v0.0.6 

func (c *OCIClient) PullAndUnpackForLinux(ctx context.Context, imageRef, digest, exportDir string) error

PullAndUnpackForLinux is an alias for PullAndUnpack (all images target Linux)

type QueuedBuild 

type QueuedBuild struct {
	ImageName string
	Request   CreateImageRequest
	StartFn   func()
}

type ResolvedRef 

type ResolvedRef struct {
	// contains filtered or unexported fields
}

ResolvedRef is a NormalizedRef that has been resolved to include the actual manifest digest from the registry. The digest is always present.

func NewResolvedRef 

func NewResolvedRef(normalized *NormalizedRef, digest string) *ResolvedRef

NewResolvedRef creates a ResolvedRef from a NormalizedRef and digest.

func (*ResolvedRef) Digest 

func (r *ResolvedRef) Digest() string

Digest returns the resolved manifest digest (e.g., "sha256:abc123..."). This is always populated after resolution.

func (*ResolvedRef) DigestHex 

func (r *ResolvedRef) DigestHex() string

DigestHex returns just the hex portion of the digest (without "sha256:" prefix).

func (*ResolvedRef) Repository 

func (r *ResolvedRef) Repository() string

Repository returns the repository path without tag or digest. Example: "docker.io/library/alpine"

func (*ResolvedRef) String 

func (r *ResolvedRef) String() string

String returns the full normalized reference (the original user input format).

func (*ResolvedRef) Tag 

func (r *ResolvedRef) Tag() string

Tag returns the tag if this was originally a tagged reference (e.g., "latest"). Returns empty string if this was originally a digest reference.

type StatusEvent added in v0.0.7 

type StatusEvent struct {
	Status string
	Err    error
}

StatusEvent represents a terminal status change for image readiness notifications.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.