nixoci

README

nixoci

Push and pull a Nix store as an OCI image layer. nixoci serializes a populated /nix into a single layer, ships it to a registry like GHCR, and restores it on the other side. The point is caching: rebuilding a Nix environment from scratch takes minutes to hours, pulling a layer takes seconds.

It talks to registries through go-containerregistry directly rather than shelling out to crane. That started as a fix (crane's stdin handling re-gzipped pre-gzipped input and buffered to disk, which broke CI in creative ways) and ended up as the better design: deterministic encoding, true streaming, and one code path shared by tests and CI.

nixoci is used by devcell to cache container Nix volumes between CI runs. The API is Push, Pull, and ResolveImage; pin a version if you build on it.

go get github.com/devcell-sh/go-nixoci

Documentation

Overview

Package nixoci implements push/pull of a nix-store tarball as an OCI image layer, against a remote registry like GHCR.

Background: the CI workflow caches the populated `/nix` Docker volume between runs by serializing it as a single OCI layer. CELL-292 made this work via the `crane` CLI, but `crane append`'s stdin handler had behavior that caused six iterations of CI breakage (re-gzipping pre-gzipped input, buffering uncompressed stdin to disk, …). This package replaces those CLI calls with direct uses of `github.com/google/go-containerregistry/pkg/v1/{stream,tarball}`, giving deterministic encoding + true streaming + a single Go code path shared between local tests and CI.

Index

• Constants
• Variables
• func ParseSize(s string) (int64, error)
• func Pull(ctx context.Context, srcRef, dstDir string, stripComponents int) error
• func PullToDockerVolume(ctx context.Context, srcRef, volName string, stripComponents int) error
• func Push(ctx context.Context, baseRef, dstRef string, r io.ReadCloser) error
• func PushFromVolume(ctx context.Context, volume, baseRef, dstRef string, opts PushOpts) error
• func ResolveImage(ctx context.Context, candidates ...string) (string, error)
• func WithRetry(ctx context.Context, maxAttempts int, fn func() error) error
• type PushOpts

Constants

const (

	// MarkerDir, relative to the destination root, holds one empty file
	// per applied layer, named by the layer's compressed sha256 hex.
	MarkerDir = ".nixoci/layers"
	// ImageMarker, relative to the destination root, holds the manifest
	// digest of the last fully applied image.
	ImageMarker = ".nixoci/image"
)

Variables

var ChunkSize int64 = 512 << 20 // 512 MB

ChunkSize is the soft cap (in uncompressed tar bytes) at which Push starts a new OCI layer. Entries are never split mid-file — a single entry larger than ChunkSize produces a one-entry chunk. Set to 0 to disable chunking (single layer, original behavior).

var Debug bool

Debug enables verbose HTTP-level logging for registry operations. Set via DEVCELL_NIX_PUSH_DEBUG=1.

var MaxDownloadAttempts = 5

MaxDownloadAttempts is how many times a single layer is streamed and applied before the pull fails, mirroring dockerd's --max-download-attempts. Retry is per layer: a failed layer is re-streamed from byte 0; committed layers are never re-downloaded.

var ProgressTick = 5 * time.Second

ProgressTick controls how often Push emits an in-flight progress line. Set short in tests; CI uses the default.

var ProgressWriter io.Writer = os.Stderr

ProgressWriter receives periodic progress lines from Push. Defaults to os.Stderr so CI logs surface upload activity; tests can swap it for a buffer. Concurrent writes are serialized inside Push.

var RetryBaseDelay = 15 * time.Second

RetryBaseDelay is the base delay between retry attempts. Actual delay is attempt * RetryBaseDelay (linear backoff matching the Taskfile's sleep $((attempt * 15))). Tests set this short.

var StallTimeout = 120 * time.Second

StallTimeout is the maximum duration Push tolerates zero byte progress before aborting the upload. Set to 0 to disable. GHCR occasionally throttles multi-GB layer uploads to near-zero; without this the process blocks until externally SIGTERMed.

var TotalSizeHint int64

TotalSizeHint is the estimated uncompressed size (in bytes) of the tar being pushed. When > 0, progress lines show "sent X / Y" so the operator can gauge completion. Set from DEVCELL_NIX_TOTAL_SIZE.

var UploadJobs = 6

UploadJobs controls how many OCI layer blobs are uploaded concurrently by remote.Write. GHCR allows ~2k req/min with a 10-min per-layer timeout; 4-8 is the safe range, above 8 risks 429s.

Functions

func ParseSize

func ParseSize(s string) (int64, error)

ParseSize parses a human-readable size string (e.g. "1GB", "500MB", "1024KB") into bytes. Case-insensitive.

func Pull

func Pull(ctx context.Context, srcRef, dstDir string, stripComponents int) error

Pull resolves srcRef, downloads all non-base layers of the image, and extracts each gzipped tarball into dstDir. For multi-layer (chunked) images, layers[1:] are extracted in order; for single-layer images, layers[0] is extracted (backward compat with legacy cache images). Archive entries are written relative to dstDir, with stripComponents leading path elements stripped per `tar --strip-components=N` semantics. Symlinks and file modes are preserved.

Commit/resume contract: each layer is applied, its uncompressed stream verified against the config DiffID, then recorded as MarkerDir/<digest-hex>. Layers with an existing marker are skipped, so a failed pull is resumed by calling Pull again. ImageMarker is written last and only once every layer is committed.

func PullToDockerVolume

func PullToDockerVolume(ctx context.Context, srcRef, volName string, stripComponents int) error

PullToDockerVolume streams all non-base layers of srcRef into the named Docker volume by spawning `docker run -i alpine sh -c 'cd /dest && tar -x --strip-components=N'` for each layer and feeding the (gunzipped) tar stream over stdin. For multi-layer (chunked) images, layers[1:] are extracted in order; for single-layer images, layers[0] is extracted (backward compat).

stripComponents has the same meaning as Pull: leading path elements to strip from each archive entry.

Commit/resume contract matches Pull: MarkerDir/<digest-hex> is touched on the volume after a layer extracts and its DiffID verifies; marked layers are skipped on the next call; ImageMarker is written last. Marker writes are separate `docker run` invocations so a layer is never committed before Go has verified the stream it fed tar.

func Push

func Push(ctx context.Context, baseRef, dstRef string, r io.ReadCloser) error

Push streams an uncompressed tar from r through to dstRef as one or more OCI tar+gzip layers atop baseRef.

When ChunkSize > 0 (default 512 MB), the input tar is split on entry boundaries into ~ChunkSize temp files. All chunks are then uploaded in parallel (UploadJobs concurrent blob streams, default 4) via a single remote.Write call — one manifest write at the end. This eliminates per-chunk manifest churn and overlaps gzip+upload across layers. Temp files are removed after the push completes.

When ChunkSize <= 0, the original single-layer streaming behavior is preserved: bytes flow r → gzip → registry with no disk staging.

func PushFromVolume

func PushFromVolume(ctx context.Context, volume, baseRef, dstRef string, opts PushOpts) error

PushFromVolume pushes the contents of a Docker volume to a registry as OCI layers. Handles size checking, retry with backoff, and tag aliasing — replacing the ~48-line shell pipeline in Taskfile's nix-cache:publish task.

func ResolveImage

func ResolveImage(ctx context.Context, candidates ...string) (string, error)

ResolveImage probes each candidate image reference in order and returns the first one whose manifest exists in the registry. Returns "" if none resolve. Empty strings in candidates are skipped.

func WithRetry

func WithRetry(ctx context.Context, maxAttempts int, fn func() error) error

WithRetry calls fn up to maxAttempts times. On failure, waits attempt * RetryBaseDelay before the next attempt (linear backoff). Returns the last error if all attempts fail. Respects context cancellation between attempts.

Types

type PushOpts

type PushOpts struct {
	TagAlias   string // create alias tag after push via docker buildx imagetools create
	Retries    int    // total push attempts (0 or 1 = single attempt, no retry)
	MinSize    int64  // skip push if volume content < MinSize bytes (0 = always push)
	ChunkBytes int64  // uncompressed tar chunk size in bytes; 0 = use package default (512 MB)
	UploadJobs int    // concurrent blob upload streams; 0 = use package default (6)
}

PushOpts configures PushFromVolume behavior.