blob

README

go-oci-blob

go-oci-blob is a Go library that uploads and downloads OCI blobs. It covers the blob subset of the OCI distribution spec and nothing else: push, pull, existence checks, and cross-repository mounts, with retries and digest verification built in.

go get github.com/imgoci/go-oci-blob

client := blob.New(blob.WithTransport(authenticatedTransport))
repo := blob.Repository{Host: "registry.example.com", Name: "myorg/myrepo"}

dgst := digest.FromBytes(data)
err := client.Push(ctx, repo, dgst, int64(len(data)), bytes.NewReader(data))

Documentation:

• Getting started tutorial
• API reference
• Registry compatibility — verified behavior across nine registries
• Design

Design constraints, in short:

• Runtime dependencies are the Go standard library plus github.com/opencontainers/go-digest.
• Authentication is the caller's job: inject an authenticated registry http.RoundTripper, for example from oras-go or go-containerregistry. Off-origin storage and CDN requests use a separate transport with registry credentials removed, so they do not follow an absolute upload location.
• Defaults use the code paths every registry serves correctly. Chunked upload and parallel pull exist behind explicit toggles.

Development

mise provisions every pinned tool from mise.toml and mise.lock: Go, Moon, Python and uv (for the docs site), and golangci-lint. Run mise install once; there is nothing else to install by hand.

mise install runs with locked = true, so it fails closed if a tool lacks a pre-resolved, checksummed entry for the current platform. To bump a tool, edit its version in mise.toml, run mise lock --platform linux-x64,linux-arm64,macos-x64,macos-arm64, and commit mise.toml and mise.lock together.

Moon is the task front door:

moon run root:format
moon run root:lint
moon run root:build
moon run root:test
moon run root:check

CI runs the same aggregate check with moon ci --summary minimal.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Documentation

Overview

Package blob uploads and downloads OCI blobs.

The library covers exactly the blob subset of the OCI distribution spec: push, pull, existence checks, and cross-repository mounts. It does not touch manifests, tags, or authentication; callers inject an authenticated http.RoundTripper.

The design is documented in docs/docs/explanation/design.md.

Index

• Variables
• type Client
  • func New(opts ...Option) *Client
  • func (c *Client) Exists(ctx context.Context, repo Repository, dgst digest.Digest) (bool, error)
  • func (c *Client) Mount(ctx context.Context, dst, src Repository, dgst digest.Digest) (bool, error)
  • func (c *Client) Pull(ctx context.Context, repo Repository, dgst digest.Digest, ...) (io.ReadCloser, error)
  • func (c *Client) PullRange(ctx context.Context, repo Repository, dgst digest.Digest, offset, length int64, ...) (io.ReadCloser, error)
  • func (c *Client) Push(ctx context.Context, repo Repository, dgst digest.Digest, size int64, ...) error
• type Option
  • func WithChunkedUpload(chunkSize int64) Option
  • func WithParallelPull(workers int, chunkSize int64) Option
  • func WithPlainHTTP(plain bool) Option
  • func WithRetryPolicy(policy RetryPolicy) Option
  • func WithStorageTransport(rt http.RoundTripper) Option
  • func WithTransport(rt http.RoundTripper) Option
• type Repository
  • func (r Repository) Validate() error
• type RetryPolicy
  • func DefaultRetryPolicy() RetryPolicy
• type TransferOption
  • func WithProgress(fn func(done, total int64)) TransferOption

Variables

var ErrDigestMismatch = errors.New("digest mismatch")

ErrDigestMismatch reports that transferred bytes did not hash to the expected digest. The verifying reader returned by Client.Pull yields it in place of io.EOF when the stream ends on content that fails verification.

var ErrNotFound = errors.New("blob not found")

ErrNotFound reports that the requested blob or repository does not exist on the registry. Operations that treat absence as a normal answer, such as Client.Exists, translate it instead of returning it.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client transfers blobs to and from OCI registries. It is safe for concurrent use. Create one with New.

func New

func New(opts ...Option) *Client

New builds a Client from the given options.

With no options the Client uses http.DefaultTransport, speaks https, and retries transient failures per DefaultRetryPolicy. Authentication is the caller's job: inject a transport that attaches credentials with WithTransport.

Example:

client := blob.New(blob.WithTransport(authTransport))
ok, err := client.Exists(ctx, repo, dgst)

func (*Client) Exists

func (c *Client) Exists(ctx context.Context, repo Repository, dgst digest.Digest) (bool, error)

Exists reports whether the repository holds a blob with the given digest.

Exists issues a HEAD request to the blob endpoint. A 200 response reports true. A 404 reports false with a nil error: absence is a normal answer here, not a failure. Every other outcome is an error.

Example:

ok, err := client.Exists(ctx, blob.Repository{
	Host: "localhost:5000",
	Name: "library/ubuntu",
}, dgst)

func (*Client) Mount

func (c *Client) Mount(
	ctx context.Context, dst, src Repository, dgst digest.Digest,
) (bool, error)

Mount asks the registry to mount a blob from the src repository into dst without moving bytes, via POST /v2/<dst>/blobs/uploads/?mount=<digest>&from=<src>.

A 201 Created reports a successful mount. A 202 Accepted means the registry declined and opened a regular upload session instead; Mount cancels that unused session and reports (false, nil), leaving the caller to decide whether to Push. A failed cancellation is returned as an error because otherwise the unused session would be leaked. Both repositories must live on the same registry host.

func (*Client) Pull

func (c *Client) Pull(
	ctx context.Context, repo Repository, dgst digest.Digest, opts ...TransferOption,
) (io.ReadCloser, error)

Pull downloads a blob and returns a reader over its bytes.

The reader verifies content as it flows: when the stream ends, the final Read returns io.EOF only if the bytes hashed to dgst, and ErrDigestMismatch otherwise. Nothing is buffered; the blob streams straight from the registry (or from the blob storage the registry redirects to). Close the reader when done. A missing blob is an error matching ErrNotFound.

A stream that breaks mid-body resumes under the client's RetryPolicy with a ranged request from the last delivered byte; digest verification carries across the resume, so no byte is hashed twice. With WithParallelPull the blob arrives via concurrent ranged fetches instead, emitted in order through the same verifying reader, falling back to a single stream when the registry does not serve ranges.

Example:

rc, err := client.Pull(ctx, repo, dgst)
if err != nil {
	return err
}
defer rc.Close()
if _, err := io.Copy(dst, rc); err != nil {
	return err // includes ErrDigestMismatch on corrupt content
}

func (*Client) PullRange

func (c *Client) PullRange(
	ctx context.Context,
	repo Repository,
	dgst digest.Digest,
	offset, length int64,
	opts ...TransferOption,
) (io.ReadCloser, error)

PullRange downloads length bytes of a blob starting at offset.

The returned reader is deliberately unverified: the digest covers the whole blob, so a partial body cannot be checked against it. Callers that need integrity on partial reads build it above the library. When the registry ignores the Range header and answers with the whole blob, PullRange discards the leading offset bytes and caps the reader at length, so the reader serves the requested window either way. Shorter valid portions are supported for up to 16 successful partial responses; further fragmentation stops with an error instead of issuing unbounded requests. A window that starts at or past the end of the blob is an error.

func (*Client) Push

func (c *Client) Push(
	ctx context.Context,
	repo Repository,
	dgst digest.Digest,
	size int64,
	r io.Reader,
	opts ...TransferOption,
) error

Push uploads a blob monolithically: one POST to open an upload session, one PUT to send the bytes and commit them under dgst. With WithChunkedUpload the bytes travel in verified PATCH chunks instead; see that option for why chunked stays opt-in.

The size is mandatory and must match the number of bytes r yields; registries need it as Content-Length, and there is no unknown-length upload. A caller that does not know the size spools the data first and comes back with a number. The reader must reach EOF immediately after size bytes; a streaming producer must close its pipe so Push can prove there is no trailing data. The registry rejects the commit when the content does not hash to dgst, so a corrupt stream cannot be stored silently.

Push does not access r after returning. To preserve that ownership rule, it waits for every in-flight r.Read call to finish even after an early registry response or context cancellation. A reader that can block must arrange for its producer to unblock it when ctx ends; an arbitrary io.Reader has no general cancellation operation.

A failed upload restarts from byte zero under the client's RetryPolicy. Restarting needs the reader's bytes again, so r must implement io.Seeker (as bytes.Reader, strings.Reader, and os.File do); with a non-seekable reader the upload fails on the first transient error instead of retrying.

Example:

dgst := digest.FromBytes(data)
err := client.Push(ctx, repo, dgst, int64(len(data)), bytes.NewReader(data))

type Option

type Option func(*options)

Option configures a Client built by New.

func WithChunkedUpload

func WithChunkedUpload(chunkSize int64) Option

WithChunkedUpload switches Push from the default monolithic upload to chunked PATCH uploads of chunkSize bytes. Values below one are ignored and leave monolithic upload in place.

Chunked upload is spec-optional and broken on major hosted registries (ECR discards chunks after the first and still reports success), because mainstream clients never exercise it. It is an explicit opt-in, never a fallback: leave it off unless you have verified your registry against it. The client checks the registry's Range acknowledgement after every chunk and abandons the upload rather than store a blob silently missing bytes.

func WithParallelPull

func WithParallelPull(workers int, chunkSize int64) Option

WithParallelPull switches Pull to fetch blobs with workers concurrent ranged requests of chunkSize bytes each. Values below one for either parameter are ignored and leave single-stream pull in place. Worker counts above 1,024 and configurations whose workers × chunkSize memory bound cannot be represented by an int64 are likewise ignored.

Pull's contract does not change: chunks are emitted in order through the same digest-verifying reader. Memory use is bounded by roughly workers × chunkSize — the library's one deliberate exception to never buffering — and the caller sets that bound with these two parameters. When the registry does not serve ranges, Pull quietly falls back to a single stream: the toggle states intent, not a requirement. The library sizes its default HTTP/1 idle pool for the requested worker count. Caller-supplied transports are not modified; configure their connection pools for the intended concurrency.

func WithPlainHTTP

func WithPlainHTTP(plain bool) Option

WithPlainHTTP selects plain http:// registry URLs instead of https. Meant for local registries served without TLS; leave it off for anything reachable from the internet.

func WithRetryPolicy

func WithRetryPolicy(policy RetryPolicy) Option

WithRetryPolicy replaces the client's retry policy. Pass a zero RetryPolicy to disable retries entirely.

func WithStorageTransport

func WithStorageTransport(rt http.RoundTripper) Option

WithStorageTransport sets the http.RoundTripper used for off-origin storage and CDN requests. It receives requests after registry credentials, cookies, proxy credentials, and referrer data have been removed; use it for storage-specific TLS, proxy, or authentication behavior. A nil or typed-nil transport keeps the library-managed default transport.

func WithTransport

func WithTransport(rt http.RoundTripper) Option

WithTransport sets the http.RoundTripper used for requests to the registry origin. This is the seam where registry authentication is injected: pass an authenticated transport from a library such as oras-go or go-containerregistry. Off-origin redirects and absolute upload locations do not pass through this transport. A nil or typed-nil transport keeps the library-managed default transport.

type Repository

type Repository struct {
	// Host is the registry host with an optional port, such as
	// "registry.example.com" or "localhost:5000". It carries no scheme;
	// the Client decides between https and plain http.
	Host string

	// Name is the repository path within the registry, such as
	// "library/ubuntu". It must match the OCI distribution spec's
	// repository name grammar.
	Name string
}

Repository addresses a blob store: a registry host plus a repository name within it.

func (Repository) Validate

func (r Repository) Validate() error

Validate reports whether the Repository can address a registry.

The host must be non-empty and free of a scheme or path. The name must match the OCI distribution spec grammar: slash-separated components of lowercase alphanumerics joined by ".", "_", "__", or one or more "-".

type RetryPolicy

type RetryPolicy struct {
	// MaxAttempts is the total number of tries for one operation,
	// including the first. Values below one behave as one.
	MaxAttempts int

	// InitialDelay seeds the exponential backoff: attempt n waits a
	// full-jittered duration in [0, InitialDelay * 2^(n-1)].
	InitialDelay time.Duration

	// MaxDelay caps every wait, including waits requested by a
	// registry's Retry-After header.
	MaxDelay time.Duration
}

RetryPolicy bounds how the client retries failed requests.

The zero value means a single attempt with no retries. Retries trigger on connection errors, request timeouts, 429, and 5xx; other 4xx statuses mean the request is wrong, not unlucky, and are never retried. The caller's context bounds retry scheduling: a canceled context stops new attempts and backoff immediately.

func DefaultRetryPolicy

func DefaultRetryPolicy() RetryPolicy

DefaultRetryPolicy returns the policy New applies when WithRetryPolicy is not given: four total attempts, backoff seeded at 250ms, and no wait longer than 30 seconds.

type TransferOption

type TransferOption func(*transferConfig)

TransferOption adjusts a single byte-moving call: Pull, PullRange, or Push. Settings that apply to every call belong on the Client instead.

func WithProgress

func WithProgress(fn func(done, total int64)) TransferOption

WithProgress reports transfer progress to fn.

fn receives the cumulative number of bytes moved so far and the total (-1 when the total is unknown). Pull counts bytes delivered to the caller. Monolithic Push reports only after the final 201 response; chunked Push advances after each PATCH Range acknowledgement, so reaching total does not prove the final commit succeeded. Only a nil Push error does. Within one transfer, counts never move backward across retries or resumes, and calls to fn do not overlap, including during a parallel Pull. Concurrent transfers may call the same fn at the same time, so fn must protect any state shared between transfers. fn runs synchronously on the transfer path, so it must return quickly.