blob

package module
v1.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2026 License: Apache-2.0, MIT Imports: 17 Imported by: 0

README

go-oci-blob

go-oci-blob is a Go library that uploads and downloads OCI blobs. It covers the blob subset of the OCI distribution spec and nothing else: push, pull, existence checks, and cross-repository mounts, with retries and digest verification built in.

go get github.com/imgoci/go-oci-blob

client := blob.New(blob.WithTransport(authenticatedTransport))
repo := blob.Repository{Host: "registry.example.com", Name: "myorg/myrepo"}

dgst := digest.FromBytes(data)
err := client.Push(ctx, repo, dgst, int64(len(data)), bytes.NewReader(data))

Documentation:

Design constraints, in short:

  • Runtime dependencies are the Go standard library plus github.com/opencontainers/go-digest.
  • Authentication is the caller's job: inject an authenticated registry http.RoundTripper, for example from oras-go or go-containerregistry. Off-origin storage and CDN requests use a separate transport with registry credentials removed, so they do not follow an absolute upload location.
  • Defaults use the code paths every registry serves correctly. Chunked upload and parallel pull exist behind explicit toggles.

Development

mise provisions every pinned tool from mise.toml and mise.lock: Go, Moon, Python and uv (for the docs site), and golangci-lint. Run mise install once; there is nothing else to install by hand.

mise install runs with locked = true, so it fails closed if a tool lacks a pre-resolved, checksummed entry for the current platform. To bump a tool, edit its version in mise.toml, run mise lock --platform linux-x64,linux-arm64,macos-x64,macos-arm64, and commit mise.toml and mise.lock together.

Moon is the task front door:

moon run root:format
moon run root:lint
moon run root:build
moon run root:test
moon run root:check

CI runs the same aggregate check with moon ci --summary minimal.

License

Licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Documentation

Overview

Package blob transfers blobs to and from OCI registries.

The package covers the blob subset of the OCI distribution specification: push, pull, existence checks, and cross-repository mounts. It does not manage manifests, tags, authentication, credentials, or destination policy.

A Client routes registry-origin requests through the transport supplied to WithTransport. Callers must supply an authenticated transport when the registry requires credentials. Requests to registry-selected off-origin storage use WithStorageTransport instead. The caller remains responsible for deciding which storage destinations that transport may reach.

Embedding with an outer retry loop

RetryPolicy has a zero-value one-attempt mode. This lets an embedding orchestrator own its operation-level retry budget without nested retries: 

client := blob.New(
	blob.WithTransport(authenticatedRegistryTransport),
	blob.WithStorageTransport(guardedStorageTransport),
	blob.WithRetryPolicy(blob.RetryPolicy{}),
	blob.WithWriteRedirects(false),
)

err := client.Push(
	ctx, repo, dgst, size, body,
	blob.WithWireProgress(reportWireBytes),
)

After an operation fails, Retryable reports whether a fresh operation may succeed and returns the usable Retry-After floor, if one was supplied. StatusCode exposes a retained HTTP response status. Errors can also be inspected with errors.Is for ErrNotFound, ErrUnauthorized, ErrTooLarge, and ErrDigestMismatch. These inspection APIs survive contextual wrapping; callers do not need to parse rendered error text.

Progress

WithProgress reports cumulative committed transfer progress. WithWireProgress reports positive upload-byte deltas as the HTTP transport consumes request bodies, including bytes consumed by failed attempts, redirects, and transparent retries. Source read-ahead does not count as wire progress.

Redirects

By default, the client follows method-preserving write redirects to preserve v1 behavior. WithWriteRedirects can reject redirects that would reissue POST, PUT, PATCH, or DELETE. This option does not reject upload-session Location values returned by successful registry responses.

The design is documented in docs/docs/explanation/design.md.

Index

Constants

This section is empty.

Variables

View Source 
var ErrDigestMismatch = errors.New("digest mismatch")

ErrDigestMismatch reports that transferred bytes did not hash to the expected digest. The verifying reader returned by Client.Pull yields it in place of io.EOF when the stream ends on content that fails verification.

View Source 
var ErrNotFound = errors.New("blob not found")

ErrNotFound reports that the requested blob or repository does not exist on the registry. Operations that treat absence as a normal answer, such as Client.Exists, translate it instead of returning it.

View Source 
var ErrTooLarge = errors.New("blob too large")

ErrTooLarge reports that the registry origin refused a blob because its declared or transferred size exceeds a registry limit.

View Source 
var ErrUnauthorized = errors.New("unauthorized")

ErrUnauthorized reports that the registry origin rejected authorization. Authentication failures from off-origin storage do not match this error.

Functions

func Retryable added in v1.1.0 

func Retryable(err error) (time.Duration, bool)

Retryable reports whether a fresh operation may succeed after err.

When ok is true, after is the minimum delay requested by the peer through Retry-After, or zero when the peer requested no usable delay. The result survives contextual wrapping and retry-policy exhaustion, including the one-attempt policy selected by WithRetryPolicy(RetryPolicy{}).

Retryable describes a fresh operation by an embedding orchestrator. It does not change the client's built-in retry table or guarantee that the current request body can be replayed.

func StatusCode added in v1.1.0 

func StatusCode(err error) (int, bool)

StatusCode returns the HTTP response status retained by err.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client transfers blobs to and from OCI registries. It is safe for concurrent use. Create one with New.

func New 

func New(opts ...Option) *Client

New builds a Client from the given options.

With no options the Client uses http.DefaultTransport, speaks https, and retries transient failures per DefaultRetryPolicy. Authentication is the caller's job: inject a transport that attaches credentials with WithTransport.

Example: 

client := blob.New(blob.WithTransport(authTransport))
ok, err := client.Exists(ctx, repo, dgst)

func (*Client) Exists 

func (c *Client) Exists(ctx context.Context, repo Repository, dgst digest.Digest) (bool, error)

Exists reports whether the repository holds a blob with the given digest.

Exists issues a HEAD request to the blob endpoint. A 200 response reports true. A 404 reports false with a nil error: absence is a normal answer here, not a failure. Every other outcome is an error.

Example: 

ok, err := client.Exists(ctx, blob.Repository{
	Host: "localhost:5000",
	Name: "library/ubuntu",
}, dgst)

func (*Client) Mount 

func (c *Client) Mount(
	ctx context.Context, dst, src Repository, dgst digest.Digest,
) (bool, error)

Mount asks the registry to mount a blob from the src repository into dst without moving bytes, via POST /v2/<dst>/blobs/uploads/?mount=<digest>&from=<src>.

A 201 Created reports a successful mount. A 202 Accepted means the registry declined and opened a regular upload session instead; Mount cancels that unused session and reports (false, nil), leaving the caller to decide whether to Push. A failed cancellation is returned as an error because otherwise the unused session would be leaked. Both repositories must live on the same registry host.

func (*Client) Pull 

func (c *Client) Pull(
	ctx context.Context, repo Repository, dgst digest.Digest, opts ...TransferOption,
) (io.ReadCloser, error)

Pull downloads a blob and returns a reader over its bytes.

The reader verifies content as it flows: when the stream ends, the final Read returns io.EOF only if the bytes hashed to dgst, and ErrDigestMismatch otherwise. Nothing is buffered; the blob streams straight from the registry (or from the blob storage the registry redirects to). Close the reader when done. A missing blob is an error matching ErrNotFound.

A stream that breaks mid-body resumes under the client's RetryPolicy with a ranged request from the last delivered byte; digest verification carries across the resume, so no byte is hashed twice. With WithParallelPull the blob arrives via concurrent ranged fetches instead, emitted in order through the same verifying reader, falling back to a single stream when the registry does not serve ranges.

Example: 

rc, err := client.Pull(ctx, repo, dgst)
if err != nil {
	return err
}
defer rc.Close()
if _, err := io.Copy(dst, rc); err != nil {
	return err // includes ErrDigestMismatch on corrupt content
}

func (*Client) PullRange 

func (c *Client) PullRange(
	ctx context.Context,
	repo Repository,
	dgst digest.Digest,
	offset, length int64,
	opts ...TransferOption,
) (io.ReadCloser, error)

PullRange downloads length bytes of a blob starting at offset.

The returned reader is deliberately unverified: the digest covers the whole blob, so a partial body cannot be checked against it. Callers that need integrity on partial reads build it above the library. When the registry ignores the Range header and answers with the whole blob, PullRange discards the leading offset bytes and caps the reader at length, so the reader serves the requested window either way. Shorter valid portions are supported for up to 16 successful partial responses; further fragmentation stops with an error instead of issuing unbounded requests. A window that starts at or past the end of the blob is an error.

func (*Client) Push 

func (c *Client) Push(
	ctx context.Context,
	repo Repository,
	dgst digest.Digest,
	size int64,
	r io.Reader,
	opts ...TransferOption,
) error

Push uploads a blob monolithically: one POST to open an upload session, one PUT to send the bytes and commit them under dgst. With WithChunkedUpload the bytes travel in verified PATCH chunks instead; see that option for why chunked stays opt-in.

The size is mandatory and must match the number of bytes r yields; registries need it as Content-Length, and there is no unknown-length upload. A caller that does not know the size spools the data first and comes back with a number. The reader must reach EOF immediately after size bytes; a streaming producer must close its pipe so Push can prove there is no trailing data. The registry rejects the commit when the content does not hash to dgst, so a corrupt stream cannot be stored silently.

Push does not access r after returning. To preserve that ownership rule, it waits for every in-flight r.Read call to finish even after an early registry response or context cancellation. A reader that can block must arrange for its producer to unblock it when ctx ends; an arbitrary io.Reader has no general cancellation operation.

A failed upload restarts from byte zero under the client's RetryPolicy. Restarting needs the reader's bytes again, so r must implement io.Seeker (as bytes.Reader, strings.Reader, and os.File do); with a non-seekable reader the upload fails on the first transient error instead of retrying.

Example: 

dgst := digest.FromBytes(data)
err := client.Push(ctx, repo, dgst, int64(len(data)), bytes.NewReader(data))

type Option 

type Option func(*options)

Option configures a Client built by New.

func WithChunkedUpload 

func WithChunkedUpload(chunkSize int64) Option

WithChunkedUpload switches Push from the default monolithic upload to chunked PATCH uploads of chunkSize bytes. Values below one are ignored and leave monolithic upload in place.

Chunked upload is spec-optional and broken on major hosted registries (ECR discards chunks after the first and still reports success), because mainstream clients never exercise it. It is an explicit opt-in, never a fallback: leave it off unless you have verified your registry against it. The client checks the registry's Range acknowledgement after every chunk and abandons the upload rather than store a blob silently missing bytes.

func WithParallelPull 

func WithParallelPull(workers int, chunkSize int64) Option

WithParallelPull switches Pull to fetch blobs with workers concurrent ranged requests of chunkSize bytes each. Values below one for either parameter are ignored and leave single-stream pull in place. Worker counts above 1,024 and configurations whose workers × chunkSize memory bound cannot be represented by an int64 are likewise ignored.

Pull's contract does not change: chunks are emitted in order through the same digest-verifying reader. Memory use is bounded by roughly workers × chunkSize — the library's one deliberate exception to never buffering — and the caller sets that bound with these two parameters. When the registry does not serve ranges, Pull quietly falls back to a single stream: the toggle states intent, not a requirement. The library sizes its default HTTP/1 idle pool for the requested worker count. Caller-supplied transports are not modified; configure their connection pools for the intended concurrency.

func WithPlainHTTP 

func WithPlainHTTP(plain bool) Option

WithPlainHTTP selects plain http:// registry URLs instead of https. Meant for local registries served without TLS; leave it off for anything reachable from the internet.

func WithRetryPolicy 

func WithRetryPolicy(policy RetryPolicy) Option

WithRetryPolicy replaces the client's retry policy. Pass a zero RetryPolicy to disable retries entirely.

func WithStorageTransport 

func WithStorageTransport(rt http.RoundTripper) Option

WithStorageTransport sets the http.RoundTripper used for off-origin storage and CDN requests. It receives requests after registry credentials, cookies, proxy credentials, and referrer data have been removed. Callers use this transport to enforce destination policy or supply storage-specific TLS, proxy, or authentication behavior; the package does not block private or local destinations itself. A nil or typed-nil transport keeps the library-managed default transport.

func WithTransport 

func WithTransport(rt http.RoundTripper) Option

WithTransport sets the http.RoundTripper used for requests to the registry origin. This is the seam where callers inject registry authentication, for example through oras-go or go-containerregistry. The package does not acquire or store credentials. Off-origin redirects and absolute upload locations do not pass through this transport. A nil or typed-nil transport keeps the library-managed default transport.

func WithWriteRedirects added in v1.1.0 

func WithWriteRedirects(allow bool) Option

WithWriteRedirects controls whether redirects may reissue POST, PUT, PATCH, or DELETE requests. The default permits method-preserving write redirects. When disabled, the client rejects the redirect before sending its target request. Successful registry responses may still select an upload-session Location; those values are not HTTP redirects.

type Repository 

type Repository struct {
	// Host is the registry host with an optional port, such as
	// "registry.example.com" or "localhost:5000". It carries no scheme;
	// the Client decides between https and plain http.
	Host string

	// Name is the repository path within the registry, such as
	// "library/ubuntu". It must match the OCI distribution spec's
	// repository name grammar.
	Name string
}

Repository addresses a blob store: a registry host plus a repository name within it.

func (Repository) Validate 

func (r Repository) Validate() error

Validate reports whether the Repository can address a registry.

The host must be non-empty and free of a scheme or path. The name must match the OCI distribution spec grammar: slash-separated components of lowercase alphanumerics joined by ".", "_", "__", or one or more "-".

type RetryPolicy 

type RetryPolicy struct {
	// MaxAttempts is the total number of tries for one operation,
	// including the first. Values below one behave as one.
	MaxAttempts int

	// InitialDelay seeds the exponential backoff: attempt n waits a
	// full-jittered duration in [0, InitialDelay * 2^(n-1)].
	InitialDelay time.Duration

	// MaxDelay caps every wait, including waits requested by a
	// registry's Retry-After header.
	MaxDelay time.Duration
}

RetryPolicy bounds how the client retries failed requests.

The zero value means a single attempt with no retries. Retries trigger on connection errors, request timeouts, 429, and 5xx; other 4xx statuses mean the request is wrong, not unlucky, and are never retried. The caller's context bounds retry scheduling: a canceled context stops new attempts and backoff immediately.

func DefaultRetryPolicy 

func DefaultRetryPolicy() RetryPolicy

DefaultRetryPolicy returns the policy New applies when WithRetryPolicy is not given: four total attempts, backoff seeded at 250ms, and no wait longer than 30 seconds.

type TransferOption 

type TransferOption func(*transferConfig)

TransferOption adjusts a single byte-moving call: Pull, PullRange, or Push. Settings that apply to every call belong on the Client instead.

func WithProgress 

func WithProgress(fn func(done, total int64)) TransferOption

WithProgress reports transfer progress to fn.

fn receives the cumulative number of bytes moved so far and the total (-1 when the total is unknown). Pull counts bytes delivered to the caller. Monolithic Push reports only after the final 201 response; chunked Push advances after each PATCH Range acknowledgement, so reaching total does not prove the final commit succeeded. Only a nil Push error does. Within one transfer, counts never move backward across retries or resumes, and calls to fn do not overlap, including during a parallel Pull. Concurrent transfers may call the same fn at the same time, so fn must protect any state shared between transfers. fn runs synchronously on the transfer path, so it must return quickly.

func WithWireProgress added in v1.1.0 

func WithWireProgress(fn func(delta int64)) TransferOption

WithWireProgress reports positive upload-byte deltas consumed by the HTTP transport. It does not count source read-ahead. Failed attempts, redirects, and transparent retries all contribute because they consumed boundary traffic.

Calls are serialized within one Push and stop before Push returns. Concurrent transfers may call the same fn concurrently. fn runs synchronously on the upload-body read path and must return quickly. A nil fn disables reporting and its associated accounting.

Source Files

View all Source files

Directories

Path Synopsis
internal
perf
Package perf contains behavioral performance regressions and benchmarks for the public blob transfer paths.
 Package perf contains behavioral performance regressions and benchmarks for the public blob transfer paths.
Package mocks holds mockery-generated test doubles for the blob package's transport port.
 Package mocks holds mockery-generated test doubles for the blob package's transport port.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.