azure

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2026 License: MIT Imports: 42 Imported by: 0

Documentation

Overview

Package azure implements the store interface over Azure Blob Storage.

Blob storage has no directories, only names that happen to contain slashes. This package presents the flat namespace as a tree so that cp's rules about files and directories keep working: a name with children behaves as a directory, a zero-byte blob whose name ends in "/" is an empty directory, and listings synthesise the intermediate prefixes a filesystem would have.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IncompleteDownload added in v0.3.2 

func IncompleteDownload(path string) bool

IncompleteDownload reports whether a resume record sits beside path, meaning the file there is a download that stopped part-way.

Nothing else can tell. Ranges arrive out of order, so a partly written file is already the size of the whole blob and carries a timestamp from when it was last touched: to -n and -u it is indistinguishable from a finished copy, and skipping it would leave it that way for good.

Types

type AuthMode 

type AuthMode string

AuthMode selects how credentials are found. The default, AuthAuto, tries everything in turn so that a correctly configured environment needs no flags at all. 

const (
	// AuthAuto walks the whole chain: SAS in the URL, connection string,
	// account key, the ambient Azure identity, an interactive device-code
	// sign-in, and finally anonymous access.
	AuthAuto AuthMode = "auto"
	// AuthIdentity restricts the chain to DefaultAzureCredential.
	AuthIdentity AuthMode = "identity"
	// AuthDevice forces a device-code sign-in, which works anywhere the code
	// can be read and typed elsewhere.
	AuthDevice AuthMode = "device"
	// AuthBrowser forces a browser sign-in.
	AuthBrowser AuthMode = "browser"
	// AuthAnonymous makes unauthenticated requests, for public containers.
	AuthAnonymous AuthMode = "anonymous"
)

func ParseAuthMode 

func ParseAuthMode(s string) (AuthMode, error)

ParseAuthMode validates a --auth value.

type Config 

type Config struct {
	Auth        AuthMode
	Log         *slog.Logger
	Interactive bool
	TenantID    string
	// MaxRetries is how many times the SDK pipeline retries a single HTTP
	// request before giving up. The engine layers whole-operation retries on
	// top of this for failures the pipeline cannot recover from.
	MaxRetries int32
	// TryTimeout bounds one HTTP attempt. Zero means no per-attempt bound,
	// which is right for large block transfers.
	TryTimeout time.Duration
	// CreateContainer allows the tool to create a missing destination
	// container instead of reporting it as absent.
	CreateContainer bool
	// UserAgent is appended to the SDK's telemetry string.
	UserAgent string
	// IncludeMetadata asks listings to return each blob's metadata. It costs
	// a larger response, so it is only worth it when something will read it.
	IncludeMetadata bool
	// PeakRequests is how many requests this run can have outstanding at once.
	// It sizes the connection pool; see transport.go for why that matters.
	PeakRequests int
	// BytesPerSecond caps throughput across the whole run. Zero is unlimited.
	BytesPerSecond int64
}

Config configures the store.

type Credentials 

type Credentials struct {
	Mode        AuthMode
	Log         *slog.Logger
	Interactive bool // a terminal is attached, so a sign-in prompt can be answered
	// TenantID is the directory to authenticate against. UseTenant replaces it
	// when the storage account names another; every other reader holds mu.
	TenantID string
	// contains filtered or unexported fields
}

Credentials discovers and caches the credential for the process. Discovery is deliberately silent when it succeeds: the point of transparent login is that a user with `az login` already done, a managed identity, or the standard AZURE_* environment variables never has to think about it.

Discovery alone cannot tell whether a credential will be accepted — an `az login` session for the wrong tenant produces a perfectly good token that the storage account then refuses. Escalate exists for that case, and is driven by the service's answer rather than by guesswork here.

func (*Credentials) Consulted 

func (c *Credentials) Consulted() bool

Consulted reports whether credential discovery ran at all. It does not when a SAS token or an account key was supplied, since those bypass the identity chain entirely — which changes what a rejection means.

func (*Credentials) Escalate 

func (c *Credentials) Escalate(ctx context.Context) (azcore.TokenCredential, string, error)

Escalate discards whatever was discovered and signs in interactively, replacing the cached credential. It is called when the storage service rejects what discovery found — the one thing the credential chain cannot work out for itself. It returns the credential to try again with and where that came from.

A run prompts at most once, whether the sign-in succeeds or not: a person who declined once should not be asked again for every remaining file.

func (*Credentials) IsAnonymous 

func (c *Credentials) IsAnonymous() bool

IsAnonymous reports whether the resolved credential is no credential at all. A rejection then means sign-in was needed, rather than that the signed-in identity lacks a role.

func (*Credentials) Prompts 

func (c *Credentials) Prompts() int

Prompts reports how many interactive sign-ins this run has started. A correctly behaved run never exceeds one.

func (*Credentials) Resolve 

func (c *Credentials) Resolve(ctx context.Context) (azcore.TokenCredential, string, error)

Resolve returns the token credential to use, along with a short description of where it came from. A nil credential with a nil error means anonymous access: no identity was found, and the caller should proceed unauthenticated.

func (*Credentials) Token 

func (c *Credentials) Token(ctx context.Context) (string, error)

Token returns a bearer token for the Blob service, used to authorise server-side copies where the service itself fetches the source.

func (*Credentials) UseTenant added in v0.3.0 

func (c *Credentials) UseTenant(tenant string, refused func(error)) bool

UseTenant points the credential at a tenant learned after the fact — the storage account names the one it trusts when it turns a token away — and reports whether the identity in hand was re-pointed, which is what makes trying the operation again worthwhile.

That identity may well have access there, so it is asked for a token in the new tenant before anybody is troubled for a sign-in, and any sign-in that follows is directed at the same tenant.

type MD5Check 

type MD5Check int

MD5Check selects what to do about a blob's recorded checksum on download. 

const (
	// MD5Off does not check.
	MD5Off MD5Check = iota
	// MD5Warn logs a mismatch and carries on.
	MD5Warn
	// MD5Fail treats a mismatch as a failed transfer. A blob with no recorded
	// checksum is accepted, since most blobs have none.
	MD5Fail
	// MD5Require additionally fails when the blob has no checksum at all.
	MD5Require
)

func ParseMD5Check 

func ParseMD5Check(s string) (MD5Check, error)

ParseMD5Check maps a --check-md5 value.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store is the Azure Blob Storage namespace.

func New 

func New(cfg Config) *Store

New returns a store. No network traffic happens until the first operation.

func (*Store) Copy 

func (s *Store) Copy(ctx context.Context, src *store.Node, dst *uri.URL, o TransferOptions) error

Copy moves a blob to another blob, keeping the data off this host wherever the service allows it.

Three server-side routes are tried in turn, because which one works depends on what the endpoint implements and on how the source can be authorised:

  1. Put Blob From URL — one request, for blobs up to 256 MiB.
  2. Put Block From URL — the same idea block by block, for larger blobs. Both of these can present an OAuth token for the source, so they work across accounts.
  3. Copy Blob — the asynchronous form. It takes no source-authorisation header, so the source must be readable by the destination account (same account, or carrying a SAS), but it is widely implemented and has no size limit.

Only when none of those works do the bytes pass through this process.

func (*Store) Credentials 

func (s *Store) Credentials() *Credentials

Credentials exposes the credential resolver so the transfer path can obtain a bearer token for server-side copies.

func (*Store) Download 

func (s *Store) Download(ctx context.Context, src *store.Node, f *os.File, o TransferOptions) error

Download writes a blob to an already-open local file, fetching ranges in parallel. The file is truncated to the blob's length.

func (*Store) DownloadDiscard added in v0.2.0 

func (s *Store) DownloadDiscard(ctx context.Context, src *store.Node, o TransferOptions) error

DownloadDiscard reads a blob and throws the bytes away. It exists for the benchmark, where writing to disk would measure the disk.

func (*Store) MkdirAll 

func (s *Store) MkdirAll(ctx context.Context, u *uri.URL, mode fs.FileMode) error

MkdirAll makes u usable as a destination. Containers are the one part of the blob namespace that must really exist, so this verifies (and optionally creates) the container and does nothing else: prefixes spring into being when the first blob is written.

func (*Store) MkdirMarker 

func (s *Store) MkdirMarker(ctx context.Context, u *uri.URL) error

MkdirMarker writes the zero-byte blob that represents an empty directory, so that an empty directory survives a round trip through blob storage.

func (*Store) OpenRead 

func (s *Store) OpenRead(ctx context.Context, src *store.Node) (io.ReadCloser, error)

OpenRead returns a reader over a blob that transparently re-issues the range request if the connection drops mid-stream.

func (*Store) PutMarker added in v0.2.0 

func (s *Store) PutMarker(ctx context.Context, dst *uri.URL, o TransferOptions) error

PutMarker writes a zero-length blob carrying only metadata. It is how a symbolic link, which has no content beyond its target, is represented.

func (*Store) ReadDir 

func (s *Store) ReadDir(ctx context.Context, u *uri.URL) ([]*store.Node, error)

ReadDir lists the immediate children of a container or prefix. Containers are listed when u addresses the account root.

func (*Store) Remove 

func (s *Store) Remove(ctx context.Context, u *uri.URL) error

Remove deletes a blob.

func (*Store) Scheme 

func (s *Store) Scheme() string

func (*Store) Stat 

func (s *Store) Stat(ctx context.Context, u *uri.URL, _ bool) (*store.Node, error)

Stat describes the node at u. Because prefixes are not real objects, a name that is not a blob is probed once more as a prefix before being reported missing.

func (*Store) Upload 

func (s *Store) Upload(ctx context.Context, srcPath string, dst *uri.URL, o TransferOptions) error

Upload writes a local file to a blob, staging blocks in parallel.

func (*Store) UploadAt added in v0.2.0 

func (s *Store) UploadAt(ctx context.Context, src io.ReaderAt, size int64,
	name string, dst *uri.URL, o TransferOptions) error

UploadAt writes size bytes read from src to a blob, staging blocks in parallel. It is what the benchmark uses, since the bytes come from memory rather than from a file.

func (*Store) UploadStream 

func (s *Store) UploadStream(ctx context.Context, r io.Reader, dst *uri.URL, o TransferOptions) error

UploadStream writes an arbitrary reader to a blob. It is used when the source size is not known ahead of time, such as when reading from a pipe.

func (*Store) WalkAll 

func (s *Store) WalkAll(ctx context.Context, u *uri.URL,
	onError func(*uri.URL, error) error, fn func(*store.Node) error) error

WalkAll lists everything beneath u with a single flat listing per container, synthesising the intermediate prefixes so a "**" pattern sees the same tree shape it would on a filesystem.

type TransferOptions 

type TransferOptions struct {
	// BlockSize is the unit of parallel transfer. It is raised automatically
	// when the file is too large to fit in the block-count limit.
	BlockSize int64
	// Concurrency is how many blocks of this one file move at once.
	Concurrency int
	// Progress receives the cumulative byte count. The SDK may report a lower
	// figure than before when it retries a block, so callers must treat the
	// value as absolute rather than as an increment.
	Progress func(transferred int64)
	// ContentType overrides the type guessed from the file extension.
	ContentType string
	// The remaining content headers are set only when given.
	ContentEncoding    string
	ContentDisposition string
	ContentLanguage    string
	CacheControl       string
	// Metadata is stored on the blob. The engine merges the user's --metadata
	// with the POSIX attributes when --preserve asks for them.
	Metadata map[string]string
	// AccessTier sets the blob tier on write.
	AccessTier string
	// NoClobber makes the write fail if the destination blob already exists,
	// closing the gap between checking and writing that a plain stat leaves.
	NoClobber bool
	// PutMD5 records a checksum of the whole file on the blob, so a later
	// download can be verified against it. It costs one extra read of the
	// source, since blocks are sent out of order and cannot be hashed on the
	// way past.
	PutMD5 bool
	// CheckMD5 says what to do when a downloaded blob carries a checksum.
	CheckMD5 MD5Check
	// Resume continues an interrupted transfer instead of starting again.
	Resume bool
}

TransferOptions configures one file's worth of data movement.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.