binarystore

Documentation

Overview

Package binarystore defines the BinaryStore building block interface for reading, writing, and deleting large named binary files. It is designed to handle content ranging from a few bytes to several gigabytes via streaming I/O, so neither the Dapr runtime nor the component needs to buffer entire files in memory.

Authentication and connection details are configured in the component YAML; operations such as listing files or persisting per-file metadata are intentionally out of scope for this building block.

Index

• Constants
• Variables
• func NewSizeLimitReader(r io.Reader, max int64) io.Reader
• func ObjectPath(prefix, fileName string) (string, error)
• type BinaryStore
• type DeleteRequest
• type Feature
• type GetRequest
• type GetResponse
• type Metadata
• type SetRequest
  • func (r *SetRequest) ResetReader() error

Constants

const (
	// DefaultUploadPartSize is the buffer size, in bytes, that binary store
	// implementations use for each part of a streaming upload.
	//
	// Provider SDK defaults range from 1 MiB to 16 MiB, which makes upload
	// throughput, memory use, and (for Azure block blobs, where the 50,000
	// block limit turns the part size into a hard object-size ceiling) the
	// maximum supported file size differ from one component to the next.
	// Implementations therefore pass this value to their SDK explicitly
	// instead of relying on the SDK default.
	//
	// 16 MiB is chosen because it satisfies every provider's constraints:
	// OCI Object Storage rejects non-final multipart parts smaller than
	// 10 MiB, S3 allows at most 10,000 parts, and Azure block blobs allow at
	// most 50,000 blocks, so a smaller value would either fail outright on
	// OCI or lower the maximum object size on the other providers.
	DefaultUploadPartSize int64 = 16 * 1024 * 1024

	// DefaultUploadConcurrency is the number of upload parts that binary
	// store implementations transfer in parallel. Combined with
	// DefaultUploadPartSize this bounds the buffered data per upload to
	// roughly 32 MiB while still keeping more than one request in flight.
	DefaultUploadConcurrency = 2
)

Variables

var (
	// ErrFileAlreadyExists is returned by Set when Overwrite is false and a
	// file with the same name already exists.
	ErrFileAlreadyExists = errors.New("file already exists")

	// ErrFileNotFound is returned by Get and Delete when the named file does
	// not exist.
	ErrFileNotFound = errors.New("file not found")

	// ErrMissingFileName is returned when a required FileName field is empty.
	ErrMissingFileName = errors.New("file name is required")

	// ErrInvalidFileName is returned when FileName starts or ends with a slash.
	ErrInvalidFileName = errors.New("file name must not start or end with a slash")

	// ErrInvalidPrefix is returned when a configured prefix starts or ends with
	// a slash.
	ErrInvalidPrefix = errors.New("prefix must not start or end with a slash")

	// ErrReaderNotResettable is returned when a Set request cannot be rewound
	// for a retry.
	ErrReaderNotResettable = errors.New("data reader cannot be reset")

	// ErrFileTooLarge is returned by Set when the file exceeds the maximum
	// object size supported by the provider or configured on the component.
	ErrFileTooLarge = errors.New("file too large")

	// ErrStoreFull is returned by Set when the store cannot accept the file
	// without exceeding its configured total capacity.
	ErrStoreFull = errors.New("store is full")
)

Sentinel errors returned by BinaryStore implementations.

Functions

func NewSizeLimitReader

func NewSizeLimitReader(r io.Reader, max int64) io.Reader

NewSizeLimitReader returns a reader that passes through at most max bytes from r. Reading beyond max bytes fails with an error wrapping ErrFileTooLarge, so streaming uploads abort as soon as the limit is exceeded instead of after the whole body has been sent. A nil r is treated as an empty body rather than panicking on the first read.

func ObjectPath

func ObjectPath(prefix, fileName string) (string, error)

ObjectPath validates and applies an optional object prefix to a file name.

Types

type BinaryStore

type BinaryStore interface {
	metadata.ComponentWithMetadata

	// Init initializes the binary store with the provided metadata.
	Init(ctx context.Context, metadata Metadata) error

	// Features returns the list of optional features supported by this implementation.
	Features() []Feature

	// Set stores binary data identified by req.FileName.
	//
	// When req.Overwrite is false (POST semantics), Set returns ErrFileAlreadyExists
	// if a file with that name already exists. When req.Overwrite is true (PUT
	// semantics), any existing file is replaced.
	//
	// Set reads req.Data to completion unless it exceeds the maximum object
	// size supported by the provider or configured on the component, in which
	// case Set stops reading and returns ErrFileTooLarge. The caller must not
	// close req.Data until Set returns.
	Set(ctx context.Context, req *SetRequest) error

	// Get retrieves the binary data for the file identified by req.FileName.
	//
	// The Data field of the returned GetResponse is a streaming reader; the
	// caller is responsible for closing it after reading. If the file does not
	// exist, ErrFileNotFound is returned.
	Get(ctx context.Context, req *GetRequest) (*GetResponse, error)

	// Delete removes the file identified by req.FileName. If the file does not
	// exist, ErrFileNotFound is returned.
	Delete(ctx context.Context, req *DeleteRequest) error

	io.Closer
}

BinaryStore is the interface for binary store operations.

type DeleteRequest

type DeleteRequest struct {
	// FileName is the identifier of the file to delete. Must not be empty or
	// start or end with a slash.
	FileName string
}

DeleteRequest is the request object for the Delete operation.

type Feature

type Feature string

Feature names an optional capability provided by a BinaryStore implementation. Implementations report their supported features via the Features() method.

type GetRequest

type GetRequest struct {
	// FileName is the identifier of the file to retrieve. Must not be empty or
	// start or end with a slash.
	FileName string
}

GetRequest is the request object for the Get operation.

type GetResponse

type GetResponse struct {
	// Data is the binary content of the file as a streaming reader. The caller
	// is responsible for closing Data after reading.
	Data io.ReadCloser
}

GetResponse is the response object for the Get operation.

type Metadata

type Metadata struct {
	metadata.Base `json:",inline"`
}

Metadata contains a binary store specific set of metadata properties.

type SetRequest

type SetRequest struct {
	// FileName is the identifier of the file to store. Must not be empty or
	// start or end with a slash.
	FileName string

	// Data is the binary content to store. It is consumed as a stream; the
	// provider reads it to completion. The caller must not close it until Set
	// returns.
	Data io.Reader

	// Reset is an optional callback used by callers that retry a failed Set.
	// When set, it should rewind or recreate Data so a later retry starts from
	// the beginning. If Data implements io.Seeker, the caller may also rewind it
	// directly before retrying.
	Reset func() error

	// Overwrite controls create-vs-upsert semantics.
	//
	// false — POST semantics: return ErrFileAlreadyExists if the file exists.
	// true  — PUT semantics: create the file or replace it if it already exists.
	Overwrite bool
}

SetRequest is the request object for the Set operation.

func (*SetRequest) ResetReader

func (r *SetRequest) ResetReader() error

ResetReader rewinds Data to the beginning if possible. Callers may use it to prepare a Set request for a retry when Data supports seeking.