blob

package
v0.27.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2025 License: MIT Imports: 16 Imported by: 3

Documentation

Overview

Package blob defines a lightweight abstration for interacting with various storage services (local filesystem, S3, etc.).

NB! For compatibility with earlier PocketBase versions and to prevent unnecessary breaking changes, this package is based and implemented as a minimal, stripped down version of the previously used gocloud.dev/blob. While there is no promise that it won't diverge in the future to accommodate better some PocketBase specific use cases, currently it copies and tries to follow as close as possible the same implementations, conventions and rules for the key escaping/unescaping, blob read/write interfaces and struct options as gocloud.dev/blob, therefore the credits goes to the original Go Cloud Development Kit Authors.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound = errors.New("resource not found")
	ErrClosed   = errors.New("bucket or blob is closed")
)
View Source 
var FirstPageToken = []byte("first page")

FirstPageToken is the pageToken to pass to ListPage to retrieve the first page of results.

Functions

func HexEscape 

func HexEscape(s string, shouldEscape func(s []rune, i int) bool) string

HexEscape returns s, with all runes for which shouldEscape returns true escaped to "__0xXXX__", where XXX is the hex representation of the rune value. For example, " " would escape to "__0x20__".

Non-UTF-8 strings will have their non-UTF-8 characters escaped to unicode.ReplacementChar; the original value is lost. Please file an issue if you need non-UTF8 support.

Note: shouldEscape takes the whole string as a slice of runes and an index. Passing it a single byte or a single rune doesn't provide enough context for some escape decisions; for example, the caller might want to escape the second "/" in "//" but not the first one. We pass a slice of runes instead of the string or a slice of bytes because some decisions will be made on a rune basis (e.g., encode all non-ASCII runes).

func HexUnescape 

func HexUnescape(s string) string

HexUnescape reverses HexEscape.

Types

type Attributes 

type Attributes struct {
	// CacheControl specifies caching attributes that services may use
	// when serving the blob.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
	CacheControl string `json:"cacheControl"`

	// ContentDisposition specifies whether the blob content is expected to be
	// displayed inline or as an attachment.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
	ContentDisposition string `json:"contentDisposition"`

	// ContentEncoding specifies the encoding used for the blob's content, if any.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
	ContentEncoding string `json:"contentEncoding"`

	// ContentLanguage specifies the language used in the blob's content, if any.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Language
	ContentLanguage string `json:"contentLanguage"`

	// ContentType is the MIME type of the blob. It will not be empty.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
	ContentType string `json:"contentType"`

	// Metadata holds key/value pairs associated with the blob.
	// Keys are guaranteed to be in lowercase, even if the backend service
	// has case-sensitive keys (although note that Metadata written via
	// this package will always be lowercased). If there are duplicate
	// case-insensitive keys (e.g., "foo" and "FOO"), only one value
	// will be kept, and it is undefined which one.
	Metadata map[string]string `json:"metadata"`

	// CreateTime is the time the blob was created, if available. If not available,
	// CreateTime will be the zero time.
	CreateTime time.Time `json:"createTime"`

	// ModTime is the time the blob was last modified.
	ModTime time.Time `json:"modTime"`

	// Size is the size of the blob's content in bytes.
	Size int64 `json:"size"`

	// MD5 is an MD5 hash of the blob contents or nil if not available.
	MD5 []byte `json:"md5"`

	// ETag for the blob; see https://en.wikipedia.org/wiki/HTTP_ETag.
	ETag string `json:"etag"`
}

Attributes contains attributes about a blob.

type Bucket 

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket provides an easy and portable way to interact with blobs within a "bucket", including read, write, and list operations. To create a Bucket, use constructors found in driver subpackages.

func NewBucket 

func NewBucket(drv Driver) *Bucket

NewBucket creates a new *Bucket based on a specific driver implementation.

func (*Bucket) Attributes 

func (b *Bucket) Attributes(ctx context.Context, key string) (_ *Attributes, err error)

Attributes returns attributes for the blob stored at key.

If the blob does not exist, Attributes returns an error for which gcerrors.Code will return gcerrors.NotFound.

func (*Bucket) Close 

func (b *Bucket) Close() error

Close releases any resources used for the bucket.

func (*Bucket) Copy 

func (b *Bucket) Copy(ctx context.Context, dstKey, srcKey string) (err error)

Copy the blob stored at srcKey to dstKey. A nil CopyOptions is treated the same as the zero value.

If the source blob does not exist, Copy returns an error for which gcerrors.Code will return gcerrors.NotFound.

If the destination blob already exists, it is overwritten.

func (*Bucket) Delete 

func (b *Bucket) Delete(ctx context.Context, key string) (err error)

Delete deletes the blob stored at key.

If the blob does not exist, Delete returns an error for which gcerrors.Code will return gcerrors.NotFound.

func (*Bucket) Exists 

func (b *Bucket) Exists(ctx context.Context, key string) (bool, error)

Exists returns true if a blob exists at key, false if it does not exist, or an error.

It is a shortcut for calling Attributes and checking if it returns an error with code ErrNotFound.

func (*Bucket) List 

func (b *Bucket) List(opts *ListOptions) *ListIterator

List returns a ListIterator that can be used to iterate over blobs in a bucket, in lexicographical order of UTF-8 encoded keys. The underlying implementation fetches results in pages.

A nil ListOptions is treated the same as the zero value.

List is not guaranteed to include all recently-written blobs; some services are only eventually consistent.

func (*Bucket) ListPage 

func (b *Bucket) ListPage(ctx context.Context, pageToken []byte, pageSize int, opts *ListOptions) (retval []*ListObject, nextPageToken []byte, err error)

ListPage returns a page of ListObject results for blobs in a bucket, in lexicographical order of UTF-8 encoded keys.

To fetch the first page, pass FirstPageToken as the pageToken. For subsequent pages, pass the pageToken returned from a previous call to ListPage. It is not possible to "skip ahead" pages.

Each call will return pageSize results, unless there are not enough blobs to fill the page, in which case it will return fewer results (possibly 0).

If there are no more blobs available, ListPage will return an empty pageToken. Note that this may happen regardless of the number of returned results -- the last page might have 0 results (i.e., if the last item was deleted), pageSize results, or anything in between.

Calling ListPage with an empty pageToken will immediately return io.EOF. When looping over pages, callers can either check for an empty pageToken, or they can make one more call and check for io.EOF.

The underlying implementation fetches results in pages, but one call to ListPage may require multiple page fetches (and therefore, multiple calls to the BeforeList callback).

A nil ListOptions is treated the same as the zero value.

ListPage is not guaranteed to include all recently-written blobs; some services are only eventually consistent.

func (*Bucket) NewRangeReader 

func (b *Bucket) NewRangeReader(ctx context.Context, key string, offset, length int64) (_ *Reader, err error)

NewRangeReader returns a Reader to read content from the blob stored at key. It reads at most length bytes starting at offset (>= 0). If length is negative, it will read till the end of the blob.

For the purposes of Seek, the returned Reader will start at offset and end at the minimum of the actual end of the blob or (if length > 0) offset + length.

Note that ctx is used for all reads performed during the lifetime of the reader.

If the blob does not exist, NewRangeReader returns an error for which gcerrors.Code will return gcerrors.NotFound. Exists is a lighter-weight way to check for existence.

A nil ReaderOptions is treated the same as the zero value.

The caller must call Close on the returned Reader when done reading.

func (*Bucket) NewReader 

func (b *Bucket) NewReader(ctx context.Context, key string) (*Reader, error)

NewReader is a shortcut for NewRangeReader with offset=0 and length=-1.

func (*Bucket) NewWriter 

func (b *Bucket) NewWriter(ctx context.Context, key string, opts *WriterOptions) (_ *Writer, err error)

NewWriter returns a Writer that writes to the blob stored at key. A nil WriterOptions is treated the same as the zero value.

If a blob with this key already exists, it will be replaced. The blob being written is not guaranteed to be readable until Close has been called; until then, any previous blob will still be readable. Even after Close is called, newly written blobs are not guaranteed to be returned from List; some services are only eventually consistent.

The returned Writer will store ctx for later use in Write and/or Close. To abort a write, cancel ctx; otherwise, it must remain open until Close is called.

The caller must call Close on the returned Writer, even if the write is aborted.

type Driver 

type Driver interface {
	NormalizeError(err error) error

	// Attributes returns attributes for the blob. If the specified object does
	// not exist, Attributes must return an error for which ErrorCode returns
	// gcerrors.NotFound.
	// The portable type will not modify the returned Attributes.
	Attributes(ctx context.Context, key string) (*Attributes, error)

	// ListPaged lists objects in the bucket, in lexicographical order by
	// UTF-8-encoded key, returning pages of objects at a time.
	// Services are only required to be eventually consistent with respect
	// to recently written or deleted objects. That is to say, there is no
	// guarantee that an object that's been written will immediately be returned
	// from ListPaged.
	// opts is guaranteed to be non-nil.
	ListPaged(ctx context.Context, opts *ListOptions) (*ListPage, error)

	// NewRangeReader returns a Reader that reads part of an object, reading at
	// most length bytes starting at the given offset. If length is negative, it
	// will read until the end of the object. If the specified object does not
	// exist, NewRangeReader must return an error for which ErrorCode returns
	// gcerrors.NotFound.
	// opts is guaranteed to be non-nil.
	//
	// The returned Reader *may* also implement Downloader if the underlying
	// implementation can take advantage of that. The Download call is guaranteed
	// to be the only call to the Reader. For such readers, offset will always
	// be 0 and length will always be -1.
	NewRangeReader(ctx context.Context, key string, offset, length int64) (DriverReader, error)

	// NewTypedWriter returns Writer that writes to an object associated with key.
	//
	// A new object will be created unless an object with this key already exists.
	// Otherwise any previous object with the same key will be replaced.
	// The object may not be available (and any previous object will remain)
	// until Close has been called.
	//
	// contentType sets the MIME type of the object to be written.
	// opts is guaranteed to be non-nil.
	//
	// The caller must call Close on the returned Writer when done writing.
	//
	// Implementations should abort an ongoing write if ctx is later canceled,
	// and do any necessary cleanup in Close. Close should then return ctx.Err().
	//
	// The returned Writer *may* also implement Uploader if the underlying
	// implementation can take advantage of that. The Upload call is guaranteed
	// to be the only non-Close call to the Writer..
	NewTypedWriter(ctx context.Context, key, contentType string, opts *WriterOptions) (DriverWriter, error)

	// Copy copies the object associated with srcKey to dstKey.
	//
	// If the source object does not exist, Copy must return an error for which
	// ErrorCode returns gcerrors.NotFound.
	//
	// If the destination object already exists, it should be overwritten.
	//
	// opts is guaranteed to be non-nil.
	Copy(ctx context.Context, dstKey, srcKey string) error

	// Delete deletes the object associated with key. If the specified object does
	// not exist, Delete must return an error for which ErrorCode returns
	// gcerrors.NotFound.
	Delete(ctx context.Context, key string) error

	// Close cleans up any resources used by the Bucket. Once Close is called,
	// there will be no method calls to the Bucket other than As, ErrorAs, and
	// ErrorCode. There may be open readers or writers that will receive calls.
	// It is up to the driver as to how these will be handled.
	Close() error
}

Driver provides read, write and delete operations on objects within it on the blob service.

type DriverReader 

type DriverReader interface {
	io.ReadCloser

	// Attributes returns a subset of attributes about the blob.
	// The portable type will not modify the returned ReaderAttributes.
	Attributes() *ReaderAttributes
}

DriverReader reads an object from the blob.

type DriverWriter 

type DriverWriter interface {
	io.WriteCloser
}

DriverWriter writes an object to the blob.

type ListIterator 

type ListIterator struct {
	// contains filtered or unexported fields
}

ListIterator iterates over List results.

func (*ListIterator) Next 

func (i *ListIterator) Next(ctx context.Context) (*ListObject, error)

Next returns a *ListObject for the next blob. It returns (nil, io.EOF) if there are no more.

type ListObject 

type ListObject struct {
	// Key is the key for this blob.
	Key string `json:"key"`

	// ModTime is the time the blob was last modified.
	ModTime time.Time `json:"modTime"`

	// Size is the size of the blob's content in bytes.
	Size int64 `json:"size"`

	// MD5 is an MD5 hash of the blob contents or nil if not available.
	MD5 []byte `json:"md5"`

	// IsDir indicates that this result represents a "directory" in the
	// hierarchical namespace, ending in ListOptions.Delimiter. Key can be
	// passed as ListOptions.Prefix to list items in the "directory".
	// Fields other than Key and IsDir will not be set if IsDir is true.
	IsDir bool `json:"isDir"`
}

ListObject represents a single blob returned from List.

type ListOptions 

type ListOptions struct {
	// Prefix indicates that only blobs with a key starting with this prefix
	// should be returned.
	Prefix string

	// Delimiter sets the delimiter used to define a hierarchical namespace,
	// like a filesystem with "directories". It is highly recommended that you
	// use "" or "/" as the Delimiter. Other values should work through this API,
	// but service UIs generally assume "/".
	//
	// An empty delimiter means that the bucket is treated as a single flat
	// namespace.
	//
	// A non-empty delimiter means that any result with the delimiter in its key
	// after Prefix is stripped will be returned with ListObject.IsDir = true,
	// ListObject.Key truncated after the delimiter, and zero values for other
	// ListObject fields. These results represent "directories". Multiple results
	// in a "directory" are returned as a single result.
	Delimiter string

	// PageSize sets the maximum number of objects to be returned.
	// 0 means no maximum; driver implementations should choose a reasonable
	// max. It is guaranteed to be >= 0.
	PageSize int

	// PageToken may be filled in with the NextPageToken from a previous
	// ListPaged call.
	PageToken []byte
}

ListOptions sets options for listing blobs via Bucket.List.

type ListPage 

type ListPage struct {
	// Objects is the slice of objects found. If ListOptions.PageSize > 0,
	// it should have at most ListOptions.PageSize entries.
	//
	// Objects should be returned in lexicographical order of UTF-8 encoded keys,
	// including across pages. I.e., all objects returned from a ListPage request
	// made using a PageToken from a previous ListPage request's NextPageToken
	// should have Key >= the Key for all objects from the previous request.
	Objects []*ListObject `json:"objects"`

	// NextPageToken should be left empty unless there are more objects
	// to return. The value may be returned as ListOptions.PageToken on a
	// subsequent ListPaged call, to fetch the next page of results.
	// It can be an arbitrary []byte; it need not be a valid key.
	NextPageToken []byte `json:"nextPageToken"`
}

ListPage represents a page of results return from ListPaged.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads bytes from a blob. It implements io.ReadSeekCloser, and must be closed after reads are finished.

func (*Reader) Close 

func (r *Reader) Close() error

Close implements io.Closer (https://golang.org/pkg/io/#Closer).

func (*Reader) ContentType 

func (r *Reader) ContentType() string

ContentType returns the MIME type of the blob.

func (*Reader) ModTime 

func (r *Reader) ModTime() time.Time

ModTime returns the time the blob was last modified.

func (*Reader) Read 

func (r *Reader) Read(p []byte) (int, error)

Read implements io.Reader (https://golang.org/pkg/io/#Reader).

func (*Reader) Seek 

func (r *Reader) Seek(offset int64, whence int) (int64, error)

Seek implements io.Seeker (https://golang.org/pkg/io/#Seeker).

func (*Reader) Size 

func (r *Reader) Size() int64

Size returns the size of the blob content in bytes.

func (*Reader) WriteTo 

func (r *Reader) WriteTo(w io.Writer) (int64, error)

WriteTo reads from r and writes to w until there's no more data or an error occurs. The return value is the number of bytes written to w.

It implements the io.WriterTo interface.

type ReaderAttributes 

type ReaderAttributes struct {
	// ContentType is the MIME type of the blob object. It must not be empty.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
	ContentType string `json:"contentType"`

	// ModTime is the time the blob object was last modified.
	ModTime time.Time `json:"modTime"`

	// Size is the size of the object in bytes.
	Size int64 `json:"size"`
}

ReaderAttributes contains a subset of attributes about a blob that are accessible from Reader.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes bytes to a blob.

It implements io.WriteCloser (https://golang.org/pkg/io/#Closer), and must be closed after all writes are done.

func (*Writer) Close 

func (w *Writer) Close() (err error)

Close closes the blob writer. The write operation is not guaranteed to have succeeded until Close returns with no error.

Close may return an error if the context provided to create the Writer is canceled or reaches its deadline.

func (*Writer) ReadFrom 

func (w *Writer) ReadFrom(r io.Reader) (int64, error)

ReadFrom reads from r and writes to w until EOF or error. The return value is the number of bytes read from r.

It implements the io.ReaderFrom interface.

func (*Writer) Write 

func (w *Writer) Write(p []byte) (int, error)

Write implements the io.Writer interface (https://golang.org/pkg/io/#Writer).

Writes may happen asynchronously, so the returned error can be nil even if the actual write eventually fails. The write is only guaranteed to have succeeded if Close returns no error.

type WriterOptions 

type WriterOptions struct {
	// BufferSize changes the default size in bytes of the chunks that
	// Writer will upload in a single request; larger blobs will be split into
	// multiple requests.
	//
	// This option may be ignored by some drivers.
	//
	// If 0, the driver will choose a reasonable default.
	//
	// If the Writer is used to do many small writes concurrently, using a
	// smaller BufferSize may reduce memory usage.
	BufferSize int

	// MaxConcurrency changes the default concurrency for parts of an upload.
	//
	// This option may be ignored by some drivers.
	//
	// If 0, the driver will choose a reasonable default.
	MaxConcurrency int

	// CacheControl specifies caching attributes that services may use
	// when serving the blob.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
	CacheControl string

	// ContentDisposition specifies whether the blob content is expected to be
	// displayed inline or as an attachment.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
	ContentDisposition string

	// ContentEncoding specifies the encoding used for the blob's content, if any.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
	ContentEncoding string

	// ContentLanguage specifies the language used in the blob's content, if any.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Language
	ContentLanguage string

	// ContentType specifies the MIME type of the blob being written. If not set,
	// it will be inferred from the content using the algorithm described at
	// http://mimesniff.spec.whatwg.org/.
	// Set DisableContentTypeDetection to true to disable the above and force
	// the ContentType to stay empty.
	// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
	ContentType string

	// When true, if ContentType is the empty string, it will stay the empty
	// string rather than being inferred from the content.
	// Note that while the blob will be written with an empty string ContentType,
	// most providers will fill one in during reads, so don't expect an empty
	// ContentType if you read the blob back.
	DisableContentTypeDetection bool

	// ContentMD5 is used as a message integrity check.
	// If len(ContentMD5) > 0, the MD5 hash of the bytes written must match
	// ContentMD5, or Close will return an error without completing the write.
	// https://tools.ietf.org/html/rfc1864
	ContentMD5 []byte

	// Metadata holds key/value strings to be associated with the blob, or nil.
	// Keys may not be empty, and are lowercased before being written.
	// Duplicate case-insensitive keys (e.g., "foo" and "FOO") will result in
	// an error.
	Metadata map[string]string
}

WriterOptions sets options for NewWriter.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.