v0.37.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Mar 12, 2024 License: Apache-2.0 Imports: 23 Imported by: 264



Package fileblob provides a blob implementation that uses the filesystem. Use OpenBucket to construct a *blob.Bucket.

To avoid partial writes, fileblob writes to a temporary file and then renames the temporary file to the final path on Close. By default, it creates these temporary files in `os.TempDir`. If `os.TempDir` is on a different mount than your base bucket path, the `os.Rename` will fail with `invalid cross-device link`. To avoid this, either configure the temp dir to use by setting the environment variable `TMPDIR`, or set `Options.NoTempDir` to `true` (fileblob will create the temporary files next to the actual files instead of in a temporary directory).

By default fileblob stores blob metadata in "sidecar" files under the original filename with an additional ".attrs" suffix. This behaviour can be changed via `Options.Metadata`; writing of those metadata files can be suppressed by setting it to `MetadataDontWrite` or its equivalent "metadata=skip" in the URL for the opener. In either case, absent any stored metadata many `blob.Attributes` fields will be set to default values.


For blob.OpenBucket, fileblob registers for the scheme "file". To customize the URL opener, or for more details on the URL format, see URLOpener. See for background information.


Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for fileblob:

  • Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__". If os.PathSeparator != "/", it is also escaped. Additionally, the "/" in "../", the trailing "/" in "//", and a trailing "/" is key names are escaped in the same way. On Windows, the characters "<>:"|?*" are also escaped.


fileblob exposes the following types for As:

  • Bucket: os.FileInfo
  • Error: *os.PathError
  • ListObject: os.FileInfo
  • Reader: io.Reader
  • ReaderOptions.BeforeRead: *os.File
  • Attributes: os.FileInfo
  • CopyOptions.BeforeCopy: *os.File
  • WriterOptions.BeforeWrite: *os.File
Example (OpenBucketFromURL)
package main

import (


func main() {
	// Create a temporary directory.
	dir, err := ioutil.TempDir("", "go-cloud-fileblob-example")
	if err != nil {
	defer os.RemoveAll(dir)

	// On Unix, append the dir to "file://".
	// On Windows, convert "\" to "/" and add a leading "/":
	dirpath := filepath.ToSlash(dir)
	if os.PathSeparator != '/' && !strings.HasPrefix(dirpath, "/") {
		dirpath = "/" + dirpath

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	ctx := context.Background()
	b, err := blob.OpenBucket(ctx, "file://"+dirpath)
	if err != nil {
	defer b.Close()

	// Now we can use b to read or write files to the container.
	err = b.WriteAll(ctx, "my-key", []byte("hello world"), nil)
	if err != nil {
	data, err := b.ReadAll(ctx, "my-key")
	if err != nil {


hello world




View Source
const (
	// Metadata gets written to a separate file.
	MetadataInSidecar metadataOption = ""
	// Writes won't carry metadata, as per the package docstring.
	MetadataDontWrite metadataOption = "skip"

Settings for Options.Metadata.

View Source
const Scheme = "file"

Scheme is the URL scheme fileblob registers its URLOpener under on blob.DefaultMux.


This section is empty.


func OpenBucket

func OpenBucket(dir string, opts *Options) (*blob.Bucket, error)

OpenBucket creates a *blob.Bucket backed by the filesystem and rooted at dir, which must exist. See the package documentation for an example.

package main

import (


func main() {
	// PRAGMA: This example is used on; PRAGMA comments adjust how it is shown and can be ignored.

	// The directory you pass to fileblob.OpenBucket must exist first.
	const myDir = "path/to/local/directory"
	if err := os.MkdirAll(myDir, 0777); err != nil {

	// Create a file-based bucket.
	bucket, err := fileblob.OpenBucket(myDir, nil)
	if err != nil {
	defer bucket.Close()


type Options

type Options struct {
	// URLSigner implements signing URLs (to allow access to a resource without
	// further authorization) and verifying that a given URL is unexpired and
	// contains a signature produced by the URLSigner.
	// URLSigner is only required for utilizing the SignedURL API.
	URLSigner URLSigner

	// If true, create the directory backing the Bucket if it does not exist
	// (using os.MkdirAll).
	CreateDir bool

	// If true, don't use os.TempDir for temporary files, but instead place them
	// next to the actual files. This may result in "stranded" temporary files
	// (e.g., if the application is killed before the file cleanup runs).
	// If your bucket directory is on a different mount than os.TempDir, you will
	// need to set this to true, as os.Rename will fail across mount points.
	NoTempDir bool

	// Refers to the strategy for how to deal with metadata (such as blob.Attributes).
	// For supported values please see the Metadata* constants.
	// If left unchanged, 'MetadataInSidecar' will be used.
	Metadata metadataOption

Options sets options for constructing a *blob.Bucket backed by fileblob.

type URLOpener added in v0.10.0

type URLOpener struct {
	// Options specifies the default options to pass to OpenBucket.
	Options Options

URLOpener opens file bucket URLs like "file:///foo/bar/baz".

The URL's host is ignored unless it is ".", which is used to signal a relative path. For example, "file://./../.." uses "../.." as the path.

If os.PathSeparator != "/", any leading "/" from the path is dropped and remaining '/' characters are converted to os.PathSeparator.

The following query parameters are supported:

  • create_dir: (any non-empty value) the directory is created (using os.MkDirAll) if it does not already exist.
  • no_tmp_dir: (any non-empty value) temporary files are created next to the final path instead of in os.TempDir.
  • base_url: the base URL to use to construct signed URLs; see URLSignerHMAC
  • secret_key_path: path to read for the secret key used to construct signed URLs; see URLSignerHMAC
  • metadata: if set to "skip", won't write metadata such as blob.Attributes as per the package docstring

If either of base_url / secret_key_path are provided, both must be.

  • file:///a/directory -> Passes "/a/directory" to OpenBucket.
  • file://localhost/a/directory -> Also passes "/a/directory".
  • file://./../.. -> The hostname is ".", signaling a relative path; passes "../..".
  • file:///c:/foo/bar on Windows. -> Passes "c:\foo\bar".
  • file://localhost/c:/foo/bar on Windows. -> Also passes "c:\foo\bar".
  • file:///a/directory?base_url=/show&secret_key_path=secret.key -> Passes "/a/directory" to OpenBucket, and sets Options.URLSigner to a URLSignerHMAC initialized with base URL "/show" and secret key bytes read from the file "secret.key".

func (*URLOpener) OpenBucketURL added in v0.10.0

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.

type URLSigner added in v0.11.0

type URLSigner interface {
	// URLFromKey defines how the bucket's object key will be turned
	// into a signed URL. URLFromKey must be safe to call from multiple goroutines.
	URLFromKey(ctx context.Context, key string, opts *driver.SignedURLOptions) (*url.URL, error)

	// KeyFromURL must be able to validate a URL returned from URLFromKey.
	// KeyFromURL must only return the object if if the URL is
	// both unexpired and authentic. KeyFromURL must be safe to call from
	// multiple goroutines. Implementations of KeyFromURL should not modify
	// the URL argument.
	KeyFromURL(ctx context.Context, surl *url.URL) (string, error)

URLSigner defines an interface for creating and verifying a signed URL for objects in a fileblob bucket. Signed URLs are typically used for granting access to an otherwise-protected resource without requiring further authentication, and callers should take care to restrict the creation of signed URLs as is appropriate for their application.

type URLSignerHMAC added in v0.11.0

type URLSignerHMAC struct {
	// contains filtered or unexported fields

URLSignerHMAC signs URLs by adding the object key, expiration time, and a hash-based message authentication code (HMAC) into the query parameters. Values of URLSignerHMAC with the same secret key will accept URLs produced by others as valid.

func NewURLSignerHMAC added in v0.11.0

func NewURLSignerHMAC(baseURL *url.URL, secretKey []byte) *URLSignerHMAC

NewURLSignerHMAC creates a URLSignerHMAC. If the secret key is empty, then NewURLSignerHMAC panics.

func (*URLSignerHMAC) KeyFromURL added in v0.11.0

func (h *URLSignerHMAC) KeyFromURL(ctx context.Context, sURL *url.URL) (string, error)

KeyFromURL checks expiry and signature, and returns the object key only if the signed URL is both authentic and unexpired.

func (*URLSignerHMAC) URLFromKey added in v0.11.0

func (h *URLSignerHMAC) URLFromKey(ctx context.Context, key string, opts *driver.SignedURLOptions) (*url.URL, error)

URLFromKey creates a signed URL by copying the baseURL and appending the object key, expiry, and signature as a query params.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL