fileblob

package
Version: v0.25.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 2, 2022 License: Apache-2.0 Imports: 23 Imported by: 91

Documentation

Overview

Package fileblob provides a blob implementation that uses the filesystem. Use OpenBucket to construct a *blob.Bucket.

By default fileblob stores blob metadata in 'sidecar files' under the original filename but an additional ".attrs" suffix. That behaviour can be changed via Options.Metadata; writing of those metadata files can be suppressed by setting it to 'MetadataDontWrite' or its equivalent "metadata=skip" in the URL for the opener. In any case, absent any stored metadata many blob.Attributes fields will be set to default values.

URLs

For blob.OpenBucket, fileblob registers for the scheme "file". To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for fileblob:

- Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__".
  If os.PathSeparator != "/", it is also escaped.
  Additionally, the "/" in "../", the trailing "/" in "//", and a trailing
  "/" is key names are escaped in the same way.
  On Windows, the characters "<>:"|?*" are also escaped.

As

fileblob exposes the following types for As:

- Bucket: os.FileInfo
- Error: *os.PathError
- ListObject: os.FileInfo
- Reader: io.Reader
- ReaderOptions.BeforeRead: *os.File
- Attributes: os.FileInfo
- CopyOptions.BeforeCopy: *os.File
- WriterOptions.BeforeWrite: *os.File
Example (OpenBucketFromURL)
package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
	"strings"

	"gocloud.dev/blob"
)

func main() {
	// Create a temporary directory.
	dir, err := ioutil.TempDir("", "go-cloud-fileblob-example")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(dir)

	// On Unix, append the dir to "file://".
	// On Windows, convert "\" to "/" and add a leading "/":
	dirpath := filepath.ToSlash(dir)
	if os.PathSeparator != '/' && !strings.HasPrefix(dirpath, "/") {
		dirpath = "/" + dirpath
	}

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	ctx := context.Background()
	b, err := blob.OpenBucket(ctx, "file://"+dirpath)
	if err != nil {
		log.Fatal(err)
	}
	defer b.Close()

	// Now we can use b to read or write files to the container.
	err = b.WriteAll(ctx, "my-key", []byte("hello world"), nil)
	if err != nil {
		log.Fatal(err)
	}
	data, err := b.ReadAll(ctx, "my-key")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(data))

}
Output:

hello world

Index

Examples

Constants

View Source
const (
	// Metadata gets written to a separate file.
	MetadataInSidecar metadataOption = ""
	// Writes won't carry metadata, as per the package docstring.
	MetadataDontWrite metadataOption = "skip"
)

Settings for Options.Metadata.

View Source
const Scheme = "file"

Scheme is the URL scheme fileblob registers its URLOpener under on blob.DefaultMux.

Variables

This section is empty.

Functions

func OpenBucket

func OpenBucket(dir string, opts *Options) (*blob.Bucket, error)

OpenBucket creates a *blob.Bucket backed by the filesystem and rooted at dir, which must exist. See the package documentation for an example.

Example
package main

import (
	"log"
	"os"

	"gocloud.dev/blob/fileblob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.

	// The directory you pass to fileblob.OpenBucket must exist first.
	const myDir = "path/to/local/directory"
	if err := os.MkdirAll(myDir, 0777); err != nil {
		log.Fatal(err)
	}

	// Create a file-based bucket.
	bucket, err := fileblob.OpenBucket(myDir, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}
Output:

Types

type Options

type Options struct {
	// URLSigner implements signing URLs (to allow access to a resource without
	// further authorization) and verifying that a given URL is unexpired and
	// contains a signature produced by the URLSigner.
	// URLSigner is only required for utilizing the SignedURL API.
	URLSigner URLSigner

	// If true, create the directory backing the Bucket if it does not exist
	// (using os.MkdirAll).
	CreateDir bool

	// Refers to the strategy for how to deal with metadata (such as blob.Attributes).
	// For supported values please see the Metadata* constants.
	// If left unchanged, 'MetadataInSidecar' will be used.
	Metadata metadataOption
}

Options sets options for constructing a *blob.Bucket backed by fileblob.

type URLOpener added in v0.10.0

type URLOpener struct {
	// Options specifies the default options to pass to OpenBucket.
	Options Options
}

URLOpener opens file bucket URLs like "file:///foo/bar/baz".

The URL's host is ignored unless it is ".", which is used to signal a relative path. For example, "file://./../.." uses "../.." as the path.

If os.PathSeparator != "/", any leading "/" from the path is dropped and remaining '/' characters are converted to os.PathSeparator.

The following query parameters are supported:

- create_dir: (any non-empty value) the directory is created (using os.MkDirAll)
  if it does not already exist.
- base_url: the base URL to use to construct signed URLs; see URLSignerHMAC
- secret_key_path: path to read for the secret key used to construct signed URLs;
  see URLSignerHMAC
- metadata: if set to "skip", won't write metadata such as blob.Attributes
  as per the package docstring

If either of base_url / secret_key_path are provided, both must be.

- file:///a/directory
  -> Passes "/a/directory" to OpenBucket.
- file://localhost/a/directory
  -> Also passes "/a/directory".
- file://./../..
  -> The hostname is ".", signaling a relative path; passes "../..".
- file:///c:/foo/bar on Windows.
  -> Passes "c:\foo\bar".
- file://localhost/c:/foo/bar on Windows.
  -> Also passes "c:\foo\bar".
- file:///a/directory?base_url=/show&secret_key_path=secret.key
  -> Passes "/a/directory" to OpenBucket, and sets Options.URLSigner
     to a URLSignerHMAC initialized with base URL "/show" and secret key
     bytes read from the file "secret.key".

func (*URLOpener) OpenBucketURL added in v0.10.0

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.

type URLSigner added in v0.11.0

type URLSigner interface {
	// URLFromKey defines how the bucket's object key will be turned
	// into a signed URL. URLFromKey must be safe to call from multiple goroutines.
	URLFromKey(ctx context.Context, key string, opts *driver.SignedURLOptions) (*url.URL, error)

	// KeyFromURL must be able to validate a URL returned from URLFromKey.
	// KeyFromURL must only return the object if if the URL is
	// both unexpired and authentic. KeyFromURL must be safe to call from
	// multiple goroutines. Implementations of KeyFromURL should not modify
	// the URL argument.
	KeyFromURL(ctx context.Context, surl *url.URL) (string, error)
}

URLSigner defines an interface for creating and verifying a signed URL for objects in a fileblob bucket. Signed URLs are typically used for granting access to an otherwise-protected resource without requiring further authentication, and callers should take care to restrict the creation of signed URLs as is appropriate for their application.

type URLSignerHMAC added in v0.11.0

type URLSignerHMAC struct {
	// contains filtered or unexported fields
}

URLSignerHMAC signs URLs by adding the object key, expiration time, and a hash-based message authentication code (HMAC) into the query parameters. Values of URLSignerHMAC with the same secret key will accept URLs produced by others as valid.

func NewURLSignerHMAC added in v0.11.0

func NewURLSignerHMAC(baseURL *url.URL, secretKey []byte) *URLSignerHMAC

NewURLSignerHMAC creates a URLSignerHMAC. If the secret key is empty, then NewURLSignerHMAC panics.

func (*URLSignerHMAC) KeyFromURL added in v0.11.0

func (h *URLSignerHMAC) KeyFromURL(ctx context.Context, sURL *url.URL) (string, error)

KeyFromURL checks expiry and signature, and returns the object key only if the signed URL is both authentic and unexpired.

func (*URLSignerHMAC) URLFromKey added in v0.11.0

func (h *URLSignerHMAC) URLFromKey(ctx context.Context, key string, opts *driver.SignedURLOptions) (*url.URL, error)

URLFromKey creates a signed URL by copying the baseURL and appending the object key, expiry, and signature as a query params.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL