Back to godoc.org
gocloud.dev / blob / gcsblob

Package gcsblob

v0.20.0
Latest Go to latest

The latest major version is .

Published: Jun 10, 2020 | License: Apache-2.0 | Module: gocloud.dev

Overview

Package gcsblob provides a blob implementation that uses GCS. Use OpenBucket to construct a *blob.Bucket.

URLs

For blob.OpenBucket, gcsblob registers for the scheme "gs". The default URL opener will set up a connection using default credentials from the environment, as described in https://cloud.google.com/docs/authentication/production. Some environments, such as GCE, come without a private key. In such cases the IAM Credentials API will be configured for use in Options.MakeSignBytes, which will introduce latency to any and all calls to bucket.SignedURL that you can avoid by installing a service account credentials file or obtaining and configuring a private key: https://cloud.google.com/iam/docs/creating-managing-service-account-keys

To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for gcsblob:

- Blob keys: ASCII characters 10 and 13 are escaped to "__0x<hex>__".
  Additionally, the "/" in "../" is escaped in the same way.

As

gcsblob exposes the following types for As:

- Bucket: *storage.Client
- Error: *googleapi.Error
- ListObject: storage.ObjectAttrs
- ListOptions.BeforeList: *storage.Query
- Reader: *storage.Reader
- ReaderOptions.BeforeRead: **storage.ObjectHandle, *storage.Reader (if accessing both, must be in that order)
- Attributes: storage.ObjectAttrs
- CopyOptions.BeforeCopy: *CopyObjectHandles, *storage.Copier (if accessing both, must be in that order)
- WriterOptions.BeforeWrite: **storage.ObjectHandle, *storage.Writer (if accessing both, must be in that order)
Example (OpenBucketFromURL)

Code:

package main

import (
	"context"
	"gocloud.dev/blob"
	"log"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/gcsblob"
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	// This URL will open the bucket "my-bucket" using default credentials.
	bucket, err := blob.OpenBucket(ctx, "gs://my-bucket")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}

Index

Examples

Constants

const Scheme = "gs"

Scheme is the URL scheme gcsblob registers its URLOpener under on blob.DefaultMux.

Variables

var Set = wire.NewSet(
	wire.Struct(new(URLOpener), "Client"),
)

Set holds Wire providers for this package.

func OpenBucket

func OpenBucket(ctx context.Context, client *gcp.HTTPClient, bucketName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by an existing GCS bucket. See the package documentation for an example.

Example

Code:

package main

import (
	"context"
	"gocloud.dev/blob/gcsblob"
	"gocloud.dev/gcp"
	"log"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// Your GCP credentials.
	// See https://cloud.google.com/docs/authentication/production
	// for more info on alternatives.
	creds, err := gcp.DefaultCredentials(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create an HTTP client.
	// This example uses the default HTTP transport and the credentials
	// created above.
	client, err := gcp.NewHTTPClient(
		gcp.DefaultTransport(),
		gcp.CredentialsTokenSource(creds))
	if err != nil {
		log.Fatal(err)
	}

	// Create a *blob.Bucket.
	bucket, err := gcsblob.OpenBucket(ctx, client, "my-bucket", nil)
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}

type CopyObjectHandles

type CopyObjectHandles struct {
	Dst, Src *storage.ObjectHandle
}

CopyObjectHandles holds the ObjectHandles for the destination and source of a Copy. It is used by the BeforeCopy As hook.

type Options

type Options struct {
	// GoogleAccessID represents the authorizer for SignedURL.
	// Required to use SignedURL.
	// See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
	GoogleAccessID string

	// PrivateKey is the Google service account private key.
	// Exactly one of PrivateKey or SignBytes must be non-nil to use SignedURL.
	// See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
	PrivateKey []byte

	// SignBytes is a function for implementing custom signing.
	// Exactly one of PrivateKey, SignBytes, or MakeSignBytes must be non-nil to use SignedURL.
	// See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
	SignBytes func([]byte) ([]byte, error)

	// MakeSignBytes is a factory for functions that are being used in place of an empty SignBytes.
	// If your implementation of 'SignBytes' needs a request context, set this instead.
	MakeSignBytes func(requestCtx context.Context) SignBytesFunc
}

Options sets options for constructing a *blob.Bucket backed by GCS.

type SignBytesFunc

type SignBytesFunc func([]byte) ([]byte, error)

SignBytesFunc is shorthand for the signature of Options.SignBytes.

type URLOpener

type URLOpener struct {
	// Client must be set to a non-nil HTTP client authenticated with
	// Cloud Storage scope or equivalent.
	Client *gcp.HTTPClient

	// Options specifies the default options to pass to OpenBucket.
	Options Options
}

URLOpener opens GCS URLs like "gs://mybucket".

The URL host is used as the bucket name.

The following query parameters are supported:

- access_id: sets Options.GoogleAccessID
- private_key_path: path to read for Options.PrivateKey

Currently their use is limited to SignedURL.

func (*URLOpener) OpenBucketURL

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens the GCS bucket with the same name as the URL's host.

Package Files

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to identifier