gcsblob

Documentation

Overview

Package gcsblob provides a blob implementation that uses GCS. Use OpenBucket to construct a *blob.Bucket.

URLs

For blob.OpenBucket, gcsblob registers for the scheme "gs". The default URL opener will set up a connection using default credentials from the environment, as described in https://cloud.google.com/docs/authentication/production. You may force the use of an unauthenticated client by setting GoogleAccessID to "-" (via Options or via the URL parameter "access_id"). Some environments, such as GCE, come without a private key. In such cases the IAM Credentials API will be configured for use in Options.MakeSignBytes, which will introduce latency to any and all calls to bucket.SignedURL that you can avoid by installing a service account credentials file or obtaining and configuring a private key: https://cloud.google.com/iam/docs/creating-managing-service-account-keys

To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for gcsblob:

• Blob keys: ASCII characters 10 and 13 are escaped to "__0x<hex>__". Additionally, the "/" in "../" is escaped in the same way.

As

gcsblob exposes the following types for As:

• Bucket: *storage.Client
• Error: *googleapi.Error
• ListObject: storage.ObjectAttrs
• ListOptions.BeforeList: *storage.Query
• Reader: *storage.Reader
• ReaderOptions.BeforeRead: **storage.ObjectHandle, *storage.Reader (if accessing both, must be in that order)
• Attributes: storage.ObjectAttrs
• CopyOptions.BeforeCopy: *CopyObjectHandles, *storage.Copier (if accessing both, must be in that order)
• WriterOptions.BeforeWrite: **storage.ObjectHandle, *storage.Writer (if accessing both, must be in that order)
• SignedURLOptions.BeforeSign: *storage.SignedURLOptions

Example (OpenBucketFromURL)

package main

import (
	"context"
	"log"

	"gocloud.dev/blob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/gcsblob"
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	// This URL will open the bucket "my-bucket" using default credentials.
	bucket, err := blob.OpenBucket(ctx, "gs://my-bucket")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}

Index

• Constants
• Variables
• func OpenBucket(ctx context.Context, client *gcp.HTTPClient, bucketName string, opts *Options) (*blob.Bucket, error)
• type CopyObjectHandles
• type Options
• type SignBytesFunc
• type URLOpener
  • func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

Examples

• Package (OpenBucketFromURL)
• OpenBucket

Constants

const Scheme = "gs"

Scheme is the URL scheme gcsblob registers its URLOpener under on blob.DefaultMux.

Variables

var Set = wire.NewSet(
	wire.Struct(new(URLOpener), "Client"),
)

Set holds Wire providers for this package.

Functions

func OpenBucket

func OpenBucket(ctx context.Context, client *gcp.HTTPClient, bucketName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by an existing GCS bucket. See the package documentation for an example.

Example

package main

import (
	"context"
	"log"

	"gocloud.dev/blob/gcsblob"
	"gocloud.dev/gcp"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// Your GCP credentials.
	// See https://cloud.google.com/docs/authentication/production
	// for more info on alternatives.
	creds, err := gcp.DefaultCredentials(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create an HTTP client.
	// This example uses the default HTTP transport and the credentials
	// created above.
	client, err := gcp.NewHTTPClient(
		gcp.DefaultTransport(),
		gcp.CredentialsTokenSource(creds))
	if err != nil {
		log.Fatal(err)
	}

	// Create a *blob.Bucket.
	bucket, err := gcsblob.OpenBucket(ctx, client, "my-bucket", nil)
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}

Types

type CopyObjectHandles

type CopyObjectHandles struct {
	Dst, Src *storage.ObjectHandle
}

CopyObjectHandles holds the ObjectHandles for the destination and source of a Copy. It is used by the BeforeCopy As hook.

type Options

type Options struct {
	// GoogleAccessID represents the authorizer for SignedURL.
	// If set to "-", an unauthenticated client will be used.
	// Required to use SignedURL.
	// See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
	GoogleAccessID string

	// PrivateKey is the Google service account private key.
	// Exactly one of PrivateKey or SignBytes must be non-nil to use SignedURL.
	// See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
	// Deprecated: Use MakeSignBytes instead.
	PrivateKey []byte

	// SignBytes is a function for implementing custom signing.
	// Exactly one of PrivateKey, SignBytes, or MakeSignBytes must be non-nil to use SignedURL.
	// See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
	// Deprecated: Use MakeSignBytes instead.
	SignBytes func([]byte) ([]byte, error)

	// MakeSignBytes is a factory for functions that are being used in place of an empty SignBytes.
	// If your implementation of 'SignBytes' needs a request context, set this instead.
	MakeSignBytes func(requestCtx context.Context) SignBytesFunc
}

Options sets options for constructing a *blob.Bucket backed by GCS.

type SignBytesFunc

type SignBytesFunc func([]byte) ([]byte, error)

SignBytesFunc is shorthand for the signature of Options.SignBytes.

type URLOpener

type URLOpener struct {
	// Client must be set to a non-nil HTTP client authenticated with
	// Cloud Storage scope or equivalent.
	Client *gcp.HTTPClient

	// Options specifies the default options to pass to OpenBucket.
	Options Options
}

URLOpener opens GCS URLs like "gs://mybucket".

The URL host is used as the bucket name.

The following query parameters are supported:

• access_id: sets Options.GoogleAccessID
• private_key_path: path to read for Options.PrivateKey

Currently their use is limited to SignedURL, except that setting access_id to "-" forces the use of an unauthenticated client.

func (*URLOpener) OpenBucketURL

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens the GCS bucket with the same name as the URL's host.