gcs

Documentation

Overview

An incomplete API for interacting with Google Cloud Storage.

Index

• Constants
• type Bucket
• type ByteRange
  • func (br ByteRange) String() string
• type ComposeObjectsRequest
• type ComposeSource
• type Conn
  • func NewConn(cfg *ConnConfig) (c Conn, err error)
• type ConnConfig
• type CopyObjectRequest
• type CreateObjectRequest
• type DeleteObjectRequest
• type ListObjectsRequest
• type Listing
• type MockBucket
  • func NewMockBucket(c oglemock.Controller, desc string) MockBucket
• type MoveObjectRequest
• type NotFoundError
  • func (nfe *NotFoundError) Error() string
• type Object
• type PreconditionError
  • func (pe *PreconditionError) Error() string
• type ReadObjectRequest
• type ReadSeekCloser
• type StatObjectRequest
• type UpdateObjectRequest

Constants

const (
	Scope_FullControl = storagev1.DevstorageFullControlScope
	Scope_ReadOnly    = storagev1.DevstorageReadOnlyScope
	Scope_ReadWrite   = storagev1.DevstorageReadWriteScope
)

OAuth scopes for GCS. For use with e.g. google.DefaultTokenSource.

const MaxComponentCount = 1024

MaxComponentCount is the maximum number of components that a composite object may have. The sum of the component counts of the sources in a ComposeObjectsRequest must be no more than this value.

Cf. https://cloud.google.com/storage/docs/composite-objects#_Count

const MaxSourcesPerComposeRequest = 32

MaxSourcesPerComposeRequest is the maximum number of sources that a ComposeObjectsRequest may contain.

Cf. https://cloud.google.com/storage/docs/composite-objects#_Compose

Types

type Bucket

type Bucket interface {
	Name() string

	// Create a reader for the contents of a particular generation of an object.
	// On a nil error, the caller must arrange for the reader to be closed when
	// it is no longer needed.
	//
	// Non-existent objects cause either this method or the first read from the
	// resulting reader to return an error of type *NotFoundError.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/get
	NewReader(
		ctx context.Context,
		req *ReadObjectRequest) (ReadSeekCloser, error)

	// Create or overwrite an object according to the supplied request. The new
	// object is guaranteed to exist immediately for the purposes of reading (and
	// eventually for listing) after this method returns a nil error. It is
	// guaranteed not to exist before req.Contents returns io.EOF.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/insert
	//     https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload
	CreateObject(
		ctx context.Context,
		req *CreateObjectRequest) (*Object, error)

	// Copy an object to a new name, preserving all metadata. Any existing
	// generation of the destination name will be overwritten.
	//
	// Returns a record for the new object.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/copy
	CopyObject(
		ctx context.Context,
		req *CopyObjectRequest) (*Object, error)

	// Move an object to a new name, preserving all metadata. Any existing
	// generation of the destination name will be overwritten.
	//
	// Returns a record for the new object.
	MoveObject(
		ctx context.Context,
		req *MoveObjectRequest) (*Object, error)

	// Compose one or more source objects into a single destination object by
	// concatenating. Any existing generation of the destination name will be
	// overwritten.
	//
	// Returns a record for the new object.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/compose
	ComposeObjects(
		ctx context.Context,
		req *ComposeObjectsRequest) (*Object, error)

	// Return current information about the object with the given name.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/get
	StatObject(
		ctx context.Context,
		req *StatObjectRequest) (*Object, error)

	// List the objects in the bucket that meet the criteria defined by the
	// request, returning a result object that contains the results and
	// potentially a cursor for retrieving the next portion of the larger set of
	// results.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/list
	ListObjects(
		ctx context.Context,
		req *ListObjectsRequest) (*Listing, error)

	// Update the object specified by newAttrs.Name, patching using the non-zero
	// fields of newAttrs.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/patch
	UpdateObject(
		ctx context.Context,
		req *UpdateObjectRequest) (*Object, error)

	// Delete an object. Non-existence of the object is not treated as an error.
	//
	// Official documentation:
	//     https://cloud.google.com/storage/docs/json_api/v1/objects/delete
	DeleteObject(
		ctx context.Context,
		req *DeleteObjectRequest) error
}

Bucket represents a GCS bucket, pre-bound with a bucket name and necessary authorization information.

Each method that may block accepts a context object that is used for deadlines and cancellation. Users need not package authorization information into the context object (using cloud.WithContext or similar).

All methods are safe for concurrent access.

type ByteRange

type ByteRange struct {
	Start uint64
	Limit uint64
}

ByteRange is a [start, limit) range of bytes within an object.

Its semantics are as follows:

• If Limit is less than or equal to Start, the range is treated as empty.

• The effective range is [start, limit) intersected with [0, L), where L is the length of the object.

  For example, a read for [L-1, L+10) returns the last byte of the object, and [L+2, L+10) is legal but returns nothing.

func (ByteRange) String

func (br ByteRange) String() string

type ComposeObjectsRequest

type ComposeObjectsRequest struct {
	// The name of the destination composite object.
	DstName string

	// If non-nil, the destination object will be created/overwritten only if the
	// current generation for its name is equal to the given value. Zero means
	// the object does not exist.
	DstGenerationPrecondition *int64

	// If non-nil, the destination object will be created/overwritten only if the
	// current meta-generation for its name is equal to the given value.
	//
	// This is only meaningful if DstGenerationPrecondition is also specified.
	DstMetaGenerationPrecondition *int64

	// The source objects from which to compose. This must be non-empty.
	//
	// Make sure to see the notes on MaxSourcesPerComposeRequest and
	// MaxComponentCount.
	Sources []ComposeSource

	// Optional information with which to create the object. See here for more
	// information:
	//
	//     https://cloud.google.com/storage/docs/json_api/v1/objects#resource
	//
	ContentType string
	Metadata    map[string]string
}

A request to compose one or more objects into a single composite object.

type ComposeSource

type ComposeSource struct {
	// The name of the source object.
	Name string

	// The generation of the source object to compose from. Zero means the latest
	// generation.
	Generation int64
}

type Conn

type Conn interface {
	// Return a Bucket object representing the GCS bucket with the given name.
	// Attempt to fail early in the case of bad credentials.
	OpenBucket(
		ctx context.Context,
		name string) (b Bucket, err error)
}

Conn represents a connection to GCS, pre-bound with a project ID and information required for authorization.

func NewConn

func NewConn(cfg *ConnConfig) (c Conn, err error)

Open a connection to GCS.

type ConnConfig

type ConnConfig struct {
	// An oauth2 token source to use for authenticating to GCS.
	//
	// You probably want this one:
	//     http://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource
	TokenSource oauth2.TokenSource

	// The value to set in User-Agent headers for outgoing HTTP requests. If
	// empty, a default will be used.
	UserAgent string

	// The HTTP transport to use for communication with GCS. If not supplied,
	// http.DefaultTransport will be used.
	Transport httputil.CancellableRoundTripper

	// The maximum amount of time to spend sleeping in a retry loop with
	// exponential backoff for failed requests. The default of zero disables
	// automatic retries.
	//
	// If you enable automatic retries, beware of the following:
	//
	//  *  Bucket.CreateObject will buffer the entire object contents in memory,
	//     so your object contents must not be too large to fit.
	//
	//  *  Bucket.NewReader needs to perform an additional round trip to GCS in
	//     order to find the latest object generation if you don't specify a
	//     particular generation.
	//
	//  *  Make sure your operations are idempotent, or that your application can
	//     tolerate it if not.
	//
	MaxBackoffSleep time.Duration

	// Loggers for GCS events, and (much more verbose) HTTP requests and
	// responses. If nil, no logging is performed.
	GCSDebugLogger  *log.Logger
	HTTPDebugLogger *log.Logger
}

ConnConfig contains options accepted by NewConn.

type CopyObjectRequest

type CopyObjectRequest struct {
	SrcName string
	DstName string

	// The generation of the source object to copy, or zero for the latest
	// generation.
	SrcGeneration int64

	// If non-nil, the destination object will be created/overwritten only if the
	// current meta-generation for the source object is equal to the given value.
	//
	// This is probably only meaningful in conjunction with SrcGeneration.
	SrcMetaGenerationPrecondition *int64
}

A request to copy an object to a new name, preserving all metadata.

type CreateObjectRequest

type CreateObjectRequest struct {
	// The name with which to create the object. This field must be set.
	//
	// Object names must:
	//
	// *  be non-empty.
	// *  be no longer than 1024 bytes.
	// *  be valid UTF-8.
	// *  not contain the code point U+000A (line feed).
	// *  not contain the code point U+000D (carriage return).
	//
	// See here for authoritative documentation:
	//     https://cloud.google.com/storage/docs/bucket-naming#objectnames
	Name string

	// Optional information with which to create the object. See here for more
	// information:
	//
	//     https://cloud.google.com/storage/docs/json_api/v1/objects#resource
	//
	ContentType     string
	ContentLanguage string
	ContentEncoding string
	CacheControl    string
	Metadata        map[string]string

	// A reader from which to obtain the contents of the object. Must be non-nil.
	Contents io.Reader

	// If non-nil, the object will not be created if the checksum of the received
	// contents does not match the supplied value.
	CRC32C *uint32

	// If non-nil, the object will not be created if the MD5 sum of the received
	// contents does not match the supplied value.
	MD5 *[md5.Size]byte

	// If non-nil, the object will be created/overwritten only if the current
	// generation for the object name is equal to the given value. Zero means the
	// object does not exist.
	GenerationPrecondition *int64

	// If non-nil, the object will be created/overwritten only if the current
	// meta-generation for the object name is equal to the given value. This is
	// only meaningful in conjunction with GenerationPrecondition.
	MetaGenerationPrecondition *int64
}

A request to create an object, accepted by Bucket.CreateObject.

type DeleteObjectRequest

type DeleteObjectRequest struct {
	// The name of the object to delete. Must be specified.
	Name string

	// The generation of the object to delete. Zero means the latest generation.
	Generation int64

	// If non-nil, the request will fail without effect if there is an object
	// with the given name (and optionally generation), and its meta-generation
	// is not equal to this value.
	MetaGenerationPrecondition *int64
}

A request to delete an object by name. Non-existence is not treated as an error.

type ListObjectsRequest

type ListObjectsRequest struct {
	// List only objects whose names begin with this prefix.
	Prefix string

	// Collapse results based on a delimiter.
	//
	// If non-empty, enable the following behavior. For each run of one or more
	// objects whose names are of the form:
	//
	//     <Prefix><S><Delimiter><...>
	//
	// where <S> is a string that doesn't itself contain Delimiter and <...> is
	// anything, return a single Collaped entry in the listing consisting of
	//
	//     <Prefix><S><Delimiter>
	//
	// instead of one Object record per object. If a collapsed entry consists of
	// a large number of objects, this may be more efficient.
	Delimiter string

	// Used to continue a listing where a previous one left off. See
	// Listing.ContinuationToken for more information.
	ContinuationToken string

	// The maximum number of objects and collapsed runs to return. Fewer than
	// this number may actually be returned. If this is zero, a sensible default
	// is used.
	MaxResults int
}

type Listing

type Listing struct {
	// Records for objects matching the listing criteria.
	//
	// Guaranteed to be strictly increasing under a lexicographical comparison on
	// (name, generation) pairs.
	Objects []*Object

	// Collapsed entries for runs of names sharing a prefix followed by a
	// delimiter. See notes on ListObjectsRequest.Delimiter.
	//
	// Guaranteed to be strictly increasing.
	CollapsedRuns []string

	// A continuation token, for fetching more results.
	//
	// If non-empty, this listing does not represent the full set of matching
	// objects in the bucket. Call ListObjects again with the request's
	// ContinuationToken field set to this value to continue where you left off.
	//
	// Guarantees, for replies R1 and R2, with R2 continuing from R1:
	//
	//  *  All of R1's object names are strictly less than all object names and
	//     collapsed runs in R2.
	//
	//  *  All of R1's collapsed runs are strictly less than all object names and
	//     prefixes in R2.
	//
	// (Cf. Google-internal bug 19286144)
	//
	// Note that there is no guarantee of atomicity of listings. Objects written
	// and deleted concurrently with a single or multiple listing requests may or
	// may not be returned.
	ContinuationToken string
}

Listing contains a set of objects and delimter-based collapsed runs returned by a call to ListObjects. See also ListObjectsRequest.

type MockBucket

type MockBucket interface {
	Bucket
	oglemock.MockObject
}

func NewMockBucket

func NewMockBucket(
	c oglemock.Controller,
	desc string) MockBucket

type MoveObjectRequest

type MoveObjectRequest struct {
	SrcName string
	DstName string

	// The generation of the source object to move, or zero for the latest
	// generation.
	SrcGeneration int64

	// If non-nil, the destination object will be created/overwritten only if the
	// current meta-generation for the source object is equal to the given value.
	//
	// This is probably only meaningful in conjunction with SrcGeneration.
	SrcMetaGenerationPrecondition *int64
}

A request to move an object to a new name, preserving all metadata.

type NotFoundError

type NotFoundError struct {
	Err error
}

A *NotFoundError value is an error that indicates an object name or a particular generation for that name were not found.

func (*NotFoundError) Error

func (nfe *NotFoundError) Error() string

type Object

type Object struct {
	Name            string
	ContentType     string
	ContentLanguage string
	CacheControl    string
	Owner           string
	Size            uint64
	ContentEncoding string
	MD5             *[md5.Size]byte // Missing for composite objects
	CRC32C          uint32
	MediaLink       string
	Metadata        map[string]string
	Generation      int64
	MetaGeneration  int64
	StorageClass    string
	Deleted         time.Time
	Updated         time.Time

	// NOTE(jacobsa): As of 2015-06-03, the official GCS documentation for this
	// property (https://goo.gl/GwD5Dq) says this:
	//
	//     Newly uploaded objects have a component count of 1, and composing a
	//     sequence of objects creates an object whose component count is equal
	//     to the sum of component counts in the sequence.
	//
	// However, in Google-internal bug 21572928 it was clarified that this
	// doesn't match the actual implementation, which can be documented as:
	//
	//     Newly uploaded objects do not have a component count. Composing a
	//     sequence of objects creates an object whose component count is equal
	//     to the sum of the component counts of the objects in the sequence,
	//     where objects that do not have a component count are treated as having
	//     a component count of 1.
	//
	// This is a much less elegant and convenient rule, so this package emulates
	// the officially documented behavior above. That is, it synthesizes a
	// component count of 1 for objects that do not have a component count.
	ComponentCount int64
}

Object is a record representing a particular generation of a particular object name in GCS.

See here for more information about its fields:

https://cloud.google.com/storage/docs/json_api/v1/objects#resource

type PreconditionError

type PreconditionError struct {
	Err error
}

A *PreconditionError value is an error that indicates a precondition failed.

func (*PreconditionError) Error

func (pe *PreconditionError) Error() string

Returns pe.Err.Error().

type ReadObjectRequest

type ReadObjectRequest struct {
	// The name of the object to read.
	Name string

	// The generation of the object to read. Zero means the latest generation.
	Generation int64

	// If present, limit the contents returned to a range within the object.
	Range *ByteRange
}

A request to read the contents of an object at a particular generation.

type ReadSeekCloser

type ReadSeekCloser interface {
	io.ReadCloser
	io.Seeker
}

type StatObjectRequest

type StatObjectRequest struct {
	// The name of the object in question.
	Name string
}

type UpdateObjectRequest

type UpdateObjectRequest struct {
	// The name of the object to update. Must be specified.
	Name string

	// The generation of the object to update. Zero means the latest generation.
	Generation int64

	// If non-nil, the request will fail without effect if there is an object
	// with the given name (and optionally generation), and its meta-generation
	// is not equal to this value.
	MetaGenerationPrecondition *int64

	// String fields in the object to update (or not). The semantics are as
	// follows, for a given field F:
	//
	//  *  If F is set to nil, the corresponding GCS object field is untouched.
	//
	//  *  If *F is the empty string, then the corresponding GCS object field is
	//     removed.
	//
	//  *  Otherwise, the corresponding GCS object field is set to *F.
	//
	//  *  There is no facility for setting a GCS object field to the empty
	//     string, since many of the fields do not actually allow that as a legal
	//     value.
	//
	// Note that the GCS object's content type field cannot be removed.
	ContentType     *string
	ContentEncoding *string
	ContentLanguage *string
	CacheControl    *string

	// User-provided metadata updates. Keys that are not mentioned are untouched.
	// Keys whose values are nil are deleted, and others are updated to the
	// supplied string. There is no facility for completely removing user
	// metadata.
	Metadata map[string]*string
}

A request to update the metadata of an object, accepted by Bucket.UpdateObject.