gcs

package

v0.0.0-...-03d6fc4 Latest Latest Go to latest Published: Jan 23, 2019 License: BSD-3-Clause Imports: 23 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/google/skia-buildbot

Documentation ¶

Overview ¶

Package gs implements utility for accessing data in Google Storage.

Index ¶

Constants
Variables
func AllFilesInDir(s *storage.Client, bucket, folder string, ...) error
func DeleteAllFilesInDir(s *storage.Client, bucket, folder string, processes int) error
func DownloadTestDataArchive(t assert.TestingT, bucket, gsPath, targetDir string) error
func DownloadTestDataFile(t assert.TestingT, bucket, gsPath, targetPath string) error
func FileContentsFromGCS(s *storage.Client, bucketName, fileName string) ([]byte, error)
func RequestForStorageURL(url string) (*http.Request, error)
func SplitGSPath(path string) (string, string)
func WriteObj(o *storage.ObjectHandle, content []byte) (err error)
type DownloadHelper
- func NewDownloadHelper(s *storage.Client, gsBucket, gsSubdir, workdir string) *DownloadHelper
type FileWriteOptions
type GCSClient
- func NewGCSClient(s *storage.Client, bucket string) GCSClient
type MemoryGCSClient
- func NewMemoryGCSClient(bucket string) *MemoryGCSClient

Constants ¶

View Source

const (
	// GCS bucket where we store test data. Add a folder to this bucket
	// with the tests for a particular component.
	TEST_DATA_BUCKET = "skia-infra-testdata"
)

Variables ¶

View Source

var FILE_WRITE_OPTS_TEXT = FileWriteOptions{ContentEncoding: "text/plain"}

Functions ¶

func AllFilesInDir ¶

func AllFilesInDir(s *storage.Client, bucket, folder string, callback func(item *storage.ObjectAttrs)) error

AllFilesInDir synchronously iterates through all the files in a given Google Storage folder. The callback function is called on each item in the order it is in the bucket. It returns an error if the bucket or folder cannot be accessed.

func DeleteAllFilesInDir ¶

func DeleteAllFilesInDir(s *storage.Client, bucket, folder string, processes int) error

DeleteAllFilesInDir deletes all the files in a given folder. If processes is set to > 1, that many go routines will be spun up to delete the file simultaneously. Otherwise, it will be done one one process.

func DownloadTestDataArchive ¶

func DownloadTestDataArchive(t assert.TestingT, bucket, gsPath, targetDir string) error

DownloadTestDataArchive downloads testfiles that are stored in a gz compressed tar archive and decompresses them into the provided target directory.

func DownloadTestDataFile ¶

func DownloadTestDataFile(t assert.TestingT, bucket, gsPath, targetPath string) error

DownloadTestDataFile downloads a file with test data from Google Storage. The uriPath identifies what to download from the test bucket in GCS. The content must be publicly accessible. The file will be downloaded and stored at provided target path (regardless of what the original name is). If the the uri ends with '.gz' it will be transparently unzipped.

func FileContentsFromGCS ¶

func FileContentsFromGCS(s *storage.Client, bucketName, fileName string) ([]byte, error)

FileContentsFromGCS returns the contents of a file in the given bucket or an error.

func RequestForStorageURL ¶

func RequestForStorageURL(url string) (*http.Request, error)

RequestForStorageURL returns an http.Request for a given Cloud Storage URL. This is workaround of a known issue: embedded slashes in URLs require use of URL.Opaque property

func SplitGSPath ¶

func SplitGSPath(path string) (string, string)

SplitGSPath takes a GCS path and splits it into a <bucket,path> pair. It assumes the format: {bucket_name}/{path_within_bucket}.

func WriteObj ¶

func WriteObj(o *storage.ObjectHandle, content []byte) (err error)

Write the given content to the given object in Google Storage.

Types ¶

type DownloadHelper ¶

type DownloadHelper struct {
	// contains filtered or unexported fields
}

DownloadHelper provides convenience methods for downloading binaries by SHA1 sum.

func NewDownloadHelper ¶

func NewDownloadHelper(s *storage.Client, gsBucket, gsSubdir, workdir string) *DownloadHelper

NewDownloadHelper returns a DownloadHelper instance.

func (*DownloadHelper) Close ¶

func (d *DownloadHelper) Close() error

Close should be called when finished with the DownloadHelper.

func (*DownloadHelper) Download ¶

func (d *DownloadHelper) Download(name, hash string) error

Download downloads the given binary from Google Storage.

func (*DownloadHelper) MaybeDownload ¶

func (d *DownloadHelper) MaybeDownload(name, hash string) error

MaybeDownload downloads the given binary from Google Storage if necessary.

type FileWriteOptions ¶

type FileWriteOptions struct {
	ContentEncoding    string
	ContentType        string
	ContentLanguage    string
	ContentDisposition string
	Metadata           map[string]string
}

FileWriteOptions represents the metadata for a GCS file. See storage.ObjectAttrs for a more detailed description of what these are.

type GCSClient ¶

type GCSClient interface {
	// FileReader returns an io.ReadCloser pointing to path on GCS, using the provided
	// context. storage.ErrObjectNotExist will be returned if the file is not found.
	// The caller must call Close on the returned Reader when done reading.
	FileReader(ctx context.Context, path string) (io.ReadCloser, error)
	// FileReader returns an io.WriteCloser that writes to the GCS file given by path
	// using the provided context. A new GCS file will be created if it doesn't already exist.
	// Otherwise, the existing file will be overwritten. The caller must call Close on
	// the returned Writer to flush the writes.
	FileWriter(ctx context.Context, path string, opts FileWriteOptions) io.WriteCloser
	// DoesFileExist returns true if the specified path exists and false if it does not.
	// This is a convenience wrapper around
	// https://godoc.org/cloud.google.com/go/storage#ObjectHandle.Attrs
	// If any error, other than storage.ErrObjectNotExist, is encountered then it will be
	// returned.
	DoesFileExist(ctx context.Context, path string) (bool, error)
	// GetFileContents returns the []byte represented by the GCS file at path. This is a
	// convenience wrapper around FileReader. storage.ErrObjectNotExist will be returned
	// if the file is not found.
	GetFileContents(ctx context.Context, path string) ([]byte, error)
	// SetFileContents writes the []byte to the GCS file at path. This is a
	// convenience wrapper around FileWriter. The GCS file will be created if it doesn't exist.
	SetFileContents(ctx context.Context, path string, opts FileWriteOptions, contents []byte) error
	// AllFilesInDirectory executes the callback on all GCS files with the given prefix,
	// i.e. in the directory prefix. It returns an error if it fails to read any of the
	// ObjectAttrs belonging to files.
	AllFilesInDirectory(ctx context.Context, prefix string, callback func(item *storage.ObjectAttrs)) error
	// DeleteFile deletes the given file, returning any error.
	DeleteFile(ctx context.Context, path string) error
	// Bucket() returns the bucket name of this client
	Bucket() string
}

GCSClient is an interface for interacting with Google Cloud Storage (GCS). Introducing the interface allows for easier mocking and testing for unit (small) tests. GCSClient should have common, general functionality. Users should feel free to create a instance-specific GCSClient that creates an abstraction for more instance-specific method calls (see fuzzer for an example). One intentional thing missing from these method calls is bucket name. The bucket name is given at creation time, so as to simplify the method signatures. In all methods, context.Background() is a safe value for ctx if you don't want to use the context of the web request, for example. See also mock_gcs_client.New() for mocking this for unit tests.

func NewGCSClient ¶

func NewGCSClient(s *storage.Client, bucket string) GCSClient

NewGCSClient returns a GCSClient. See the interface for more information.

type MemoryGCSClient ¶

type MemoryGCSClient struct {
	// contains filtered or unexported fields
}

MemoryGCSClient is a struct used for testing. Instead of writing to GCS, it stores data in memory. Not thread-safe.

func NewMemoryGCSClient ¶

func NewMemoryGCSClient(bucket string) *MemoryGCSClient

Return a MemoryGCSClient instance.