gcs

Documentation

Overview

package gcs provides useful utilities for GCS This package isn't specific to our groups sync. We should probably move it to a more generic location and make it its own go module to make it easy to reuse.

Index

• func Base(uri string) string
• func Dir(uri string) string
• func ListObjects(ctx context.Context, client *storage.Client, uri string) ([]string, error)
• func ListObjectsWithPrefix(ctx context.Context, client *storage.Client, prefix string) ([]string, error)
• func ObjectExists(ctx context.Context, o *storage.ObjectHandle) bool
• type FileHelper
• type GcsHelper
  • func (h *GcsHelper) BuildInputOutputList(input string, output string) (map[string]string, error)
  • func (h *GcsHelper) Exists(uri string) (bool, error)
  • func (h *GcsHelper) NewReader(uri string) (io.Reader, error)
  • func (h *GcsHelper) NewWriter(uri string) (io.Writer, error)
• type GcsPath
  • func Parse(path string) (*GcsPath, error)
  • func (p *GcsPath) ToURI() string
• type LocalFileHelper
  • func (h *LocalFileHelper) Exists(uri string) (bool, error)
  • func (h *LocalFileHelper) NewReader(uri string) (io.Reader, error)
  • func (h *LocalFileHelper) NewWriter(uri string) (io.Writer, error)

Functions

func Base

func Base(uri string) string

Base should have the same semantics as Path.Base except it should work with URIs e.g. file://some and gs:// as well as non URIs

func Dir

func Dir(uri string) string

Dir should have the same semantics as Path.Dir except it should work with URIs e.g. file://some and gs:// as well as non URIs

func ListObjects

func ListObjects(ctx context.Context, client *storage.Client, uri string) ([]string, error)

ListObjects lists all objects matching some regex.

This is listing all files in the parent directory.

func ListObjectsWithPrefix

func ListObjectsWithPrefix(ctx context.Context, client *storage.Client, prefix string) ([]string, error)

ListObjectsWithPrefix returns a list of all GCS objects within the given prefix.

func ObjectExists

func ObjectExists(ctx context.Context, o *storage.ObjectHandle) bool

Types

type FileHelper

type FileHelper interface {
	Exists(path string) (bool, error)
	NewReader(path string) (io.Reader, error)
	NewWriter(path string) (io.Writer, error)
}

FileHelper is an interface intended to transparently handle working with GCS and local files. TODO(jlewi): Move into the util package?

type GcsHelper

type GcsHelper struct {
	Ctx    context.Context
	Client *storage.Client
}

func (*GcsHelper) BuildInputOutputList

func (h *GcsHelper) BuildInputOutputList(input string, output string) (map[string]string, error)

BuildInputOutputList builds a map from input files to the files that they should be mapped to.

input is a regex as specified by TransformFiles. This is used to find existing files and generate the corresponding output files.

func (*GcsHelper) Exists

func (h *GcsHelper) Exists(uri string) (bool, error)

Exists checks whether the URI exists.

If error is not nil the boolean value will be random.

func (*GcsHelper) NewReader

func (h *GcsHelper) NewReader(uri string) (io.Reader, error)

NewReader creates a new Reader for GCS path or local file.

func (*GcsHelper) NewWriter

func (h *GcsHelper) NewWriter(uri string) (io.Writer, error)

NewWriter creates a new Writer for GCS path or local file.

TODO(jlewi): Can we add options to control filemode?

type GcsPath

type GcsPath struct {
	Bucket string
	Path   string
}

func Parse

func Parse(path string) (*GcsPath, error)

func (*GcsPath) ToURI

func (p *GcsPath) ToURI() string

type LocalFileHelper

type LocalFileHelper struct{}

func (*LocalFileHelper) Exists

func (h *LocalFileHelper) Exists(uri string) (bool, error)

Exists checks whether the file exists.

func (*LocalFileHelper) NewReader

func (h *LocalFileHelper) NewReader(uri string) (io.Reader, error)

NewReader creates a new Reader for local file.

func (*LocalFileHelper) NewWriter

func (h *LocalFileHelper) NewWriter(uri string) (io.Writer, error)

NewWriter creates a new Writer for the local file.

TODO(jlewi): Can we add options to control filemode?