storage

README

Storage - read/write utility for Google Cloud Platform Storage buckets.

A Go utility to facilitate reading/writing files from/to Google Cloud Platform Storage.

What?

This utility uses Google Application Credentials or GCP service accounts to connect to a Google Cloud Platform project and read/write files from/to Storage buckets.

Why?

This was written to provide GCP storage bucket functionality within auth. It wraps the Google provided Storage Go client.

How?

The best place to start is with the tests. If running locally, then ensure that Google Application Credentials have been created. If running from a GCP virtual machine, then ensure that the relevant service account (compute, appengine etc.) has the following IAM scopes: 'Storage Object Viewer' and 'Storage Object Creator', or 'Storage Object Admin'. See GCP service accounts for further details.

Examples

See the tests for usage examples.

Dependencies and services

This utilises the following fine pieces of work:

• GCP's Datastore Go client and Storage Go client

Installation

Install with

$ go get -u github.com/lidstromberg/storage

Environment Variables

You will also need to export (linux/macOS) or create (Windows) some environment variables.

Change LB_DEBUGON to true/false if you want verbose logging on/off. The other variables don't need to be changed.

################################
# STORAGE
################################
export STOR_DEBUGON='true'
export STOR_CLIPOOL='5'

################################
# GCP CREDENTIALS
################################
export GOOGLE_APPLICATION_CREDENTIALS="/PATH/TO/GCPCREDENTIALS.JSON"

(See Google Application Credentials)

Main Files

File	Purpose
storage.go	Logic manager
storage_test.go	Tests

Ancillary Files

File	Purpose
config.go	Boot package parameters, environment var collection
const.go	Package constants
errors.go	Package error definitions
env	Package environment variables for local/dev installation
storagetester.json	test content file
gogets	Statements for go-getting required packages

Documentation

Index

• Constants
• Variables
• func DrainFn(c <-chan interface{})
• type StorMgr
  • func NewJSONMgr(ctx context.Context, bc lbcf.ConfigSetting, cred []byte) (*StorMgr, error)
  • func NewMgr(ctx context.Context, bc lbcf.ConfigSetting) (*StorMgr, error)
  • func (sto *StorMgr) GetBucketFileData(ctx context.Context, bucketName string, fileName string) ([]byte, error)
  • func (sto *StorMgr) ListBucket(ctx context.Context, bucketName, prefix string, bufferSize int) (<-chan interface{}, error)
  • func (sto *StorMgr) ListBucketByTime(ctx context.Context, bucketName, prefix string, start, end *time.Time, ...) (<-chan interface{}, error)
  • func (sto *StorMgr) RemoveFile(ctx context.Context, bucketName string, fileName string) error
  • func (sto *StorMgr) WriteBucketFile(ctx context.Context, bucketName string, fileName string, data []byte) error

Constants

const (
	//ObjAttrName is the object name
	ObjAttrName = "name"
	//ObjAttrContentType is the content type
	ObjAttrContentType = "contenttype"
	//ObjAttrOwner is the GCP owner account
	ObjAttrOwner = "owner"
	//ObjAttrSize is the size of the object
	ObjAttrSize = "size"
	//ObjAttrContentEncoding is the file encoding
	ObjAttrContentEncoding = "contentencoding"
	//ObjAttrCreated created timestamp
	ObjAttrCreated = "created"
)

Variables

var (
	//EnvDebugOn controls verbose logging
	EnvDebugOn bool
	//EnvClientPool is the size of the client pool
	EnvClientPool int
)

var (
	//ErrMissingDateRange  message
	ErrMissingDateRange = errors.New("start and end dates must be supplied")
)

Functions

func DrainFn

func DrainFn(c <-chan interface{})

DrainFn drains a channel until it is closed

Types

type StorMgr

type StorMgr struct {
	// contains filtered or unexported fields
}

StorMgr handles interactions with GCS

func NewJSONMgr

func NewJSONMgr(ctx context.Context, bc lbcf.ConfigSetting, cred []byte) (*StorMgr, error)

NewJSONMgr returns a new storage manager based on a GCP credential supplied as a byte array

func NewMgr

func NewMgr(ctx context.Context, bc lbcf.ConfigSetting) (*StorMgr, error)

NewMgr returns a new storage manager

func (*StorMgr) GetBucketFileData

func (sto *StorMgr) GetBucketFileData(ctx context.Context, bucketName string, fileName string) ([]byte, error)

GetBucketFileData returns a byte array for a bucket file

func (*StorMgr) ListBucket

func (sto *StorMgr) ListBucket(ctx context.Context, bucketName, prefix string, bufferSize int) (<-chan interface{}, error)

ListBucket returns a configurable buffered channel which contains a subset of object metadata

func (*StorMgr) ListBucketByTime

func (sto *StorMgr) ListBucketByTime(ctx context.Context, bucketName, prefix string, start, end *time.Time, bufferSize int) (<-chan interface{}, error)

ListBucketByTime returns a configurable buffered channel which contains a subset of object metadata

func (*StorMgr) RemoveFile

func (sto *StorMgr) RemoveFile(ctx context.Context, bucketName string, fileName string) error

RemoveFile deletes a bucket file

func (*StorMgr) WriteBucketFile

func (sto *StorMgr) WriteBucketFile(ctx context.Context, bucketName string, fileName string, data []byte) error

WriteBucketFile writes a file byte array to a bucket file