README

Storage

Build Status GoDoc

storage is a Go package which abstracts file systems (local, in-memory, Google Cloud Storage, S3) into a few interfaces. It includes convenience wrappers for simplifying common file system use cases such as caching, prefix isolation and more!

Requirements

Installation

$ go get code.sajari.com/storage

Usage

For full documentation see: http://godoc.org/code.sajari.com/storage/.

All storage in this package follow two simple interfaces designed for using file systems.

type FS interface {
	Walker

	// Open opens an existing file at path in the filesystem.  Callers must close the
	// File when done to release all underlying resources.
	Open(ctx context.Context, path string) (*File, error)

	// Create makes a new file in the filesystem.  Callers must close the
	// returned WriteCloser and check the error to be sure that the file
	// was successfully written.
	Create(ctx context.Context, path string) (io.WriteCloser, error)

	// Delete removes a file from the filesystem.
	Delete(ctx context.Context, path string) error
}

// WalkFn is a function type which is passed to Walk.
type WalkFn func(path string) error

// Walker is an interface which defines the Walk method.
type Walker interface {
	// Walk traverses a path listing by prefix, calling fn with each object path rewritten
	// to be relative to the underlying filesystem and provided path.
	Walk(ctx context.Context, path string, fn WalkFn) error
}

Local

Local is the default implementation of a local file system (i.e. using os.Open etc).

local := storage.Local("/some/root/path")
f, err := local.Open(context.Background(), "file.json") // will open "/some/root/path/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

Memory

Mem is the default in-memory implementation of a file system.

mem := storage.Mem()
wc, err := mem.Create(context.Background(), "file.txt")
if err != nil {
	// ...
}
if _, err := io.WriteString(wc, "Hello World!"); err != nil {
	// ...
}
if err := wc.Close(); err != nil {
	// ...
}

And now:

f, err := mem.Open(context.Background(), "file.txt")
if err != nil {
	// ...
}
// ...
f.Close()

Google Cloud Storage

CloudStorage is the default implementation of Google Cloud Storage. This uses https://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource for autentication.

store := storage.CloudStorage{Bucket:"some-bucket"}
f, err := store.Open(context.Background(), "file.json") // will fetch "gs://some-bucket/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

S3

Not yet implemented! Watch this space.

Wrappers and Helpers

Simple Caching

To use Cloud Storage as a source file system, but cache all opened files in a local filesystem:

src := storage.CloudStorage{Bucket:"some-bucket"}
local := storage.Local("/scratch-space")

fs := storage.Cache(src, local)
f, err := fs.Open(context.Background(), "file.json") // will try src then jump to cache ("gs://some-bucket/file.json")
if err != nil {
	// ...
}
// ...
f.Close()

f, err := fs.Open(context.Background(), "file.json") // should now be cached ("/scratch-space/file.json")
if err != nil {
	// ...
}
// ...
f.Close()

This is particularly useful when distributing files across multiple regions or between cloud providers. For instance, we could add the following code to the previous example:

mainSrc := storage.CloudStorage{Bucket:"some-bucket-in-another-region"}
fs2 := storage.Cache(mainSrc, fs) // fs is from previous snippet

// Open will:
// 1. Try local (see above)
// 2. Try gs://some-bucket
// 3. Try gs://some-bucket-in-another-region, which will be cached in gs://some-bucket and then local on its
//    way back to the caller.
f, err := fs2.Open(context.Background(), "file.json") // will fetch "gs://some-bucket-in-another-region/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

f, err := fs2.Open(context.Background(), "file.json") // will fetch "/scratch-space/file.json"
if err != nil {
	// ...
}
// ...
f.Close()
Adding prefixes to paths

If you're writing code that relies on a set directory structure, it can be very messy to have to pass path-patterns around. You can avoid this by wrapping storage.FS implementations with storage.Prefix that rewrites all incoming paths.

modelFS := storage.Prefix(rootFS, "models/")
f, err := modelFS.Open(context.Background(), "file.json") // will call rootFS.Open with path "models/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

It's also now simple to write wrapper functions to abstract out more complex directory structures.

func UserFS(fs storage.FS, userID, mediaType string) FS {
	return storage.Prefix(fs, fmt.Sprintf("%v/%v", userID, userType))
}

userFS := UserFS(rootFS, "1111", "pics")
f, err := userFS.Open(context.Background(), "beach.png") // will call rootFS.Open with path "1111/pics/beach.png"
if err != nil {
	// ...
}
// ...
f.Close()
Expand ▾ Collapse ▴

Documentation

Overview

    Package storage provides types and functionality for abstracting storage systems (Local, in memory, S3, Google Cloud storage) into a common interface.

    Index

    Constants

    View Source
    const DefaultLocalCreatePathMode = os.FileMode(0755)

      DefaultLocalCreatePathMode is the default os.FileMode used when creating directories during a Local.Create call.

      Variables

      View Source
      var LocalCreatePathMode = DefaultLocalCreatePathMode

        LocalCreatePathMode is the os.FileMode used when creating directories via Local.Create

        Functions

        func IsNotExist

        func IsNotExist(err error) bool

          IsNotExist returns a boolean indicating whether the error is known to report that a path does not exist.

          func List

          func List(ctx context.Context, w Walker, path string) ([]string, error)

            List runs the Walker on the given path and returns the list of visited paths.

            func WalkN

            func WalkN(ctx context.Context, w Walker, path string, n int, fn WalkFn) error

              WalkN creates n workers which accept paths from the Walker. If a WalkFn returns non-nil error we wait for other running WalkFns to finish before returning.

              Types

              type CloudStorage

              type CloudStorage struct {
              	Bucket string
              }

                CloudStorage implements FS and uses Google Cloud Storage as the underlying file storage.

                func (*CloudStorage) Create

                func (c *CloudStorage) Create(ctx context.Context, path string) (io.WriteCloser, error)

                  Create implements FS.

                  func (*CloudStorage) Delete

                  func (c *CloudStorage) Delete(ctx context.Context, path string) error

                    Delete implements FS.

                    func (*CloudStorage) Open

                    func (c *CloudStorage) Open(ctx context.Context, path string) (*File, error)

                      Open implements FS.

                      func (*CloudStorage) Walk

                      func (c *CloudStorage) Walk(ctx context.Context, path string, fn WalkFn) error

                        Walk implements FS.

                        type ErrCountFS

                        type ErrCountFS struct {
                        	// contains filtered or unexported fields
                        }

                          ErrCountFS is an FS which records error counts for an FS.

                          func NewErrCountFS

                          func NewErrCountFS(fs FS, name string, err error) *ErrCountFS

                            NewErrCountFS creates an FS which records stats based on usage.

                            func (ErrCountFS) Create

                            func (s ErrCountFS) Create(ctx context.Context, path string) (io.WriteCloser, error)

                              Create implements FS. All errors from Create are counted.

                              func (ErrCountFS) Delete

                              func (s ErrCountFS) Delete(ctx context.Context, path string) error

                                Delete implements FS. All errors from Delete are counted.

                                func (ErrCountFS) Open

                                func (s ErrCountFS) Open(ctx context.Context, path string) (*File, error)

                                  Open implements FS. All errors from Open are counted.

                                  func (ErrCountFS) Walk

                                  func (s ErrCountFS) Walk(ctx context.Context, path string, fn WalkFn) error

                                    Walk implements FS. No stats are recorded at this time.

                                    type FS

                                    type FS interface {
                                    	Walker
                                    
                                    	// Open opens an existing file at path in the filesystem.  Callers must close the
                                    	// File when done to release all underlying resources.
                                    	Open(ctx context.Context, path string) (*File, error)
                                    
                                    	// Create makes a new file at path in the filesystem.  Callers must close the
                                    	// returned WriteCloser and check the error to be sure that the file
                                    	// was successfully written.
                                    	Create(ctx context.Context, path string) (io.WriteCloser, error)
                                    
                                    	// Delete removes a path from the filesystem.
                                    	Delete(ctx context.Context, path string) error
                                    }

                                      FS is an interface which defines a virtual filesystem.

                                      func Cache

                                      func Cache(src, cache FS) FS

                                        Cache creates an FS implementation which caches files opened from src into cache.

                                        func FSFromURL

                                        func FSFromURL(path string) FS

                                          FSFromURL takes a file system path and returns a FSWalker corresponding to a supported storage system (CloudStorage, S3, or Local if no platform-specific prefix is used).

                                          func HashFS

                                          func HashFS(h hash.Hash, fs FS, gs GetSetter) FS

                                            HashFS creates a content addressable filesystem using hash.Hash to sum the content and store it using that name.

                                            func Mem

                                            func Mem() FS

                                              Mem creates a a basic in-memory implementation of FS.

                                              func Prefix

                                              func Prefix(fs FS, prefix string) FS

                                                Prefix creates a FS which wraps fs and prefixes all paths with prefix.

                                                type File

                                                type File struct {
                                                	io.ReadCloser           // Underlying data.
                                                	Name          string    // Name of the file (likely basename).
                                                	ModTime       time.Time // Modified time of the file.
                                                	Size          int64     // Size of the file.
                                                }

                                                  File contains the metadata required to define a file (for reading).

                                                  type GetSetter

                                                  type GetSetter interface {
                                                  	Get(key string) (string, error)
                                                  	Set(key string, value string) error
                                                  	Delete(key string) error
                                                  }

                                                    GetSetter implements a key-value store which is concurrency safe (can be used in multiple go-routines concurrently).

                                                    type Local

                                                    type Local string

                                                      Local is a local FS and Walker implementation.

                                                      func (Local) Create

                                                      func (l Local) Create(_ context.Context, path string) (io.WriteCloser, error)

                                                        Create implements FS. If the path contains any directories which do not already exist then Create will try to make them, returning an error if it fails.

                                                        func (Local) Delete

                                                        func (l Local) Delete(_ context.Context, path string) error

                                                          Delete implements FS. All files underneath path will be removed.

                                                          func (Local) Open

                                                          func (l Local) Open(_ context.Context, path string) (*File, error)

                                                            Open implements FS.

                                                            func (Local) Walk

                                                            func (l Local) Walk(_ context.Context, path string, fn WalkFn) error

                                                              Walk implements Walker.

                                                              type LogFS

                                                              type LogFS struct {
                                                              	// contains filtered or unexported fields
                                                              }

                                                                LogFS is an FS implementation which logs all filesystem calls.

                                                                func NewLogFS

                                                                func NewLogFS(fs FS, name string, l *log.Logger) *LogFS

                                                                  NewLogFS creates a new FS which logs all calls to FS.

                                                                  func (*LogFS) Create

                                                                  func (l *LogFS) Create(ctx context.Context, path string) (io.WriteCloser, error)

                                                                    Create implements FS. All calls to Create are logged and errors are logged seperately.

                                                                    func (*LogFS) Delete

                                                                    func (l *LogFS) Delete(ctx context.Context, path string) error

                                                                      Delete implements FS. All calls to Delete are logged and errors are logged seperately.

                                                                      func (*LogFS) Open

                                                                      func (l *LogFS) Open(ctx context.Context, path string) (*File, error)

                                                                        Open implements FS. All calls to Open are logged and errors are logged seperately.

                                                                        func (*LogFS) Walk

                                                                        func (l *LogFS) Walk(ctx context.Context, path string, fn WalkFn) error

                                                                          Walk implements FS. No logs are written at this time.

                                                                          type S3

                                                                          type S3 struct {
                                                                          	Bucket string // Bucket is the name of the bucket to use as the underlying storage.
                                                                          }

                                                                            S3 is an implementation of FS which uses AWS S3 as the underlying storage layer.

                                                                            func (*S3) Create

                                                                            func (s *S3) Create(ctx context.Context, path string) (io.WriteCloser, error)

                                                                              Create implements FS.

                                                                              func (*S3) Delete

                                                                              func (s *S3) Delete(ctx context.Context, path string) error

                                                                                Delete implements FS.

                                                                                func (*S3) Open

                                                                                func (s *S3) Open(ctx context.Context, path string) (*File, error)

                                                                                  Open implements FS.

                                                                                  func (*S3) Walk

                                                                                  func (s *S3) Walk(ctx context.Context, path string, fn WalkFn) error

                                                                                    Walk implements FS.

                                                                                    type TraceFS

                                                                                    type TraceFS struct {
                                                                                    	// contains filtered or unexported fields
                                                                                    }

                                                                                      TraceFS is a FS implementation which wraps an FS and records calls using golang.org/x/net/trace.

                                                                                      func NewTraceFS

                                                                                      func NewTraceFS(fs FS, name string) *TraceFS

                                                                                        NewTraceFS creates a new FS which wraps an FS and records calls using golang.org/x/net/trace.

                                                                                        func (*TraceFS) Create

                                                                                        func (t *TraceFS) Create(ctx context.Context, path string) (wc io.WriteCloser, err error)

                                                                                          Create implements FS. All calls to Create are logged via golang.org/x/net/trace.

                                                                                          func (*TraceFS) Delete

                                                                                          func (t *TraceFS) Delete(ctx context.Context, path string) (err error)

                                                                                            Delete implements FS. All calls to Delete are logged via golang.org/x/net/trace.

                                                                                            func (*TraceFS) Open

                                                                                            func (t *TraceFS) Open(ctx context.Context, path string) (f *File, err error)

                                                                                              Open implements FS. All calls to Open are logged via golang.org/x/net/trace.

                                                                                              func (*TraceFS) Walk

                                                                                              func (t *TraceFS) Walk(ctx context.Context, path string, fn WalkFn) error

                                                                                                Walk implements FS. Nothing is traced at this time.

                                                                                                type WalkFn

                                                                                                type WalkFn func(path string) error

                                                                                                  WalkFn is a function type which is passed to Walk.

                                                                                                  type Walker

                                                                                                  type Walker interface {
                                                                                                  	// Walk traverses a path listing by prefix, calling fn with each path.
                                                                                                  	Walk(ctx context.Context, path string, fn WalkFn) error
                                                                                                  }

                                                                                                    Walker is an interface which defines the Walk method.