stow

README

Cloud storage abstraction package for Go.

• Version: 0.1.0

• Project status: Stable. Approaching v1 release

• It is recommended that you vendor this package as changes are possible before v1

• Contributors: Mat Ryer, David Hernandez, Ernesto Jiménez, Corey Prak, Piotr Rojek, Jason Hancock

• Artwork by Mel Jensen inspired by Renee French and and is licensed under the Creative Commons Attribution 3.0 License

How it works

Stow provides implementations for storage services, blob stores, cloud storage etc. Read the blog post announcing the project.

Implementations

• Local (folders are containers, files are items)
• Amazon S3
• Google Cloud Storage
• Microsoft Azure Blob Storage
• Openstack Swift (with auth v2)
• Oracle Storage Cloud Service
• SFTP

Concepts

The concepts of Stow are modeled around the most popular object storage services, and are made up of three main objects:

• Location - a place where many Container objects are stored
• Container - a named group of Item objects
• Item - an individual file

location1 (e.g. Azure)
├── container1
├───── item1.1
├───── item1.2
├───── item1.3
├── container2
├───── item2.1
├───── item2.2
location2 (e.g. local storage)
├── container1
├───── item1.1
├───── item1.2
├───── item1.3
├── container2
├───── item2.1
├───── item2.2

• A location contains many containers
• A container contains many items
• Containers do not contain other containers
• Items must belong to a container
• Item names may be a path

Guides

• Using Stow
• Connecting to locations
• Walking containers
• Walking items
• Downloading a file
• Uploading a file
• Stow URLs
• Cursors

Using Stow

Import Stow plus any of the implementation packages that you wish to provide. For example, to support Google Cloud Storage and Amazon S3 you would write:

import (
	"github.com/graymeta/stow"
	_ "github.com/graymeta/stow/google"
	_ "github.com/graymeta/stow/s3"
)

The underscore indicates that you do not intend to use the package in your code. Importing it is enough, as the implementation packages register themselves with Stow during initialization.

• For more information about using Stow, see the Best practices documentation.
• Some implementation packages provide ways to access the underlying connection details for use-cases where more control over a specific service is needed. See the implementation package documentation for details.

Connecting to locations

To connect to a location, you need to know the kind string (available by accessing the Kind constant in the implementation package) and a stow.Config object that contains any required configuration information (such as account names, API keys, credentials, etc). Configuration is implementation specific, so you should consult each implementation to see what fields are required.

kind := "s3"
config := stow.ConfigMap{
	s3.ConfigAccessKeyID: "246810",
	s3.ConfigSecretKey:   "abc123",
	s3.ConfigRegion:      "eu-west-1",
}
location, err := stow.Dial(kind, config)
if err != nil {
	return err
}
defer location.Close()

// TODO: use location

Walking containers

You can walk every Container using the stow.WalkContainers function:

func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error

For example:

err = stow.WalkContainers(location, stow.NoPrefix, 100, func(c stow.Container, err error) error {
	if err != nil {
		return err
	}
	switch c.Name() {
	case c1.Name(), c2.Name(), c3.Name():
		found++
	}
	return nil
})
if err != nil {
	return err
}

Walking items

Once you have a Container, you can walk every Item inside it using the stow.Walk function:

func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error

For example:

err = stow.Walk(containers[0], stow.NoPrefix, 100, func(item stow.Item, err error) error {
	if err != nil {
		return err
	}
	log.Println(item.Name())
	return nil
})
if err != nil {
	return err
}

Downloading a file

Once you have found a stow.Item that you are interested in, you can stream its contents by first calling the Open method and reading from the returned io.ReadCloser (remembering to close the reader):

r, err := item.Open()
if err != nil {
    return err
}
defer r.Close()

// TODO: stream the contents by reading from r

Uploading a file

If you want to write a new item into a Container, you can do so using the container.Put method passing in an io.Reader for the contents along with the size:

contents := "This is a new file stored in the cloud"
r := strings.NewReader(contents)
size := int64(len(contents))

item, err := container.Put(name, r, size, nil)
if err != nil {
    return err
}

// item represents the newly created/updated item

Stow URLs

An Item can return a URL via the URL() method. While a valid URL, they are useful only within the context of Stow. Within a Location, you can get items using these URLs via the Location.ItemByURL method.

Getting an Item by URL

If you have a Stow URL, you can use it to lookup the kind of location:

kind, err := stow.KindByURL(url)

kind will be a string describing the kind of storage. You can then pass kind along with a Config to stow.New to create a new Location where the item for the URL is:

location, err := stow.Dial(kind, config)

You can then get the Item for the specified URL from the location:

item, err := location.ItemByURL(url)

Cursors

Cursors are strings that provide a pointer to items in sets allowing for paging over the entire set.

Call such methods first passing in stow.CursorStart as the cursor, which indicates the first item/page. The method will, as one of its return arguments, provide a new cursor which you can pass into subsequent calls to the same method.

When stow.IsCursorEnd(cursor) returns true, you have reached the end of the set.

Documentation

Overview

Package stow provides an abstraction on cloud storage capabilities.

Index

• Variables
• func IsCursorEnd(cursor string) bool
• func IsNotSupported(err error) bool
• func KindByURL(u *url.URL) (string, error)
• func Kinds() []string
• func NotSupported(feature string) error
• func Register(kind string, makefn func(Config) (Location, error), ...)
• func Validate(kind string, config Config) error
• func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error
• func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error
• type ClientMethod
  • func ClientMethodString(s string) (ClientMethod, error)
  • func ClientMethodValues() []ClientMethod
  • func (i ClientMethod) IsAClientMethod() bool
  • func (i ClientMethod) MarshalJSON() ([]byte, error)
  • func (i ClientMethod) String() string
  • func (i *ClientMethod) UnmarshalJSON(data []byte) error
• type Config
• type ConfigMap
  • func (c ConfigMap) Config(name string) (string, bool)
  • func (c ConfigMap) Set(name, value string)
• type Container
• type HttpMethod
• type Item
• type ItemRanger
• type Location
  • func Dial(kind string, config Config) (Location, error)
• type PresignRequestParams
• type Taggable
• type WalkContainersFunc
• type WalkFunc

Variables

var (
	// ErrNotFound is returned when something could not be found.
	ErrNotFound = errors.New("not found")
	// ErrBadCursor is returned by paging methods when the specified
	// cursor is invalid.
	ErrBadCursor = errors.New("bad cursor")
)

var (
	// CursorStart is a string representing a cursor pointing
	// to the first page of items or containers.
	CursorStart = ""

	// NoPrefix is a string representing no prefix. It can be used
	// in any function that asks for a prefix value, but where one is
	// not appropriate.
	NoPrefix = ""
)

Functions

func IsCursorEnd

func IsCursorEnd(cursor string) bool

IsCursorEnd checks whether the cursor indicates there are no more items or not.

func IsNotSupported

func IsNotSupported(err error) bool

IsNotSupported gets whether the error is due to a feature not being supported by a specific implementation.

func KindByURL

func KindByURL(u *url.URL) (string, error)

KindByURL gets the kind represented by the given URL. It consults all registered locations. Error returned if no match is found.

func Kinds

func Kinds() []string

Kinds gets a list of installed location kinds.

func NotSupported

func NotSupported(feature string) error

NotSupported gets an error describing the feature as not supported by this implementation.

func Register

func Register(kind string, makefn func(Config) (Location, error), kindmatchfn func(*url.URL) bool, validatefn func(Config) error)

Register adds a Location implementation, with two helper functions. makefn should make a Location with the given Config. kindmatchfn should inspect a URL and return whether it represents a Location of this kind or not. Code can call KindByURL to get a kind string for any given URL and all registered implementations will be consulted. Register is usually called in an implementation package's init method.

func Validate

func Validate(kind string, config Config) error

Validate validates the config for a location

func Walk

func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error

Walk walks all Items in the Container. Returns the first error returned by the WalkFunc or nil if no errors were returned. The pageSize is the number of Items to get per request.

func WalkContainers

func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error

WalkContainers walks all Containers in the Location. Returns the first error returned by the WalkContainersFunc or nil if no errors were returned. The pageSize is the number of Containers to get per request.

Types

type ClientMethod

type ClientMethod int

ClientMethod defines common client methods across storage providers

const (
	ClientMethodGet ClientMethod = iota
	ClientMethodPut
)

func ClientMethodString

func ClientMethodString(s string) (ClientMethod, error)

ClientMethodString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func ClientMethodValues

func ClientMethodValues() []ClientMethod

ClientMethodValues returns all values of the enum

func (ClientMethod) IsAClientMethod

func (i ClientMethod) IsAClientMethod() bool

IsAClientMethod returns "true" if the value is listed in the enum definition. "false" otherwise

func (ClientMethod) MarshalJSON

func (i ClientMethod) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for ClientMethod

func (ClientMethod) String

func (i ClientMethod) String() string

func (*ClientMethod) UnmarshalJSON

func (i *ClientMethod) UnmarshalJSON(data []byte) error

UnmarshalJSON implements the json.Unmarshaler interface for ClientMethod

type Config

type Config interface {
	// Config gets a string configuration value and a
	// bool indicating whether the value was present or not.
	Config(name string) (string, bool)
	// Set sets the configuration name to specified value
	Set(name, value string)
}

Config represents key/value configuration.

type ConfigMap

type ConfigMap map[string]string

ConfigMap is a map[string]string that implements the Config method.

func (ConfigMap) Config

func (c ConfigMap) Config(name string) (string, bool)

Config gets a string configuration value and a bool indicating whether the value was present or not.

func (ConfigMap) Set

func (c ConfigMap) Set(name, value string)

Set sets name configuration to value

type Container

type Container interface {
	// ID gets a unique string describing this Container.
	ID() string
	// Name gets a human-readable name describing this Container.
	Name() string
	// Item gets an item by its ID.
	Item(id string) (Item, error)
	// Items gets a page of items with the specified
	// prefix for this Container.
	// The specified cursor is a pointer to the start of
	// the items to get. It should be obtained from a previous
	// call to this method, or should be CursorStart for the
	// first page.
	// count is the number of items to return per page.
	// The returned cursor can be checked with IsCursorEnd to
	// decide if there are any more items or not.
	Items(prefix, cursor string, count int) ([]Item, string, error)
	// RemoveItem removes the Item with the specified ID.
	RemoveItem(id string) error
	// Put creates a new Item with the specified name, and contents
	// read from the reader.
	Put(name string, r io.Reader, size int64, metadata map[string]interface{}) (Item, error)
	// PreSignRequest generates a pre-signed url for the given id (key after bucket/container) and a given clientMethod.
	PreSignRequest(ctx context.Context, clientMethod ClientMethod, id string, params PresignRequestParams) (url string, err error)
}

Container represents a container.

type HttpMethod

type HttpMethod = string

HttpMethod defines an alias type for string to represent http Methods. These are defined in RFC 7231 section 4.3.

type Item

type Item interface {
	// ID gets a unique string describing this Item.
	ID() string
	// Name gets a human-readable name describing this Item.
	Name() string
	// URL gets a URL for this item.
	// For example:
	// local: file:///path/to/something
	// azure: azure://host:port/api/something
	//    s3: s3://host:post/etc
	URL() *url.URL
	// Size gets the size of the Item's contents in bytes.
	Size() (int64, error)
	// Open opens the Item for reading.
	// Calling code must close the io.ReadCloser.
	Open() (io.ReadCloser, error)
	// ETag is a string that is different when the Item is
	// different, and the same when the item is the same.
	// Usually this is the last modified datetime.
	ETag() (string, error)
	// LastMod returns the last modified date of the file.
	LastMod() (time.Time, error)
	// Metadata gets a map of key/values that belong
	// to this Item.
	Metadata() (map[string]interface{}, error)
}

Item represents an item inside a Container. Such as a file.

type ItemRanger

type ItemRanger interface {
	// OpenRange opens the item for reading starting at byte start and ending
	// at byte end.
	OpenRange(start, end uint64) (io.ReadCloser, error)
}

ItemRanger represents an item that can be partially downloaded.

type Location

type Location interface {
	io.Closer
	// CreateContainer creates a new Container with the
	// specified name.
	CreateContainer(name string) (Container, error)
	// Containers gets a page of containers
	// with the specified prefix from this Location.
	// The specified cursor is a pointer to the start of
	// the containers to get. It it obtained from a previous
	// call to this method, or should be CursorStart for the
	// first page.
	// count is the number of items to return per page.
	// The returned cursor can be checked with IsCursorEnd to
	// decide if there are any more items or not.
	Containers(prefix string, cursor string, count int) ([]Container, string, error)
	// Container gets the Container with the specified
	// identifier.
	Container(id string) (Container, error)
	// RemoveContainer removes the container with the specified ID.
	RemoveContainer(id string) error
	// ItemByURL gets an Item at this location with the
	// specified URL.
	ItemByURL(url *url.URL) (Item, error)
}

Location represents a storage location.

func Dial

func Dial(kind string, config Config) (Location, error)

Dial gets a new Location with the given kind and configuration.

type PresignRequestParams

type PresignRequestParams struct {
	ExpiresIn   time.Duration
	ContentMD5  string
	ExtraParams map[string]interface{}
	HttpMethod  HttpMethod
}

type Taggable

type Taggable interface {
	// Tags returns a list of tags that belong to a given Item
	Tags() (map[string]interface{}, error)
}

Taggable represents a taggable Item

type WalkContainersFunc

type WalkContainersFunc func(container Container, err error) error

WalkContainersFunc is a function called for each Container visited by WalkContainers. If there was a problem, the incoming error will describe the problem and the function can decide how to handle that error. If an error is returned, processing stops.

type WalkFunc

type WalkFunc func(item Item, err error) error

WalkFunc is a function called for each Item visited by Walk. If there was a problem, the incoming error will describe the problem and the function can decide how to handle that error. If an error is returned, processing stops.