pathio

package module
Version: v3.7.1+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 9, 2019 License: Apache-2.0 Imports: 11 Imported by: 0

README

pathio

GoDoc

-- import "gopkg.in/Clever/pathio.v3"

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Usage

Pathio has a very easy to use interface, with 5 main functions:

import "gopkg.in/Clever/pathio.v3"
var err error
ListFiles
// func ListFiles(path string) ([]string, error)
files, err = pathio.ListFiles("s3://bucket/my/key") // s3
files, err = pathio.ListFiles("/home/me")           // local
Write / WriteReader
// func Write(path string, input []byte) error
toWrite := []byte("hello world\n")
err = pathio.Write("s3://bucket/my/key", toWrite)   // s3
err = pathio.Write("/home/me/hello_world", toWrite) // local

// func WriteReader(path string, input io.ReadSeeker) error
toWriteReader, err := os.Open("test.txt") // this implements Read and Seek
err = pathio.WriteReader("s3://bucket/my/key", toWriteReader)   // s3
err = pathio.WriteReader("/home/me/hello_world", toWriteReader) // local
Read
// func Reader(path string) (rc io.ReadCloser, err error)
reader, err = pathio.Reader("s3://bucket/key/to/read") // s3
reader, err = pathio.Reader("/home/me/file/to/read")   // local
Delete
// func Delete(path string) error
err = pathio.Delete("s3://bucket/key/to/read") // s3
err = pathio.Delete("/home/me/file/to/read")   // local

Documentation

Overview

Package pathio is a generated GoMock package.

Package pathio is a package that allows writing to and reading from different types of paths transparently. It supports two types of paths:

1. Local file paths
2. S3 File Paths (s3://bucket/key)

Note that using s3 paths requires setting two environment variables

1. AWS_SECRET_ACCESS_KEY
2. AWS_ACCESS_KEY_ID

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Delete

func Delete(path string) error

Delete calls DefaultClient's Delete method.

func Exists

func Exists(path string) (bool, error)

Exists calls DefaultClient's Exists method.

func ListFiles

func ListFiles(path string) ([]string, error)

ListFiles calls DefaultClient's ListFiles method.

func Reader

func Reader(path string) (rc io.ReadCloser, err error)

Reader calls DefaultClient's Reader method.

func Write

func Write(path string, input []byte) error

Write calls DefaultClient's Write method.

func WriteReader

func WriteReader(path string, input io.ReadSeeker) error

WriteReader calls DefaultClient's WriteReader method.

Types

type Client

type Client struct {
	Region string
	// contains filtered or unexported fields
}

Client is the pathio client used to access the local file system and S3. To configure options on the client, create a new Client and call its methods directly.

&Client{
	disableS3Encryption: true, // disables encryption
	Region: "us-east-1", // hardcodes the s3 region, instead of looking it up
}.Write(...)

func NewClient

func NewClient(cfg *aws.Config) *Client

NewClient creates a new client that utilizes the provided AWS config. This can be leveraged to enforce more limited permissions.

func (*Client) Delete

func (c *Client) Delete(path string) error

Delete deletes the object at the specified path. The path can be either a local file path or an S3 path.

func (*Client) Exists

func (c *Client) Exists(path string) (bool, error)

Exists determines if a path does or does not exist. NOTE: S3 is eventually consistent so keep in mind that there is a delay.

func (*Client) ListFiles

func (c *Client) ListFiles(path string) ([]string, error)

ListFiles lists all the files/directories in the directory. It does not recurse

func (*Client) Reader

func (c *Client) Reader(path string) (rc io.ReadCloser, err error)

Reader returns an io.Reader for the specified path. The path can either be a local file path or an S3 path. It is the caller's responsibility to close rc.

func (*Client) Write

func (c *Client) Write(path string, input []byte) error

Write writes a byte array to the specified path. The path can be either a local file path or an S3 path.

func (*Client) WriteReader

func (c *Client) WriteReader(path string, input io.ReadSeeker) error

WriteReader writes all the data read from the specified io.Reader to the output path. The path can either a local file path or an S3 path.

type MockPathio

type MockPathio struct {
	// contains filtered or unexported fields
}

MockPathio is a mock of Pathio interface

func NewMockPathio

func NewMockPathio(ctrl *gomock.Controller) *MockPathio

NewMockPathio creates a new mock instance

func (*MockPathio) Delete

func (m *MockPathio) Delete(path string) error

Delete mocks base method

func (*MockPathio) EXPECT

func (m *MockPathio) EXPECT() *MockPathioMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockPathio) Exists

func (m *MockPathio) Exists(path string) (bool, error)

Exists mocks base method

func (*MockPathio) ListFiles

func (m *MockPathio) ListFiles(path string) ([]string, error)

ListFiles mocks base method

func (*MockPathio) Reader

func (m *MockPathio) Reader(path string) (io.ReadCloser, error)

Reader mocks base method

func (*MockPathio) Write

func (m *MockPathio) Write(path string, input []byte) error

Write mocks base method

func (*MockPathio) WriteReader

func (m *MockPathio) WriteReader(path string, input io.ReadSeeker) error

WriteReader mocks base method

type MockPathioMockRecorder

type MockPathioMockRecorder struct {
	// contains filtered or unexported fields
}

MockPathioMockRecorder is the mock recorder for MockPathio

func (*MockPathioMockRecorder) Delete

func (mr *MockPathioMockRecorder) Delete(path interface{}) *gomock.Call

Delete indicates an expected call of Delete

func (*MockPathioMockRecorder) Exists

func (mr *MockPathioMockRecorder) Exists(path interface{}) *gomock.Call

Exists indicates an expected call of Exists

func (*MockPathioMockRecorder) ListFiles

func (mr *MockPathioMockRecorder) ListFiles(path interface{}) *gomock.Call

ListFiles indicates an expected call of ListFiles

func (*MockPathioMockRecorder) Reader

func (mr *MockPathioMockRecorder) Reader(path interface{}) *gomock.Call

Reader indicates an expected call of Reader

func (*MockPathioMockRecorder) Write

func (mr *MockPathioMockRecorder) Write(path, input interface{}) *gomock.Call

Write indicates an expected call of Write

func (*MockPathioMockRecorder) WriteReader

func (mr *MockPathioMockRecorder) WriteReader(path, input interface{}) *gomock.Call

WriteReader indicates an expected call of WriteReader

type Mocks3Handler

type Mocks3Handler struct {
	// contains filtered or unexported fields
}

Mocks3Handler is a mock of s3Handler interface

func NewMocks3Handler

func NewMocks3Handler(ctrl *gomock.Controller) *Mocks3Handler

NewMocks3Handler creates a new mock instance

func (*Mocks3Handler) DeleteObject

func (m *Mocks3Handler) DeleteObject(input *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error)

DeleteObject mocks base method

func (*Mocks3Handler) EXPECT

EXPECT returns an object that allows the caller to indicate expected use

func (*Mocks3Handler) GetBucketLocation

func (m *Mocks3Handler) GetBucketLocation(input *s3.GetBucketLocationInput) (*s3.GetBucketLocationOutput, error)

GetBucketLocation mocks base method

func (*Mocks3Handler) GetObject

func (m *Mocks3Handler) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)

GetObject mocks base method

func (*Mocks3Handler) HeadObject

func (m *Mocks3Handler) HeadObject(input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error)

HeadObject mocks base method

func (*Mocks3Handler) ListObjects

func (m *Mocks3Handler) ListObjects(input *s3.ListObjectsInput) (*s3.ListObjectsOutput, error)

ListObjects mocks base method

func (*Mocks3Handler) PutObject

func (m *Mocks3Handler) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)

PutObject mocks base method

type Mocks3HandlerMockRecorder

type Mocks3HandlerMockRecorder struct {
	// contains filtered or unexported fields
}

Mocks3HandlerMockRecorder is the mock recorder for Mocks3Handler

func (*Mocks3HandlerMockRecorder) DeleteObject

func (mr *Mocks3HandlerMockRecorder) DeleteObject(input interface{}) *gomock.Call

DeleteObject indicates an expected call of DeleteObject

func (*Mocks3HandlerMockRecorder) GetBucketLocation

func (mr *Mocks3HandlerMockRecorder) GetBucketLocation(input interface{}) *gomock.Call

GetBucketLocation indicates an expected call of GetBucketLocation

func (*Mocks3HandlerMockRecorder) GetObject

func (mr *Mocks3HandlerMockRecorder) GetObject(input interface{}) *gomock.Call

GetObject indicates an expected call of GetObject

func (*Mocks3HandlerMockRecorder) HeadObject

func (mr *Mocks3HandlerMockRecorder) HeadObject(input interface{}) *gomock.Call

HeadObject indicates an expected call of HeadObject

func (*Mocks3HandlerMockRecorder) ListObjects

func (mr *Mocks3HandlerMockRecorder) ListObjects(input interface{}) *gomock.Call

ListObjects indicates an expected call of ListObjects

func (*Mocks3HandlerMockRecorder) PutObject

func (mr *Mocks3HandlerMockRecorder) PutObject(input interface{}) *gomock.Call

PutObject indicates an expected call of PutObject

type Pathio

type Pathio interface {
	Reader(path string) (rc io.ReadCloser, err error)
	Write(path string, input []byte) error
	WriteReader(path string, input io.ReadSeeker) error
	Delete(path string) error
	ListFiles(path string) ([]string, error)
	Exists(path string) (bool, error)
}

Pathio is a defined interface for accessing both S3 and local files.

var DefaultClient Pathio = &Client{}

DefaultClient is the default pathio client called by the Reader, Writer, and WriteReader methods. It has S3 encryption enabled.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL