archive

Documentation

Overview

Package archive provides functionality to read and write to an archive file that is written to local, azure blob store, or amazon s3.

Features include

1. Ability to maintain the file as a temporary `.tmp` until it is `.Close()`-ed at which point it is automatically renamed to the final name
2. Ability to keep file as a local file until uploaded to S3 or AZ Blob NOTE: direct streaming upload to S3/AZ is not yet performant.
3. Ability to specify local,s3,az all in a unified URL format s3://bucket/path/to/diretory/ az://containers/path/to/directory/ file:///path/to/directory Anything else is assumed to be a local file path

Index

• func Delete(dir string, files []string) (err error)
• func List(dir string) (files []string, err error)
• type Archive
  • func NewArchive(outDir, prefix, extension string, options ...func(*common.BasicArchive) error) (rf Archive, err error)
  • func OpenArchive(fileName string, bufferSize int) (rf Archive, err error)

Functions

func Delete

func Delete(dir string, files []string) (err error)

func List

func List(dir string) (files []string, err error)

List lists all files under the given path. All 3 urls formats (file, s3, az) are supported. Example: "az://<container-name>/some/path/inside"

Types

type Archive

type Archive interface {
	io.ReadWriteCloser
	Rotate() (err error)
	Flush() (err error)
	Name() string
	FinalizedFiles() map[string]common.ArchiveFileDetails
}

func NewArchive

func NewArchive(outDir, prefix, extension string, options ...func(*common.BasicArchive) error) (rf Archive, err error)

NewArchive opens an archive file for write. Once rotated after calling Rotate(), new writes will go to a new file. If outDir starts with "az://<container-name>/some/path/inside" or "s3://<bucket-name>/some/path/inside", the archive file would be uploaded to Azure Blobstore or S3 respectively. Please note settings are not all similar. Azure side code expects to environment variables AZURE_STORAGE_ACCOUNT as well as AZURE_STORAGE_ACCESS_KEY . However S3 side expects a `aws configure` performed with default settings in ~/.aws/credentials. There is no support for "profiles" yet.

func OpenArchive

func OpenArchive(fileName string, bufferSize int) (rf Archive, err error)

OpenArchive opens a single archive file for read. If outDir starts with "az://<container-name>/some/path/inside" or "s3://<bucket-name>/some/path/inside", the archive file would be uploaded to Azure Blobstore or S3 respectively. Please note settings are not all similar. Azure side code expects to environment variables AZURE_STORAGE_ACCOUNT as well as AZURE_STORAGE_ACCESS_KEY . However S3 side expects a `aws configure` performed with default settings in ~/.aws/credentials. There is no support for "profiles" yet.