s3backup

README

s3backup

Shorten your strings using common abbreviations.

Motivation

This tool has been developed so that I can conveniently backup all of my personal photos to an AWS S3 bucket.

But why develop this? Aren't there other tools that can solve your problem?

Of course, but this is more fun.

Installation

git clone https://github.com/dnnrly/s3backup
cd s3backup
make build

Usage

$ s3backup --help
This too backs up your files to S3 so that you can have them in
the cloud. It will scan the location(s) that you specify and
attempt rudimentary de-duplication.

Examples:

$ s3backup

Code of Conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code.

Contributing

Pull requests are welcome. See the contributing guide for more details.

Please make sure to update tests as appropriate.

License

Apache 2

Documentation

Index

• Variables
• func FileHasher(path string) (string, error)
• func FilePathWalker(root string, index *Index, hasher PathHasher) filepath.WalkFunc
• func UploadBatch(diffFiles *Index, batchHash []string, toUpload *Index, store IndexStore, ...) error
• func UploadDifferences(localIndex, remoteIndex *Index, parallelLimit int, batchSize int, ...) error
• type Config
  • func NewConfigFromFile(p string) (*Config, error)
  • func NewConfigFromString(data string) (*Config, error)
• type FileGetter
• type FileRepository
• type Index
  • func CopyIndex(from *Index) *Index
  • func NewIndex(buf string) (*Index, error)
  • func NewIndexFromRoot(bucketRoot, path string, walker PathWalker, hasher PathHasher) (*Index, error)
  • func (i *Index) Add(f string, src Sourcefile)
  • func (local *Index) Diff(remote *Index) *Index
  • func (i *Index) Encode() (string, error)
  • func (i *Index) GetNextN(n int) *Index
• type IndexStore
• type Limiter
• type PathHasher
• type PathWalker
• type Sourcefile

Variables

var (
	// Verbose enables verbose logging in this package
	Verbose = false
)

Functions

func FileHasher

func FileHasher(path string) (string, error)

FileHasher returns a hash of the contents of a file

func FilePathWalker

func FilePathWalker(root string, index *Index, hasher PathHasher) filepath.WalkFunc

FilePathWalker is a PathWalker that accesses files on the disk when walking a directory tree

func UploadBatch

func UploadBatch(diffFiles *Index, batchHash []string, toUpload *Index, store IndexStore, limiter *Limiter, batchSize int, getFile FileGetter) error

This uploads batches given by UploadDifferences. The files in the batch are uploaded in parallel

func UploadDifferences

func UploadDifferences(localIndex, remoteIndex *Index, parallelLimit int, batchSize int, store IndexStore, getFile FileGetter) error

UploadDifferences will upload the files that are missing from the remote index

Types

type Config

type Config struct {
	S3 s3.Config `yaml:"s3"`
}

Config defines the configuration for the whole tool

func NewConfigFromFile

func NewConfigFromFile(p string) (*Config, error)

NewConfigFromFile generates a config object from a file

func NewConfigFromString

func NewConfigFromString(data string) (*Config, error)

NewConfigFromString generates a config object from the string

type FileGetter

type FileGetter func(p string) io.ReadCloser

FileGetter allows you to get the contents of a file

type FileRepository

type FileRepository interface {
	// GetByKey retrieves the data at a certain location in your store
	GetByKey(key string) (io.Reader, error)
	// Save puts the data at a location in your store
	Save(key string, data io.Reader) error
}

FileRepository allows you to access files in your remote location

type Index

type Index struct {
	// Files maps the local file location to its metadata
	Files map[string]Sourcefile `yaml:"files"`
}

Index holds all of the metadata for files backed up

func CopyIndex

func CopyIndex(from *Index) *Index

CopyIndex creates an Index from Yaml

func NewIndex

func NewIndex(buf string) (*Index, error)

NewIndex creates an Index from Yaml

func NewIndexFromRoot

func NewIndexFromRoot(
	bucketRoot,
	path string,
	walker PathWalker,
	hasher PathHasher,
) (*Index, error)

NewIndexFromRoot creates a new Index populated from a filesystem directory

func (*Index) Add

func (i *Index) Add(f string, src Sourcefile)

Add a single source file to the index

func (*Index) Diff

func (local *Index) Diff(remote *Index) *Index

Diff finds all entries in this Index that do not exist or are different from the remote entry.

func (*Index) Encode

func (i *Index) Encode() (string, error)

Encode the index data as Yaml

func (*Index) GetNextN

func (i *Index) GetNextN(n int) *Index

GetNextN gets any N items from the index

type IndexStore

type IndexStore interface {
	// Save an indexed object to the specified location
	Save(key string, data io.Reader) error
}

IndexStore allows you to persist indexed objects

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter object for limiting concurrency of go routines

type PathHasher

type PathHasher func(path string) (string, error)

PathHasher is a function that will hash the file at 'path' location

type PathWalker

type PathWalker func(root string, index *Index, hasher PathHasher) filepath.WalkFunc

PathWalker is a function that can walk a directory tree and populate the Index that is passed in

type Sourcefile

type Sourcefile struct {
	// Key is the location of this file in a bucket
	Key string `yaml:"key"`
	// Hash is the hashed value of the file contents
	Hash string `yaml:"hash"`
}

Sourcefile represents the metadata for a single backed up file