gof3r

README

Command gof3r provides a command-line interface to Amazon AWS S3.

Documentation:
http://godoc.org/github.com/rlmcpherson/s3gof3r/gof3r

Documentation

Overview

Command gof3r is a command-line interface for s3gof3r: fast, concurrent, streaming access to Amazon S3.

Usage:

To stream up to S3:
   $  <input_stream> | gof3r put -b <bucket> -k <s3_path>
To stream down from S3:
   $ gof3r get -b <bucket> -k <s3_path> | <output_stream>
To upload a file to S3:
   $ gof3r  put --path=<local_path> --bucket=<bucket> --key=<s3_path> -m<http_header1> -m<http_header2>...
To download a file from S3:
   $ gof3r  get --bucket=<bucket> --key=<path>

Set AWS keys as environment Variables (required):

$ export AWS_ACCESS_KEY_ID=<access_key>
$ export AWS_SECRET_ACCESS_KEY=<secret_key>

Examples:

$ tar -cf - /foo_dir/ | gof3r put -b my_s3_bucket -k bar_dir/s3_object -m x-amz-meta-custom-metadata:abc123 -m x-amz-server-side-encryption:AES256
$ gof3r get -b my_s3_bucket -k bar_dir/s3_object | tar -x

Complete Usage: get command:

gof3r [OPTIONS] get [get-OPTIONS]

get (download) from S3

Help Options:
-h, --help          Show this help message

get (download) from S3:
-p, --path=         Path to file. Defaults to standard output for streaming. (/dev/stdout)
-k, --key=          key of s3 object
-b, --bucket=       s3 bucket
--md5Check-off      Do not use md5 hash checking to ensure data integrity. By default, the md5 hash of is calculated concurrently
                    during puts, stored at <bucket>.md5/<key>.md5, and verified on gets.
-c, --concurrency=  Concurrency of transfers (20)
-s, --partsize=     initial size of concurrent parts, in bytes (20 MB)
--endpoint=     Amazon S3 endpoint (s3.amazonaws.com)
--debug             Print debug statements and dump stacks.
-v, --versionId=    The version ID of the object. Not compatible with md5 checking.

Help Options:
-h, --help          Show this help message

Complete Usage: put command:

gof3r [OPTIONS] put [put-OPTIONS]

put (upload)to S3

Help Options:
  -h, --help          Show this help message

put (upload) to S3:
  -p, --path=         Path to file. Defaults to standard input for streaming. (/dev/stdin)
  -m, --header=       HTTP headers
  -k, --key=          key of s3 object
  -b, --bucket=       s3 bucket
      --md5Check-off  Do not use md5 hash checking to ensure data integrity. By default, the md5 hash of is calculated concurrently
                      during puts, stored at <bucket>.md5/<key>.md5, and verified on gets.
  -c, --concurrency=  Concurrency of transfers (20)
  -s, --partsize=     initial size of concurrent parts, in bytes (20 MB)
  --endpoint=     Amazon S3 endpoint (s3.amazonaws.com)
  --debug         Print debug statements and dump stacks.

Help Options:
  -h, --help          Show this help message