s3resource

README

S3 Resource

Versions objects in an S3 bucket, by pattern-matching filenames to identify version numbers.

Source Configuration

• bucket: Required. The name of the bucket.

• access_key_id: Optional. The AWS access key to use when accessing the bucket.

• secret_access_key: Optional. The AWS secret key to use when accessing the bucket.

• session_token: Optional. The AWS session token to use when accessing the bucket.

• region_name: Optional. The region the bucket is in. Defaults to us-east-1.

• private: Optional. Indicates that the bucket is private, so that any URLs provided are signed.

• cloudfront_url: Optional. The URL (scheme and domain) of your CloudFront distribution that is fronting this bucket. This will be used in the url file that is given to the following task.

• endpoint: Optional. Custom endpoint for using S3 compatible provider.

File Names

One of the following two options must be specified:

• regexp: Optional. The pattern to match filenames against within S3. The first grouped match is used to extract the version, or if a group is explicitly named version, that group is used. At least one capture group must be specified, with parentheses.

  The version extracted from this pattern is used to version the resource. Semantic versions, or just numbers, are supported. Accordingly, full regular expressions are supported, to specify the capture groups.

• versioned_file: Optional If you enable versioning for your S3 bucket then you can keep the file name the same and upload new versions of your file without resorting to version numbers. This property is the path to the file in your S3 bucket.

Behavior

check: Extract versions from the bucket.

Objects will be found via the pattern configured by regexp. The versions will be used to order them (using semver). Each object's filename is the resulting version.

in: Fetch an object from the bucket.

Places the following files in the destination:

• (filename): The file fetched from the bucket.

• url: A file containing the URL of the object. If private is true, this URL will be signed.

• version: The version identified in the file name.

Parameters

None.

out: Upload an object to the bucket.

Given a file specified by file, upload it to the S3 bucket. If regexp is specified, the new file will be uploaded to the directory that the regex searches in. If versioned_file is specified, the new file will be uploaded as a new version of that file.

Parameters

• file: Required. Path to the file to upload, provided by an output of a task. If multiple files are matched by the glob, an error is raised. The file which matches will be placed into the directory structure on S3 as defined in regexp in the resource definition. The matching syntax is bash glob expansion, so no capture groups, etc.

• acl: Optional. Canned Acl for the uploaded object.

Example Configuration

Resource

- name: release
  type: s3
  source:
    bucket: releases
    regexp: directory_on_s3/release-(.*).tgz
    access_key_id: ACCESS-KEY
    secret_access_key: SECRET

Plan

- get: release

- put: release
  params:
    file: path/to/release-*.tgz
    acl: public-read

Required IAM Permissions

Non-versioned Buckets

• s3:PutObject
• s3:PutObjectAcl
• s3:GetObject
• s3:ListBucket

Versioned Buckets

Everything above and...

• s3:GetBucketVersioning
• s3:GetObjectVersion
• s3:ListBucketVersions
• s3:PutObjectVersionAcl

Developing on this resource

First get the resource via: go get github.com/concourse/s3-resource

Get the dependencies: go get -d -t ./...

Documentation

Index

• func Fatal(doing string, err error)
• func NewAwsConfig(accessKey string, secretKey string, sessionToken string, regionName string, ...) *aws.Config
• func Sayf(message string, args ...interface{})
• type MetadataPair
• type S3Client
  • func NewS3Client(progressOutput io.Writer, awsConfig *aws.Config) S3Client
• type Source
  • func (source Source) IsValid() (bool, string)
• type Version

Functions

func Fatal

func Fatal(doing string, err error)

func NewAwsConfig

func NewAwsConfig(
	accessKey string,
	secretKey string,
	sessionToken string,
	regionName string,
	endpoint string,
) *aws.Config

func Sayf

func Sayf(message string, args ...interface{})

Types

type MetadataPair

type MetadataPair struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

type S3Client

type S3Client interface {
	BucketFiles(bucketName string, prefixHint string) ([]string, error)
	BucketFileVersions(bucketName string, remotePath string) ([]string, error)

	UploadFile(bucketName string, remotePath string, localPath string, acl string) (string, error)
	DownloadFile(bucketName string, remotePath string, versionID string, localPath string) error

	DeleteFile(bucketName string, remotePath string) error
	DeleteVersionedFile(bucketName string, remotePath string, versionID string) error

	URL(bucketName string, remotePath string, private bool, versionID string) string
}

func NewS3Client

func NewS3Client(
	progressOutput io.Writer,
	awsConfig *aws.Config,
) S3Client

type Source

type Source struct {
	AccessKeyID     string `json:"access_key_id"`
	SecretAccessKey string `json:"secret_access_key"`
	SessionToken    string `json:"session_token"`
	Bucket          string `json:"bucket"`
	Regexp          string `json:"regexp"`
	VersionedFile   string `json:"versioned_file"`
	Private         bool   `json:"private"`
	RegionName      string `json:"region_name"`
	CloudfrontURL   string `json:"cloudfront_url"`
	Endpoint        string `json:"endpoint"`
}

func (Source) IsValid

func (source Source) IsValid() (bool, string)

type Version

type Version struct {
	Path      string `json:"path,omitempty"`
	VersionID string `json:"version_id,omitempty"`
}