s2

package module
v0.0.0-...-af4af81 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 25, 2022 License: Apache-2.0 Imports: 23 Imported by: 5

README

s2

GoDoc

"It's one less than s3"

This library facilitates the creation of servers with S3-like APIs in go. With this library, you just need to implement a few interfaces to create an S3-like API - s2 handles all of the glue code and edge cases. Additionally, we provide a fairly extensive suite of conformance and integration tests to ensure your implementation works with the most popular s3 libraries and executables.

See a complete example in the example directory.

s2 is used in production for pachyderm's s3gateway feature.

Adding new functionality

s2 only supports a small subset of the S3 API. When an unsupported feature is called, a NotImplemented error (with a 501 status code) is returned. If s2 is missing functionality that you'd like, we'd gladly take a PR that adds it!

Documentation

Index

Constants

View Source 
const (

	// VersioningDisabled specifies that versioning is not enabled on a bucket
	VersioningDisabled string = ""
	// VersioningDisabled specifies that versioning is suspended on a bucket
	VersioningSuspended string = "Suspended"
	// VersioningDisabled specifies that versioning is enabled on a bucket
	VersioningEnabled string = "Enabled"
)

Variables

View Source 
var (

	// InvalidChunk is an error returned when reading a multi-chunk object
	// upload that contains an invalid chunk header or body
	InvalidChunk = errors.New("invalid chunk")
)

Functions

func NotImplementedEndpoint 

func NotImplementedEndpoint(logger *logrus.Entry) http.HandlerFunc

NotImplementedEndpoint creates an endpoint that returns `NotImplementedError` responses. This can be used in places expecting a `HandlerFunc`, e.g. mux middleware.

func WriteError 

func WriteError(logger *logrus.Entry, w http.ResponseWriter, r *http.Request, err error)

WriteError serializes an error to a response as XML

Types

type AuthController 

type AuthController interface {
	// SecretKey is called when a request is made using AWS' auth V4 or V2. If
	// the given access key exists, a non-nil secret key should be returned.
	// Otherwise nil should be returned.
	SecretKey(r *http.Request, accessKey string, region *string) (*string, error)
	// CustomAuth handles requests that are not using AWS' auth V4 or V2. You
	// can use this to implement custom auth algorithms. Return true if the
	// request passes the auth check.
	CustomAuth(r *http.Request) (bool, error)
}

AuthController is an interface defining authentication

type Bucket 

type Bucket struct {
	// Name is the bucket name
	Name string `xml:"Name"`
	// CreationDate is when the bucket was created
	CreationDate time.Time `xml:"CreationDate"`
}

Bucket is an XML marshallable representation of a bucket

type BucketController 

type BucketController interface {
	// GetLocation gets the location of a bucket
	GetLocation(r *http.Request, bucket string) (string, error)

	// ListObjects lists objects within a bucket
	ListObjects(r *http.Request, bucket, prefix, marker, delimiter string, maxKeys int) (*ListObjectsResult, error)

	// ListObjectVersions lists objects' versions within a bucket
	ListObjectVersions(r *http.Request, bucket, prefix, keyMarker, versionMarker string, delimiter string, maxKeys int) (*ListObjectVersionsResult, error)

	// CreateBucket creates a bucket
	CreateBucket(r *http.Request, bucket string) error

	// DeleteBucket deletes a bucket
	DeleteBucket(r *http.Request, bucket string) error

	// GetBucketVersioning gets the state of versioning on the given bucket
	GetBucketVersioning(r *http.Request, bucket string) (string, error)

	// SetBucketVersioning sets the state of versioning on the given bucket
	SetBucketVersioning(r *http.Request, bucket, status string) error
}

BucketController is an interface that specifies bucket-level functionality.

type CommonPrefixes 

type CommonPrefixes struct {
	// Prefix specifies the common prefix value.
	Prefix string `xml:"Prefix"`
	// Owner specifies the owner of the object
	Owner User `xml:"Owner"`
}

CommonPrefixes specifies a common prefix of S3 keys. This is akin to a directory.

type CompleteMultipartResult 

type CompleteMultipartResult struct {
	// Location is the location of the newly uploaded object
	Location string
	// ETag is a hex encoding of the hash of the object contents, with or
	// without surrounding quotes.
	ETag string
	// Version is the version of the object, or an empty string if versioning
	// is not enabled or supported.
	Version string
}

CompleteMultipartResult is a response from a CompleteMultipart call

type Contents 

type Contents struct {
	// Key specifies the object key
	Key string `xml:"Key"`
	// LastModified specifies when the object was last modified
	LastModified time.Time `xml:"LastModified"`
	// ETag is a hex encoding of the hash of the object contents, with or
	// without surrounding quotes.
	ETag string `xml:"ETag"`
	// Size specifies the size of the object
	Size uint64 `xml:"Size"`
	// StorageClass specifies the storage class used for the object
	StorageClass string `xml:"StorageClass"`
	// Owner specifies the owner of the object
	Owner User `xml:"Owner"`
}

Contents is an individual file/object

type DeleteMarker 

type DeleteMarker struct {
	// Key specifies the object key
	Key string `xml:"Key"`
	// Version is the version of the object, or an empty string if versioning
	// is not enabled or supported.
	Version string `xml:"VersionId"`
	// IsLatest specifies whether this is the latest version of the object.
	IsLatest bool `xml:"IsLatest"`
	// LastModified specifies when the object was last modified
	LastModified time.Time `xml:"LastModified"`
	// Owner specifies the owner of the object
	Owner User `xml:"Owner"`
}

DeleteMarker specifies an object that has been deleted from a versioning-enabled bucket.

type DeleteObjectResult 

type DeleteObjectResult struct {
	// Version is the version of the object, or an empty string if versioning
	// is not enabled or supported.
	Version string
	// DeleteMarker specifies whether there's a delete marker in place of the
	// object.
	DeleteMarker bool
}

DeleteObjectResult is a response from a DeleteObject call

type Error 

type Error struct {
	// HTTPStatus is the HTTP status that will be set in the response
	HTTPStatus int `xml:"-"`

	Code      string `xml:"Code"`
	Message   string `xml:"Message"`
	Resource  string `xml:"Resource"`
	RequestID string `xml:"RequestId"`
}

Error is an XML marshallable error response

func AccessDeniedError 

func AccessDeniedError(r *http.Request) *Error

AccessDeniedError creates a new S3 error with a standard AccessDenied S3 code.

func AuthorizationHeaderMalformedError 

func AuthorizationHeaderMalformedError(r *http.Request) *Error

AuthorizationHeaderMalformedError creates a new S3 error with a standard AuthorizationHeaderMalformed S3 code.

func BadDigestError 

func BadDigestError(r *http.Request) *Error

BadDigestError creates a new S3 error with a standard BadDigest S3 code.

func BucketAlreadyOwnedByYouError 

func BucketAlreadyOwnedByYouError(r *http.Request) *Error

BucketAlreadyOwnedByYouError creates a new S3 error with a standard BucketAlreadyOwnedByYou S3 code.

func BucketNotEmptyError 

func BucketNotEmptyError(r *http.Request) *Error

BucketNotEmptyError creates a new S3 error with a standard BucketNotEmpty S3 code.

func EntityTooLargeError 

func EntityTooLargeError(r *http.Request) *Error

EntityTooLargeError creates a new S3 error with a standard EntityTooLarge S3 code.

func EntityTooSmallError 

func EntityTooSmallError(r *http.Request) *Error

EntityTooSmallError creates a new S3 error with a standard EntityTooSmall S3 code.

func IllegalVersioningConfigurationError 

func IllegalVersioningConfigurationError(r *http.Request) *Error

IllegalVersioningConfigurationError creates a new S3 error with a standard IllegalVersioningConfigurationException S3 code.

func IncompleteBodyError 

func IncompleteBodyError(r *http.Request) *Error

IncompleteBodyError creates a new S3 error with a standard IncompleteBody S3 code.

func InternalError 

func InternalError(r *http.Request, err error) *Error

InternalError creates a new S3 error with a standard InternalError S3 code.

func InvalidAccessKeyIDError 

func InvalidAccessKeyIDError(r *http.Request) *Error

InvalidAccessKeyIDError creates a new S3 error with a standard InvalidAccessKeyId S3 code.

func InvalidArgumentError 

func InvalidArgumentError(r *http.Request) *Error

InvalidArgumentError creates a new S3 error with a standard InvalidArgument S3 code.

func InvalidBucketNameError 

func InvalidBucketNameError(r *http.Request) *Error

InvalidBucketNameError creates a new S3 error with a standard InvalidBucketName S3 code.

func InvalidDigestError 

func InvalidDigestError(r *http.Request) *Error

InvalidDigestError creates a new S3 error with a standard InvalidDigest S3 code.

func InvalidPartError 

func InvalidPartError(r *http.Request) *Error

InvalidPartError creates a new S3 error with a standard InvalidPart S3 code.

func InvalidPartOrderError 

func InvalidPartOrderError(w http.ResponseWriter, r *http.Request) *Error

InvalidPartOrderError creates a new S3 error with a standard InvalidPartOrder S3 code.

func InvalidRequestError 

func InvalidRequestError(r *http.Request, message string) *Error

InvalidRequestError creates a new S3 error with a standard InvalidRequest S3 code.

func MalformedXMLError 

func MalformedXMLError(r *http.Request) *Error

MalformedXMLError creates a new S3 error with a standard MalformedXML S3 code.

func MethodNotAllowedError 

func MethodNotAllowedError(r *http.Request) *Error

MethodNotAllowedError creates a new S3 error with a standard MethodNotAllowed S3 code.

func MissingContentLengthError 

func MissingContentLengthError(r *http.Request) *Error

MissingContentLengthError creates a new S3 error with a standard MissingContentLength S3 code.

func MissingRequestBodyError 

func MissingRequestBodyError(r *http.Request) *Error

MissingRequestBodyError creates a new S3 error with a standard MissingRequestBodyError S3 code.

func NewError 

func NewError(r *http.Request, httpStatus int, code string, message string) *Error

NewError creates a new S3 error, to be serialized in a response

func NoSuchBucketError 

func NoSuchBucketError(r *http.Request) *Error

NoSuchBucketError creates a new S3 error with a standard NoSuchBucket S3 code.

func NoSuchKeyError 

func NoSuchKeyError(r *http.Request) *Error

NoSuchKeyError creates a new S3 error with a standard NoSuchKey S3 code.

func NoSuchUploadError 

func NoSuchUploadError(r *http.Request) *Error

NoSuchUploadError creates a new S3 error with a standard NoSuchUpload S3 code.

func NoSuchVersionError 

func NoSuchVersionError(r *http.Request) *Error

NoSuchVersionError creates a new S3 error with a standard NoSuchVersion S3 code.

func NotImplementedError 

func NotImplementedError(r *http.Request) *Error

NotImplementedError creates a new S3 error with a standard NotImplemented S3 code.

func PreconditionFailedError 

func PreconditionFailedError(r *http.Request) *Error

PreconditionFailedError creates a new S3 error with a standard PreconditionFailed S3 code.

func RequestTimeTooSkewedError 

func RequestTimeTooSkewedError(r *http.Request) *Error

RequestTimeTooSkewedError creates a new S3 error with a standard RequestTimeTooSkewed S3 code.

func RequestTimeoutError 

func RequestTimeoutError(r *http.Request) *Error

RequestTimeoutError creates a new S3 error with a standard RequestTimeout S3 code.

func SignatureDoesNotMatchError 

func SignatureDoesNotMatchError(r *http.Request) *Error

SignatureDoesNotMatchError creates a new S3 error with a standard SignatureDoesNotMatch S3 code.

func (*Error) Error 

func (e *Error) Error() string

type GetObjectResult 

type GetObjectResult struct {
	// ETag is a hex encoding of the hash of the object contents, with or
	// without surrounding quotes.
	ETag string
	// Version is the version of the object, or an empty string if versioning
	// is not enabled or supported.
	Version string
	// DeleteMarker specifies whether there's a delete marker in place of the
	// object.
	DeleteMarker bool
	// ModTime specifies when the object was modified.
	ModTime time.Time
	// Content is the contents of the object.
	Content io.ReadSeeker
}

GetObjectResult is a response from a GetObject call

type ListBucketsResult 

type ListBucketsResult struct {
	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListAllMyBucketsResult"`
	// Owner is the owner of the buckets
	Owner *User `xml:"Owner"`
	// Buckets are a list of buckets under the given owner
	Buckets []*Bucket `xml:"Buckets>Bucket"`
}

ListBucketsResult is a response from a ListBucket call

type ListMultipartChunksResult 

type ListMultipartChunksResult struct {
	// Initiator is the user that initiated the multipart upload
	Initiator *User
	// Owner specifies the owner of the object
	Owner *User
	// StorageClass specifies the storage class used for the object
	StorageClass string
	// IsTruncated specifies whether this is the end of the list or not
	IsTruncated bool
	// Parts are the list of parts returned
	Parts []*Part
}

ListMultipartChunksResult is a response from a ListMultipartChunks call

type ListMultipartResult 

type ListMultipartResult struct {
	// IsTruncated specifies whether this is the end of the list or not
	IsTruncated bool
	// Uploads are the list of uploads returned
	Uploads []*Upload
}

ListMultipartResult is a response from a ListMultipart call

type ListObjectVersionsResult 

type ListObjectVersionsResult struct {
	// Versions are the list of versions returned
	Versions []*Version
	// DeleteMarkers are the list of delete markers returned
	DeleteMarkers []*DeleteMarker
	// IsTruncated specifies whether this is the end of the list or not
	IsTruncated bool
}

ListObjectVersionsResult is a response from a ListObjectVersions call

type ListObjectsResult 

type ListObjectsResult struct {
	// Contents are the list of objects returned
	Contents []*Contents
	// CommonPrefixes are the list of common prefixes returned
	CommonPrefixes []*CommonPrefixes
	// IsTruncated specifies whether this is the end of the list or not
	IsTruncated bool
}

ListObjectsResult is a response from a ListObjects call

type MultipartController 

type MultipartController interface {
	// ListMultipart lists in-progress multipart uploads in a bucket
	ListMultipart(r *http.Request, bucket, keyMarker, uploadIDMarker string, maxUploads int) (*ListMultipartResult, error)
	// InitMultipart initializes a new multipart upload
	InitMultipart(r *http.Request, bucket, key string) (string, error)
	// AbortMultipart aborts an in-progress multipart upload
	AbortMultipart(r *http.Request, bucket, key, uploadID string) error
	// CompleteMultipart finishes a multipart upload
	CompleteMultipart(r *http.Request, bucket, key, uploadID string, parts []*Part) (*CompleteMultipartResult, error)
	// ListMultipartChunks lists the constituent chunks of an in-progress
	// multipart upload
	ListMultipartChunks(r *http.Request, bucket, key, uploadID string, partNumberMarker, maxParts int) (*ListMultipartChunksResult, error)
	// UploadMultipartChunk uploads a chunk of an in-progress multipart upload
	UploadMultipartChunk(r *http.Request, bucket, key, uploadID string, partNumber int, reader io.Reader) (string, error)
}

MultipartController is an interface that specifies multipart-related functionality

type ObjectController 

type ObjectController interface {
	// GetObject gets an object
	GetObject(r *http.Request, bucket, key, version string) (*GetObjectResult, error)
	// CopyObject copies an object
	CopyObject(r *http.Request, srcBucket, srcKey string, getResult *GetObjectResult, destBucket, destKey string) (string, error)
	// PutObject sets an object
	PutObject(r *http.Request, bucket, key string, reader io.Reader) (*PutObjectResult, error)
	// DeleteObject deletes an object
	DeleteObject(r *http.Request, bucket, key, version string) (*DeleteObjectResult, error)
}

ObjectController is an interface that specifies object-level functionality.

type Part 

type Part struct {
	// PartNumber is the index of the part
	PartNumber int `xml:"PartNumber"`
	// ETag is a hex encoding of the hash of the object contents, with or
	// without surrounding quotes.
	ETag string `xml:"ETag"`
}

Part is an XML marshallable representation of a chunk of an in-progress multipart upload

type PutObjectResult 

type PutObjectResult struct {
	// ETag is a hex encoding of the hash of the object contents, with or
	// without surrounding quotes.
	ETag string
	// Version is the version of the object, or an empty string if versioning
	// is not enabled or supported.
	Version string
}

PutObjectResult is a response from a PutObject call

type S2 

type S2 struct {
	Auth      AuthController
	Service   ServiceController
	Bucket    BucketController
	Object    ObjectController
	Multipart MultipartController
	// contains filtered or unexported fields
}

S2 is the root struct used in the s2 library

func NewS2 

func NewS2(logger *logrus.Entry, maxRequestBodyLength uint32, readBodyTimeout time.Duration) *S2

NewS2 creates a new S2 instance. One created, you set zero or more attributes to implement various S3 functionality, then create a router. `maxRequestBodyLength` specifies maximum request body size; if the value is 0, there is no limit. `readBodyTimeout` specifies the maximum amount of time s2 should spend trying to read the body of requests.

func (*S2) Router 

func (h *S2) Router() *mux.Router

Router creates a new mux router.

type ServiceController 

type ServiceController interface {
	// ListBuckets lists all buckets
	ListBuckets(r *http.Request) (*ListBucketsResult, error)
}

ServiceController is an interface defining service-level functionality

type Upload 

type Upload struct {
	// Key specifies the object key
	Key string `xml:"Key"`
	// UploadID is an ID identifying the multipart upload
	UploadID string `xml:"UploadId"`
	// Initiator is the user that initiated the multipart upload
	Initiator User `xml:"Initiator"`
	// Owner specifies the owner of the object
	Owner User `xml:"Owner"`
	// StorageClass specifies the storage class used for the object
	StorageClass string `xml:"StorageClass"`
	// Initiated is a timestamp specifying when the multipart upload was
	// started
	Initiated time.Time `xml:"Initiated"`
}

Upload is an XML marshallable representation of an in-progress multipart upload

type User 

type User struct {
	// ID is an ID of the user
	ID string `xml:"ID"`
	// DisplayName is a display name of the user
	DisplayName string `xml:"DisplayName"`
}

User is an XML marshallable representation of an S3 user

type Version 

type Version struct {
	// Key specifies the object key
	Key string `xml:"Key"`
	// Version is the version of the object, or an empty string if versioning
	// is not enabled or supported.
	Version string `xml:"VersionId"`
	// IsLatest specifies whether this is the latest version of the object.
	IsLatest bool `xml:"IsLatest"`
	// LastModified specifies when the object was last modified
	LastModified time.Time `xml:"LastModified"`
	// ETag is a hex encoding of the hash of the object contents, with or
	// without surrounding quotes.
	ETag string `xml:"ETag"`
	// Size specifies the size of the object
	Size uint64 `xml:"Size"`
	// StorageClass specifies the storage class used for the object
	StorageClass string `xml:"StorageClass"`
	// Owner specifies the owner of the object
	Owner User `xml:"Owner"`
}

Version specifies a specific version of an object in a versioning-enabled bucket.

Source Files

View all Source files

Directories

Path Synopsis
examples
sql Module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.