s3

README

s3

An S3-compatible object API over any go-filesystems driver. Pure Go, CGO_ENABLED=0, an http.Handler.

srv, err := s3.New(fsys, func(id string) (string, bool) {
    if id == "AKIA..." { return secret, true }
    return "", false
})
http.ListenAndServe(":9000", srv)

A bucket is a top-level directory

GET /                      the top-level DIRECTORIES, as buckets
GET /photos?list-type=2    the files under /photos, as keys
GET /photos/holiday.jpg    the file /photos/holiday.jpg

A key is a path and a "directory" is a prefix, which is what S3 clients already believe. A file at the root is not a bucket and is not reachable at all.

What is here

ListBuckets, HeadBucket, ListObjectsV2 (prefix, delimiter, max-keys, continuation), HeadObject, GetObject with Range. Authenticated with AWS Signature Version 4, header or presigned.

• An unknown access key is answered as a wrong signature (SignatureDoesNotMatch), through the same checks in the same order, an HMAC with a decoy secret included: AWS's InvalidAccessKeyId would tell anybody which access keys (which user names) exist. Since v0.3.0.
• A presigned URL lasts a week at most (X-Amz-Expires ≤ 604800, AWS's bound) and is refused before its own X-Amz-Date. Before v0.3.0 presigned URLs were refused whatever their signature: the signature was deleted from the query before it was compared.

Exports are read-only by default, following the rest of the family. Most of what this is pointed at is a forensic or build artefact, and an accidental write to one is unrecoverable.

What is refused by name

Multipart upload, versioning, ACLs, policies, lifecycle rules, replication and tagging. A Filesystem has nowhere to keep any of them, and answering yes to a policy nobody enforces is worse than answering no. A client that asks gets NotImplemented, which is what makes it fall back to a single PUT.

Two things worth knowing before you point it at something large

GetObject streams the object (since v0.5.0) when the driver opens files (filesystem.Opener), whatever its size; a file of the host (osfs) goes out with sendfile(2) over plain HTTP. Before, every GET and HEAD read the whole object into memory — a few concurrent requests for large objects cost the server gigabytes, and nothing over 256 MiB could be served. A driver without Opener can only hand over a whole file, so for it MaxObjectBytes (256 MiB by default) still bounds what is read into memory and answers EntityTooLarge past it.

The ETag is the MD5 of the content, which rclone and the AWS SDKs check: it is computed once per version of an object — streamed through the hash, and kept while its size and modification time are unchanged (4096 objects remembered). Past MaxObjectBytes the ETag is multipart-shaped ("…-1"), which S3 says is not an MD5 and clients do not compare with one: a HEAD of a 40 GB object does not read 40 GB.

LastModified is the epoch. filesystem.Stat carries Mode, Size and Inode, and no modification time at all. The field is not optional in the S3 schema, so something must be sent; the epoch is obviously not a real time, where time.Now() would look like a file that just changed and make every sync tool copy everything, every run.

Licence

BSD-3-Clause.

Documentation

Overview

Package s3 implements an S3-compatible object API over any github.com/go-filesystems/interface.Filesystem, as an net/http.Handler.

Every driver in the family — ext4, xfs, btrfs, zfs, ntfs, fat32, exfat, hfsplus, apfs, iso9660, squashfs, ufs, ffs, uefi, oci — becomes something the AWS SDK, minio-go, rclone, a backup tool or a browser's fetch() can read and write, from a pure-Go binary with no cgo and no root.

Why S3 alongside the mounts

github.com/go-filesystems/nfs, /sftp and /webdav make a Filesystem MOUNTABLE. S3 is the opposite shape and that is the point: nothing mounts, nothing is a path on somebody's machine, and every operation is one HTTP request that can be authorised, logged, rate-limited and served from anywhere. It is what programs speak now — a backup tool, a container registry's storage, a data pipeline, a phone app — and none of them wants a mount.

A bucket is a top-level directory

The mapping is the whole design decision, so it is worth stating plainly:

GET /                      the top-level DIRECTORIES, as buckets
GET /photos?list-type=2    the files under /photos, as keys
GET /photos/holiday.jpg    the file /photos/holiday.jpg

A key is a path and a "directory" is a prefix, which is what S3 clients already believe: they display prefixes ending in the delimiter as folders. A driver whose root holds files rather than directories therefore has no buckets, and says so rather than inventing one.

What is here, and what is refused by name

ListBuckets, HeadBucket, CreateBucket, DeleteBucket, ListObjectsV2 (with prefix, delimiter, max-keys and continuation), HeadObject, GetObject with Range, PutObject, DeleteObject and DeleteObjects.

Multipart upload is NOT here: it is a session across requests with its own storage of parts, and pretending otherwise would make a client's 5 GB upload fail at the end rather than at the start. A client that asks gets NotImplemented, which is what makes it fall back to a single PUT.

Nor are versioning, ACLs, policies, lifecycle rules, replication or tagging: a Filesystem has nowhere to keep any of them, and answering "yes" to a policy nobody enforces is worse than answering "no".

Signatures

Requests are authenticated with AWS Signature Version 4, header or presigned, which is what every client sends. Exports are READ-ONLY by default, following the rest of the family: most of what this is pointed at is a forensic or build artefact, and an accidental write to one is unrecoverable.

Index

• type CredentialLookup
• type Server
  • func New(fsys filesystem.Filesystem, creds CredentialLookup) (*Server, error)
  • func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)

Types

type CredentialLookup

type CredentialLookup func(accessKeyID string) (secret string, ok bool)

CredentialLookup returns the secret for an access key, and whether that key exists here.

⚠ Returning ("", false) for an unknown key and a real secret otherwise is the whole contract. Do NOT return ("", true) for an unknown key hoping the signature will fail anyway: it would, but the two cases then take different amounts of work and a caller can tell them apart.

type Server

type Server struct {

	// Credentials resolves an access key to its secret. Required: a server
	// with no way to check a signature would have to accept every request,
	// and New refuses to build one.
	Credentials CredentialLookup

	// Region the signatures are expected to carry. Empty accepts whatever the
	// client signed with, which is what a private deployment wants -- clients
	// invent a region and would otherwise all have to agree on the same
	// invention.
	Region string

	// ReadOnly refuses every mutating verb. Default TRUE, deliberately: see
	// the package doc.
	ReadOnly bool

	// MaxObjectBytes is the largest object whose ETag is the MD5 of its
	// content (larger ones get a multipart-shaped ETag, which clients do not
	// compare with one), and the largest a driver without Opener may serve,
	// since such a driver can only hand over a whole file in memory. A driver
	// with Opener is streamed whatever the size. 0 takes the default.
	MaxObjectBytes int64

	// ClockSkew is how far a signed request may be from this server's clock.
	// 0 takes the default. It is replay protection, not a courtesy.
	ClockSkew time.Duration

	// Clock is for tests.
	Clock func() time.Time
	// contains filtered or unexported fields
}

Server serves an S3-compatible API over one Filesystem.

The zero value is not usable: New builds one. It is an http.Handler, so it composes with anything -- TLS, a reverse proxy, a rate limiter -- rather than owning a listener.

func New

func New(fsys filesystem.Filesystem, creds CredentialLookup) (*Server, error)

New builds a Server over fsys.

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)