s3

README

s3

An S3-compatible object API over any go-filesystems driver. Pure Go, CGO_ENABLED=0, an http.Handler.

srv, err := s3.New(fsys, func(id string) (string, bool) {
    if id == "AKIA..." { return secret, true }
    return "", false
})
http.ListenAndServe(":9000", srv)

A bucket is a top-level directory

GET /                      the top-level DIRECTORIES, as buckets
GET /photos?list-type=2    the files under /photos, as keys
GET /photos/holiday.jpg    the file /photos/holiday.jpg

A key is a path and a "directory" is a prefix, which is what S3 clients already believe. A file at the root is not a bucket and is not reachable at all.

What is here

ListBuckets, HeadBucket, ListObjectsV2 (prefix, delimiter, max-keys, continuation), HeadObject, GetObject with Range. Authenticated with AWS Signature Version 4, header or presigned.

• An unknown access key is answered as a wrong signature (SignatureDoesNotMatch), through the same checks in the same order, an HMAC with a decoy secret included: AWS's InvalidAccessKeyId would tell anybody which access keys (which user names) exist. Since v0.3.0.
• A presigned URL lasts a week at most (X-Amz-Expires ≤ 604800, AWS's bound) and is refused before its own X-Amz-Date. Before v0.3.0 presigned URLs were refused whatever their signature: the signature was deleted from the query before it was compared.

Exports are read-only by default, following the rest of the family. Most of what this is pointed at is a forensic or build artefact, and an accidental write to one is unrecoverable.

What is refused by name

Multipart upload, versioning, ACLs, policies, lifecycle rules, replication and tagging. A Filesystem has nowhere to keep any of them, and answering yes to a policy nobody enforces is worse than answering no. A client that asks gets NotImplemented, which is what makes it fall back to a single PUT.

Two things worth knowing before you point it at something large

GetObject reads the whole object into memory. The driver interface offers ReadFile and nothing else — no opener, no reader — so there is no streaming read to use. MaxObjectBytes (256 MiB by default) bounds it and answers EntityTooLarge rather than taking the process down.

LastModified is the epoch. filesystem.Stat carries Mode, Size and Inode, and no modification time at all. The field is not optional in the S3 schema, so something must be sent; the epoch is obviously not a real time, where time.Now() would look like a file that just changed and make every sync tool copy everything, every run.

Licence

BSD-3-Clause.

Documentation

Overview

Package s3 implements an S3-compatible object API over any github.com/go-filesystems/interface.Filesystem, as an net/http.Handler.

Every driver in the family — ext4, xfs, btrfs, zfs, ntfs, fat32, exfat, hfsplus, apfs, iso9660, squashfs, ufs, ffs, uefi, oci — becomes something the AWS SDK, minio-go, rclone, a backup tool or a browser's fetch() can read and write, from a pure-Go binary with no cgo and no root.

Why S3 alongside the mounts

github.com/go-filesystems/nfs, /sftp and /webdav make a Filesystem MOUNTABLE. S3 is the opposite shape and that is the point: nothing mounts, nothing is a path on somebody's machine, and every operation is one HTTP request that can be authorised, logged, rate-limited and served from anywhere. It is what programs speak now — a backup tool, a container registry's storage, a data pipeline, a phone app — and none of them wants a mount.

A bucket is a top-level directory

The mapping is the whole design decision, so it is worth stating plainly:

GET /                      the top-level DIRECTORIES, as buckets
GET /photos?list-type=2    the files under /photos, as keys
GET /photos/holiday.jpg    the file /photos/holiday.jpg

A key is a path and a "directory" is a prefix, which is what S3 clients already believe: they display prefixes ending in the delimiter as folders. A driver whose root holds files rather than directories therefore has no buckets, and says so rather than inventing one.

What is here, and what is refused by name

ListBuckets, HeadBucket, CreateBucket, DeleteBucket, ListObjectsV2 (with prefix, delimiter, max-keys and continuation), HeadObject, GetObject with Range, PutObject, DeleteObject and DeleteObjects.

Multipart upload is NOT here: it is a session across requests with its own storage of parts, and pretending otherwise would make a client's 5 GB upload fail at the end rather than at the start. A client that asks gets NotImplemented, which is what makes it fall back to a single PUT.

Nor are versioning, ACLs, policies, lifecycle rules, replication or tagging: a Filesystem has nowhere to keep any of them, and answering "yes" to a policy nobody enforces is worse than answering "no".

Signatures

Requests are authenticated with AWS Signature Version 4, header or presigned, which is what every client sends. Exports are READ-ONLY by default, following the rest of the family: most of what this is pointed at is a forensic or build artefact, and an accidental write to one is unrecoverable.

Index

• type CredentialLookup
• type Server
  • func New(fsys filesystem.Filesystem, creds CredentialLookup) (*Server, error)
  • func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)

Types

type CredentialLookup

type CredentialLookup func(accessKeyID string) (secret string, ok bool)

CredentialLookup returns the secret for an access key, and whether that key exists here.

⚠ Returning ("", false) for an unknown key and a real secret otherwise is the whole contract. Do NOT return ("", true) for an unknown key hoping the signature will fail anyway: it would, but the two cases then take different amounts of work and a caller can tell them apart.

type Server

type Server struct {

	// Credentials resolves an access key to its secret. Required: a server
	// with no way to check a signature would have to accept every request,
	// and New refuses to build one.
	Credentials CredentialLookup

	// Region the signatures are expected to carry. Empty accepts whatever the
	// client signed with, which is what a private deployment wants -- clients
	// invent a region and would otherwise all have to agree on the same
	// invention.
	Region string

	// ReadOnly refuses every mutating verb. Default TRUE, deliberately: see
	// the package doc.
	ReadOnly bool

	// MaxObjectBytes bounds what GET will read into memory, because the driver
	// interface has no streaming read. 0 takes the default.
	MaxObjectBytes int64

	// ClockSkew is how far a signed request may be from this server's clock.
	// 0 takes the default. It is replay protection, not a courtesy.
	ClockSkew time.Duration

	// Clock is for tests.
	Clock func() time.Time
	// contains filtered or unexported fields
}

Server serves an S3-compatible API over one Filesystem.

The zero value is not usable: New builds one. It is an http.Handler, so it composes with anything -- TLS, a reverse proxy, a rate limiter -- rather than owning a listener.

func New

func New(fsys filesystem.Filesystem, creds CredentialLookup) (*Server, error)

New builds a Server over fsys.

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)