s3storage

README

go-s3-storage

A small, practical file-storage layer for Go over S3 and S3-compatible services (MinIO, Cloudflare R2): upload with correct content-type, download, presign, public/CDN URLs and typed errors.

Why

The official SDK is raw API surface. What a web application actually needs is seven operations with sane behavior: Upload, Download, Stat, Exists, Delete, List, PresignGet/Put — plus a PublicURL that knows about your CDN. This package is that layer, on top of aws-sdk-go-v2:

• Content type resolved properly: explicit > file extension > sniffing the first 512 bytes. No more application/octet-stream images.
• Streaming everywhere — uploads switch to multipart automatically, nothing is buffered whole.
• Typed errors: errors.Is(err, s3storage.ErrNotFound) instead of unwrapping SDK response codes.
• MinIO / R2 ready: WithBaseEndpoint + WithPathStyle.

Install

go get github.com/YusufDrymz/go-s3-storage

Usage

cfg, _ := config.LoadDefaultConfig(ctx) // aws-sdk-go-v2/config
store := s3storage.New(cfg, "my-bucket",
    s3storage.WithPublicBaseURL("https://cdn.example.com"),
)

obj, err := store.Upload(ctx, "avatars/u1.png", file) // content-type: image/png
url := store.PublicURL(obj.Key)                       // https://cdn.example.com/avatars/u1.png

rc, obj, err := store.Download(ctx, "avatars/u1.png")
defer rc.Close()

signed, err := store.PresignGet(ctx, "private/invoice.pdf", 15*time.Minute)

Iterate a prefix (pagination handled for you):

for obj, err := range store.List(ctx, "invoices/2026/") {
    if err != nil {
        return err
    }
    fmt.Println(obj.Key, obj.Size)
}

MinIO / Cloudflare R2

store := s3storage.New(cfg, "my-bucket",
    s3storage.WithBaseEndpoint("http://localhost:9000"), // or the R2 endpoint
    s3storage.WithPathStyle(),                           // required by MinIO
)

Direct browser uploads

up, err := store.PresignPut(ctx, "uploads/photo.jpg", 5*time.Minute,
    s3storage.WithMetadata(map[string]string{"owner": userID}),
)
// up.URL    → PUT target for the browser
// up.Header → signed headers the browser MUST send verbatim

Content-Type is deliberately not part of the signature (the AWS SDK strips it so clients can set their own); metadata and cache-control are signed and come back in up.Header.

Tenant / environment separation

store := s3storage.New(cfg, "shared-bucket", s3storage.WithKeyPrefix("tenant-42"))
store.Upload(ctx, "docs/a.pdf", r) // stored as tenant-42/docs/a.pdf
// every key you see stays logical: "docs/a.pdf"

Error handling

_, err := store.Stat(ctx, key)
switch {
case errors.Is(err, s3storage.ErrNotFound):
case errors.Is(err, s3storage.ErrBucketNotFound):
case errors.Is(err, s3storage.ErrAccessDenied):
}

The original SDK error stays in the chain for errors.As.

Testing your integration

Point the store at any HTTP server: WithBaseEndpoint(srv.URL) + WithPathStyle() against an httptest.Server (or a local MinIO container). This package's own tests run against a ~150-line fake S3 — see fake_s3_test.go for a copy-paste starting point.

🇹🇷 Türkçe

S3 ve S3-uyumlu servisler (MinIO, Cloudflare R2) için pratik dosya katmanı: doğru content-type ile upload (uzantı + sniff), streaming download, presign, CDN URL üretimi ve typed error'lar. Resmi SDK'nın üzerinde ince bir katman — SDK yüzeyini öğrenmeden production ihtiyacının tamamı.

Kurulum: go get github.com/YusufDrymz/go-s3-storage

store := s3storage.New(cfg, "my-bucket",
    s3storage.WithPublicBaseURL("https://cdn.example.com"))
obj, _ := store.Upload(ctx, "avatars/u1.png", file)
url := store.PublicURL(obj.Key)

Büyük dosyalar otomatik multipart ile yüklenir, hiçbir şey belleğe komple alınmaz. WithKeyPrefix ile tek bucket'ta tenant/environment ayrımı; List go 1.24 iterator'ı ile sayfalamayı içeride halleder. Hatalar ErrNotFound / ErrBucketNotFound / ErrAccessDenied olarak errors.Is ile yakalanır.

License

MIT — see LICENSE.

Documentation

Overview

Package s3storage is a small, practical file-storage layer over S3 and S3-compatible services (MinIO, Cloudflare R2): upload with correct content-type, download, presign, public/CDN URLs and typed errors — the operations a web application actually needs, without learning the SDK surface.

Index

• Variables
• type Object
• type Option
  • func WithBaseEndpoint(u string) Option
  • func WithKeyPrefix(prefix string) Option
  • func WithPathStyle() Option
  • func WithPublicBaseURL(u string) Option
  • func WithUploadPartSize(n int64) Option
• type PresignedUpload
• type Store
  • func New(cfg aws.Config, bucket string, opts ...Option) *Store
  • func (s *Store) Delete(ctx context.Context, key string) error
  • func (s *Store) Download(ctx context.Context, key string) (io.ReadCloser, *Object, error)
  • func (s *Store) Exists(ctx context.Context, key string) (bool, error)
  • func (s *Store) List(ctx context.Context, prefix string) iter.Seq2[*Object, error]
  • func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error)
  • func (s *Store) PresignPut(ctx context.Context, key string, ttl time.Duration, opts ...UploadOption) (*PresignedUpload, error)
  • func (s *Store) PublicURL(key string) string
  • func (s *Store) Stat(ctx context.Context, key string) (*Object, error)
  • func (s *Store) Upload(ctx context.Context, key string, r io.Reader, opts ...UploadOption) (*Object, error)
• type UploadOption
  • func WithCacheControl(v string) UploadOption
  • func WithContentDisposition(v string) UploadOption
  • func WithContentType(ct string) UploadOption
  • func WithMetadata(m map[string]string) UploadOption

Variables

var (
	ErrNotFound       = errors.New("s3storage: object not found")
	ErrBucketNotFound = errors.New("s3storage: bucket not found")
	ErrAccessDenied   = errors.New("s3storage: access denied")
)

Sentinel errors for matching with errors.Is. The original SDK error stays in the chain for errors.As inspection.

Types

type Object

type Object struct {
	Key          string
	Size         int64
	ContentType  string
	ETag         string
	LastModified time.Time
	Metadata     map[string]string
}

Object describes a stored file. Key is always the logical key (prefix stripped); ETag comes without surrounding quotes.

type Option

type Option func(*Store)

Option configures a Store.

func WithBaseEndpoint

func WithBaseEndpoint(u string) Option

WithBaseEndpoint points the client at an S3-compatible service (MinIO, Cloudflare R2). Usually combined with WithPathStyle for MinIO.

func WithKeyPrefix

func WithKeyPrefix(prefix string) Option

WithKeyPrefix prepends prefix/ to every key, transparently for the caller. Useful for tenant or environment separation inside one bucket.

func WithPathStyle

func WithPathStyle() Option

WithPathStyle forces path-style addressing (endpoint/bucket/key instead of bucket.endpoint/key). Required by MinIO and most self-hosted services.

func WithPublicBaseURL

func WithPublicBaseURL(u string) Option

WithPublicBaseURL makes PublicURL build links on a CDN or public domain instead of the bucket endpoint.

func WithUploadPartSize

func WithUploadPartSize(n int64) Option

WithUploadPartSize sets the multipart chunk size (min 5MB per S3).

type PresignedUpload

type PresignedUpload struct {
	URL    string
	Header http.Header
}

PresignedUpload is a time-limited direct-upload target. Header holds the signed headers (Cache-Control, x-amz-meta-*): the uploader must send them verbatim or S3 rejects the PUT with a signature mismatch.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store wraps one bucket. It is safe for concurrent use; construct once and share. All keys are "logical": the optional key prefix is applied on the way in and stripped on the way out.

func New

func New(cfg aws.Config, bucket string, opts ...Option) *Store

New builds a Store for the bucket on top of the given AWS config (config.LoadDefaultConfig covers profiles, roles and env vars).

func (*Store) Delete

func (s *Store) Delete(ctx context.Context, key string) error

Delete removes the object. Deleting a missing key is not an error — S3 deletes are idempotent.

func (*Store) Download

func (s *Store) Download(ctx context.Context, key string) (io.ReadCloser, *Object, error)

Download streams the object; the caller must Close the reader. The whole body is never held in memory.

func (*Store) Exists

func (s *Store) Exists(ctx context.Context, key string) (bool, error)

Exists reports whether the key is present.

func (*Store) List

func (s *Store) List(ctx context.Context, prefix string) iter.Seq2[*Object, error]

List iterates all objects under the prefix, paginating transparently:

for obj, err := range store.List(ctx, "invoices/") {
    if err != nil { return err }
    ...
}

On error one (nil, err) pair is yielded and iteration stops.

func (*Store) PresignGet

func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error)

PresignGet returns a time-limited download URL for a private object. Signing is local, no network call happens.

func (*Store) PresignPut

func (s *Store) PresignPut(ctx context.Context, key string, ttl time.Duration, opts ...UploadOption) (*PresignedUpload, error)

PresignPut returns a time-limited upload URL, typically handed to a browser for direct upload. Content-Type is deliberately NOT part of the signature (the AWS SDK strips it so clients can set their own); metadata and cache-control ARE signed and come back in Header.

func (*Store) PublicURL

func (s *Store) PublicURL(key string) string

PublicURL returns the non-expiring URL of a key: the CDN base when configured, otherwise the bucket endpoint. It does not check existence and assumes the object is publicly readable.

func (*Store) Stat

func (s *Store) Stat(ctx context.Context, key string) (*Object, error)

Stat returns object metadata without the body.

func (*Store) Upload

func (s *Store) Upload(ctx context.Context, key string, r io.Reader, opts ...UploadOption) (*Object, error)

Upload streams r to the bucket, switching to multipart automatically for large content — r is never buffered whole. Content type resolution: WithContentType > file extension > sniffing the first 512 bytes.

type UploadOption

type UploadOption func(*uploadConfig)

UploadOption configures a single upload (or a presigned PUT, where each value becomes a signed header the uploader must send back).

func WithCacheControl

func WithCacheControl(v string) UploadOption

WithCacheControl sets the Cache-Control header stored with the object.

func WithContentDisposition

func WithContentDisposition(v string) UploadOption

WithContentDisposition sets the Content-Disposition header stored with the object (e.g. `attachment; filename="report.pdf"`).

func WithContentType

func WithContentType(ct string) UploadOption

WithContentType skips detection and sets the value as-is.

func WithMetadata

func WithMetadata(m map[string]string) UploadOption

WithMetadata attaches user metadata (x-amz-meta-*).