s3

package module
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 27, 2026 License: MIT Imports: 14 Imported by: 0

README

s3

Go Reference Go Report Card

A samsara-compatible S3 component backed by the AWS SDK v2.

Works with any S3-compatible storage: AWS S3, Yandex Cloud Object Storage, Cloudflare R2, SeaweedFS, and others.

go get github.com/sunkek/samsara-components/s3

Usage

Register with a supervisor
store := s3.New(s3.Config{
    Endpoint: "https://s3.us-east-1.amazonaws.com",
    Region:   "us-east-1",
    KeyID:    os.Getenv("S3_KEY_ID"),
    Secret:   os.Getenv("S3_SECRET"),
})
sup.Add(store,
    samsara.WithTier(samsara.TierSignificant),
    samsara.WithRestartPolicy(samsara.AlwaysRestart(5*time.Second)),
)

For SeaweedFS (local development / CI):

store := s3.New(s3.Config{
    Endpoint:         "http://localhost:8333",
    Region:           "us-east-1",
    KeyID:            "test",
    Secret:           "test",
    PathStyleForcing: true, // required for SeaweedFS
})
Upload and download
// Upload with auto-detected content type:
err := store.Upload(ctx, s3.UploadRequest{
    Bucket: "my-bucket",
    Key:    "images/photo.jpg",
    Body:   file,
})

// Upload with explicit content type:
err := store.Upload(ctx, s3.UploadRequest{
    Bucket:      "my-bucket",
    Key:         "data/export.csv",
    Body:        csvReader,
    ContentType: "text/csv",
    ACL:         s3.ACLPrivate,
})

// Download:
rc, err := store.Download(ctx, "my-bucket", "images/photo.jpg")
if err != nil { return err }
defer rc.Close()
Delete objects
// Delete a single object:
err := store.Delete(ctx, "my-bucket", "images/old-photo.jpg")

// Delete all objects under a prefix:
n, err := store.DeleteByPrefix(ctx, "my-bucket", "tmp/session-123/")
List keys
keys, err := store.ListKeys(ctx, "my-bucket", "images/")
Presigned URLs
// Presigned download URL (valid for 1 hour):
url, err := store.PresignDownload(ctx, s3.PresignRequest{
    Bucket: "my-bucket",
    Key:    "reports/q4-2026.pdf",
    TTL:    time.Hour,
})

// Presigned upload URL (client uploads directly to S3):
url, err := store.PresignUpload(ctx, s3.PresignRequest{
    Bucket:        "my-bucket",
    Key:           "uploads/user-avatar.png",
    TTL:           15 * time.Minute,
    ContentType:   "image/png",
    ContentLength: 5 * 1024 * 1024,
})

When ContentType or ContentLength is set, the uploader must send matching Content-Type and Content-Length headers with the PUT request.

Configuration

s3.Config{
    Endpoint         string        // S3 endpoint URL; leave empty for AWS
    Region           string        // AWS region or equivalent
    KeyID            string        // access key ID
    Secret           string        // secret access key

    ConnectTimeout   time.Duration // default: 10s — startup check deadline
    PresignTTL       time.Duration // default: 15m — presigned URL lifetime
    PathStyleForcing bool          // default: false; set true for SeaweedFS / local servers
}
Options
s3.WithLogger(slog.Default())    // attach a structured logger
s3.WithName("media-store")       // override component name

API reference

Operations
Method Description
Upload(ctx, UploadRequest) Put an object; auto-detects MIME type
Download(ctx, bucket, key) Get an object; caller closes returned io.ReadCloser
Delete(ctx, bucket, key) Remove a single object
DeleteByPrefix(ctx, bucket, prefix) Remove all objects under prefix; returns count
ListKeys(ctx, bucket, prefix) List all object keys under prefix
PresignDownload(ctx, PresignRequest) Generate a time-limited GET URL
PresignUpload(ctx, PresignRequest) Generate a time-limited PUT URL; can sign exact Content-Type and Content-Length
ACL constants
Constant Value
ACLPrivate "private"
ACLPublicRead "public-read"
ACLPublicReadWrite "public-read-write"
ACLAuthenticatedRead "authenticated-read"
ACLBucketOwnerRead "bucket-owner-read"
ACLBucketOwnerFullControl "bucket-owner-full-control"

Content-type detection

Upload auto-detects the MIME type from the first 512 bytes of the body when UploadRequest.ContentType is not set. SVG files are detected by file extension (.svg) or body content (<svg prefix), since Go's http.DetectContentType does not recognise SVG natively.

Set ContentType explicitly to bypass detection:

store.Upload(ctx, s3.UploadRequest{
    ContentType: "application/octet-stream",
    ...
})

Presigned upload constraints

PresignUpload can sign exact Content-Type and Content-Length values for PUT uploads. It cannot express a size range like x-amz-content-length-range; that requires a presigned POST policy or validation in the caller before the URL is returned.

Health checking

*Component implements samsara.HealthChecker. Health is verified by sending a HeadBucket request to a synthetic bucket name. A 404 or 403 response confirms the endpoint is reachable and credentials are signing correctly — no ListBuckets permission required.

Integration tests (SeaweedFS)

Integration tests run against a local SeaweedFS instance started by docker-compose. SeaweedFS is an Apache 2.0 licensed, S3-compatible object store that needs no account, license key, or external service to run.

The seaweedfs service in docker-compose.yml runs SeaweedFS in single-node mode (server -s3) with credentials supplied via scripts/seaweedfs-s3.json. A one-shot seaweedfs-init service creates the test bucket once the gateway is healthy.

make infra-up
make test-integration

SeaweedFS credentials used in tests:

Parameter Value
Endpoint http://localhost:8333
Region us-east-1
Access key test
Secret key test
Bucket test
Path-style true

Documentation

Overview

Package s3 provides a github.com/sunkek/samsara-compatible S3 component backed by the AWS SDK v2.

It works with any S3-compatible storage provider: AWS S3, Yandex Cloud Object Storage, SeaweedFS, Cloudflare R2, and others.

Usage 

store := s3.New(s3.Config{
    Endpoint: "https://s3.us-east-1.amazonaws.com",
    Region:   "us-east-1",
    KeyID:    "AKIAIOSFODNN7EXAMPLE",
    Secret:   "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
})
sup.Add(store,
    samsara.WithTier(samsara.TierSignificant),
    samsara.WithRestartPolicy(samsara.AlwaysRestart(5*time.Second)),
)

Domain adapters receive *Component and call Upload, Download, PresignUpload, PresignDownload, Delete, and ListKeys.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ACL 

type ACL string

ACL is an S3 canned ACL value. 

const (
	ACLPrivate                ACL = "private"
	ACLPublicRead             ACL = "public-read"
	ACLPublicReadWrite        ACL = "public-read-write"
	ACLAuthenticatedRead      ACL = "authenticated-read"
	ACLBucketOwnerRead        ACL = "bucket-owner-read"
	ACLBucketOwnerFullControl ACL = "bucket-owner-full-control"
)

type Component 

type Component struct {
	// contains filtered or unexported fields
}

Component is a samsara-compatible S3 component. Obtain one with New; register it with a samsara supervisor.

func New 

func New(cfg Config, opts ...Option) *Component

New creates a Component from the supplied config. The S3 client is not initialised until Component.Start is called.

func (*Component) Delete 

func (c *Component) Delete(ctx context.Context, bucket, key string) error

Delete removes a single object from S3.

func (*Component) DeleteByPrefix 

func (c *Component) DeleteByPrefix(ctx context.Context, bucket, prefix string) (int, error)

DeleteByPrefix removes all objects whose keys begin with prefix. Returns the number of objects deleted. Handles pagination automatically.

func (*Component) Download 

func (c *Component) Download(ctx context.Context, bucket, key string) (io.ReadCloser, error)

Download retrieves an object from S3. The caller must close the returned io.ReadCloser after reading.

func (*Component) Health 

func (c *Component) Health(ctx context.Context) error

Health implements samsara.HealthChecker. Returns nil if the endpoint is reachable and credentials are valid.

func (*Component) ListKeys 

func (c *Component) ListKeys(ctx context.Context, bucket, prefix string) ([]string, error)

ListKeys returns all object keys in bucket with the given prefix. Handles pagination automatically; safe for large buckets.

func (*Component) Name 

func (c *Component) Name() string

Name implements samsara.Component.

func (*Component) PresignDownload 

func (c *Component) PresignDownload(ctx context.Context, r PresignRequest) (string, error)

PresignDownload generates a time-limited presigned URL for downloading an object. The URL is valid for [PresignRequest.TTL] or [Config.PresignTTL] if TTL is 0.

func (*Component) PresignUpload 

func (c *Component) PresignUpload(ctx context.Context, r PresignRequest) (string, error)

PresignUpload generates a time-limited presigned URL for uploading an object via HTTP PUT. The URL is valid for [PresignRequest.TTL] or [Config.PresignTTL].

If [PresignRequest.ContentType] or [PresignRequest.ContentLength] is set, the client must send matching Content-Type / Content-Length headers when using the returned URL or the upload will fail signature validation.

func (*Component) Start 

func (c *Component) Start(ctx context.Context, ready func()) error

Start loads the AWS config, initialises the S3 client, verifies connectivity, calls ready(), then blocks until Stop or ctx cancellation.

Connectivity is verified using HeadBucket on a well-formed but likely nonexistent bucket — this exercises the signing chain and endpoint without requiring ListBuckets permission (which is often not granted to service-account credentials).

Start is safe to call multiple times across restarts.

func (*Component) Stop 

func (c *Component) Stop(_ context.Context) error

Stop signals Start to return. The S3 client is stateless and holds no persistent connections, so there is nothing to close.

func (*Component) Upload 

func (c *Component) Upload(ctx context.Context, r UploadRequest) error

Upload puts an object into S3. The MIME type is auto-detected from the first 512 bytes of Body unless [UploadRequest.ContentType] is set explicitly.

type Config 

type Config struct {
	// Endpoint is the S3 endpoint URL.
	// Required for non-AWS providers (Yandex, MinIO, R2, etc.).
	// Leave empty to use the default AWS endpoint.
	Endpoint string

	// Region is the AWS region or equivalent.
	// Example: "us-east-1", "us-east-1".
	Region string

	// KeyID is the access key ID (AWS_ACCESS_KEY_ID equivalent).
	KeyID string

	// Secret is the secret access key (AWS_SECRET_ACCESS_KEY equivalent).
	Secret string

	// ConnectTimeout is the deadline for the initial connectivity check
	// during Start. Defaults to 10 s.
	ConnectTimeout time.Duration

	// PresignTTL is the default TTL for presigned URLs.
	// Individual calls can override this. Defaults to 15 minutes.
	PresignTTL time.Duration

	// PathStyleForcing enables path-style S3 addressing (bucket in URL path
	// instead of subdomain). Required by MinIO and some other providers.
	PathStyleForcing bool
}

Config holds all connection parameters for the S3 component.

type Logger 

type Logger interface {
	Info(msg string, args ...any)
	Error(msg string, args ...any)
}

Logger is satisfied by log/slog.Logger and most structured loggers.

type Option 

type Option func(*Component)

Option configures a Component.

func WithLogger 

func WithLogger(l Logger) Option

WithLogger attaches a structured logger to the component. log/slog.Logger satisfies Logger directly.

func WithName 

func WithName(name string) Option

WithName overrides the component name returned by Component.Name. Useful when using multiple S3 providers with the same supervisor.

type PresignRequest 

type PresignRequest struct {
	// Bucket is the target bucket name. Required.
	Bucket string
	// Key is the object key. Required.
	Key string
	// TTL overrides [Config.PresignTTL] for this request.
	// Use 0 to use the component default.
	TTL time.Duration
	// ContentType signs an exact Content-Type header for presigned uploads.
	// Leave empty to avoid constraining the upload MIME type at the S3 layer.
	ContentType string
	// ContentLength signs an exact Content-Length header for presigned uploads.
	// Leave 0 to avoid constraining the upload size at the S3 layer.
	//
	// Presigned PUT URLs do not support a min/max size range such as
	// x-amz-content-length-range. Use a presigned POST policy or validate
	// size before issuing the URL if you need range-based enforcement.
	ContentLength int64
}

PresignRequest carries parameters for presigned URL generation.

type UploadRequest 

type UploadRequest struct {
	// Bucket is the target bucket name. Required.
	Bucket string
	// Key is the object key (path within the bucket). Required.
	Key string
	// Body is the object content. Required.
	Body io.Reader
	// ContentType overrides auto-detected MIME type.
	// Leave empty to auto-detect from the first 512 bytes of Body.
	ContentType string
	// ACL controls object access. Defaults to [ACLPrivate].
	ACL ACL
}

UploadRequest carries all parameters needed for Component.Upload.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.