s3

README

certmagic-s3 (best-of-breed)

A CertMagic / Caddy storage module that keeps ACME certificates, keys and metadata in any S3-compatible object store (AWS S3, MinIO, Cloudflare R2, Backblaze B2, GCS S3-interop, …).

It is built on aws-sdk-go-v2 and provides:

• Distributed locking that is actually safe — atomic If-None-Match create-only lock objects with a background lease-refresh, stale-lock stealing, and blocking until the caller's context is cancelled (no fixed give-up timeout). Two Caddy instances can never both believe they hold the same lock.
• Optional client-side encryption at rest — NaCl secretbox with a 32-byte key.
• Correct directory-style listing — honours the requested prefix, uses the S3 delimiter for non-recursive listing (immediate children only) and returns keys relative to the storage root, exactly as CertMagic expects.
• Normalized keys — keys never carry a leading slash, matching CertMagic's storage contract and staying portable across every S3 provider and console.
• Rich configuration: custom endpoint, region, static keys / AWS profile / assumed IAM role, path-style addressing, TLS-skip for testing.

Install (build Caddy with xcaddy)

xcaddy build --with github.com/igk1972/certmagic-s3

Or with Docker (see Dockerfile):

docker build -t caddy-s3 .

The Caddy storage module ID is caddy.storage.s3. Only one storage module with that ID can be compiled into a Caddy binary, so this is a drop-in replacement for other caddy.storage.s3 implementations, not an addition alongside them.

Configure

In the Caddyfile global options block:

{
	storage s3 {
		bucket         my-bucket
		region         us-east-1
		access_key     {env.AWS_ACCESS_KEY_ID}
		secret_key     {env.AWS_SECRET_ACCESS_KEY}
		prefix         acme
		encryption_key {env.CERTMAGIC_S3_ENCRYPTION_KEY}
	}
}

Custom S3 provider (e.g. MinIO):

{
	storage s3 {
		endpoint   https://minio.example.com
		bucket     my-bucket
		access_key {env.S3_ACCESS_KEY}
		secret_key {env.S3_SECRET_KEY}
	}
}

Options

Option	Default	Description
bucket	— (required)	Bucket name.
endpoint	—	Custom endpoint URL for non-AWS providers. Enables path-style addressing automatically.
host	—	Deprecated, use endpoint. If set, becomes https://<host>. Mutually exclusive with endpoint.
region	us-east-1	AWS region.
access_key / secret_key	—	Static credentials. If omitted, the default AWS credential chain is used (env vars, shared config, IMDS…).
profile	—	Named AWS shared-config profile (used when no static keys given).
role_arn	—	IAM role ARN to assume via STS.
prefix	acme	Key prefix (namespace) inside the bucket. Surrounding slashes are ignored.
encryption_key	—	Exactly 32 bytes → client-side secretbox encryption. Empty → plaintext (rely on server-side encryption if desired).
use_path_style	false	Force path-style addressing. Auto-enabled when a custom endpoint is set.
insecure	false	Skip TLS verification. Testing only.

Values support Caddy placeholders, so secrets are best supplied via {env.*}.

Credentials also work without access_key/secret_key through the standard AWS chain (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, AWS_PROFILE, EC2/ECS roles), or by setting profile / role_arn.

Migrating from a leading-slash layout

Older plugins (ss098's certmagic-s3, and earlier revisions of this one) could write object keys with a leading slash — e.g. /certificates/... — when configured with a / prefix. This module normalizes keys (no leading slash) per CertMagic's contract, so it will not find objects stored under the old leading-slash layout.

If you are migrating such a bucket, run the one-off helper in migrate/ once. It is not part of the built plugin (it carries a //go:build ignore tag); run it directly with go run. It is dry-run by default:

# preview
go run migrate/leading-slash.go \
    --endpoint https://minio.example.com --bucket my-bucket \
    --access-key "$S3_ACCESS_KEY" --secret-key "$S3_SECRET_KEY"

# apply (server-side CopyObject + DeleteObject for every "/…" key)
go run migrate/leading-slash.go \
    --endpoint https://minio.example.com --bucket my-bucket \
    --access-key "$S3_ACCESS_KEY" --secret-key "$S3_SECRET_KEY" \
    --dry-run=false

CertMagic re-obtains anything genuinely missing, so the migration is safe to run once.

Development

make test        # go test ./...
make test-race   # go test -race ./...
make lint        # golangci-lint
make fmt         # gofumpt -w .

The TestS3_MinIOInterop end-to-end test is opt-in — set CERTMAGIC_S3_RUN_MINIO_INTEGRATION=1 plus CERTMAGIC_S3_MINIO_BUCKET (and optionally CERTMAGIC_S3_MINIO_ENDPOINT / _ACCESS_KEY / _SECRET_KEY) to run it against a real MinIO.

Credits & license

Forked lineage: @thomersch (original generic-S3 library) → @techknowlogick (aws-sdk-go-v2 rewrite + Caddy module) → this best-of-breed variant.

Licensed under Apache 2.0.

Documentation

Index

• Constants
• Variables
• type CleartextIO
  • func (ci *CleartextIO) ByteReader(buf []byte) Reader
  • func (ci *CleartextIO) WrapReader(r io.Reader) io.Reader
• type IO
• type Reader
  • func (r *Reader) Len() int64
  • func (r *Reader) Read(buf []byte) (int, error)
  • func (r *Reader) Seek(offset int64, whence int) (int64, error)
• type S3
  • func (s3 *S3) CaddyModule() caddy.ModuleInfo
  • func (s3 *S3) CertMagicStorage() (certmagic.Storage, error)
  • func (s3 *S3) Delete(ctx context.Context, key string) error
  • func (s3 *S3) Exists(ctx context.Context, key string) bool
  • func (s3 *S3) List(ctx context.Context, prefix string, recursive bool) ([]string, error)
  • func (s3 *S3) Load(ctx context.Context, key string) ([]byte, error)
  • func (s3 *S3) Lock(ctx context.Context, key string) error
  • func (s3 *S3) Provision(ctx caddy.Context) error
  • func (s3 *S3) Stat(ctx context.Context, key string) (certmagic.KeyInfo, error)
  • func (s3 *S3) Store(ctx context.Context, key string, value []byte) error
  • func (s3 *S3) Unlock(ctx context.Context, key string) error
  • func (s3 *S3) UnmarshalCaddyfile(d *caddyfile.Dispenser) error
• type SecretBoxIO
  • func NewSecretBoxIO(key [32]byte) *SecretBoxIO
  • func (sb *SecretBoxIO) ByteReader(msg []byte) Reader
  • func (sb *SecretBoxIO) IsValid() bool
  • func (sb *SecretBoxIO) WrapReader(r io.Reader) io.Reader

Constants

const (
	NonceSize = 24
)

Variables

var (
	// LockExpiration is how long a held lock may go without a refresh before
	// another instance is allowed to steal it (e.g. after a crash). It must be
	// safely larger than LockRefreshInterval.
	LockExpiration = 2 * time.Minute
	// LockPollInterval is how long to wait between attempts to acquire a lock
	// that is currently held by someone else.
	LockPollInterval = 1 * time.Second
	// LockRefreshInterval is how often a held lock's timestamp is rewritten so
	// it does not appear stale while issuance (which certmagic serializes under
	// this lock across its full retry sequence) is still in progress.
	LockRefreshInterval = 30 * time.Second
)

Types

type CleartextIO

type CleartextIO struct{}

func (*CleartextIO) ByteReader

func (ci *CleartextIO) ByteReader(buf []byte) Reader

func (*CleartextIO) WrapReader

func (ci *CleartextIO) WrapReader(r io.Reader) io.Reader

type IO

type IO interface {
	WrapReader(io.Reader) io.Reader
	ByteReader([]byte) Reader
}

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func (*Reader) Len

func (r *Reader) Len() int64

func (*Reader) Read

func (r *Reader) Read(buf []byte) (int, error)

func (*Reader) Seek

func (r *Reader) Seek(offset int64, whence int) (int64, error)

type S3

type S3 struct {
	Logger *zap.Logger

	// S3
	Client       *s3sdk.Client
	Host         string `json:"host"`
	Endpoint     string `json:"endpoint"`
	Insecure     bool   `json:"insecure"`
	Bucket       string `json:"bucket"`
	Region       string `json:"region"`
	AccessKey    string `json:"access_key"`
	SecretKey    string `json:"secret_key"`
	Profile      string `json:"profile"`
	RoleARN      string `json:"role_arn"`
	Prefix       string `json:"prefix"`
	UsePathStyle bool   `json:"use_path_style,omitempty"`

	// EncryptionKey is optional. If you do not wish to encrypt your certficates and key inside the S3 bucket, leave it empty.
	EncryptionKey string `json:"encryption_key"`
	// contains filtered or unexported fields
}

func (*S3) CaddyModule

func (s3 *S3) CaddyModule() caddy.ModuleInfo

func (*S3) CertMagicStorage

func (s3 *S3) CertMagicStorage() (certmagic.Storage, error)

CertMagicStorage converts s to a certmagic.Storage instance.

func (*S3) Delete

func (s3 *S3) Delete(ctx context.Context, key string) error

func (*S3) Exists

func (s3 *S3) Exists(ctx context.Context, key string) bool

func (*S3) List

func (s3 *S3) List(ctx context.Context, prefix string, recursive bool) ([]string, error)

func (*S3) Load

func (s3 *S3) Load(ctx context.Context, key string) ([]byte, error)

func (*S3) Lock

func (s3 *S3) Lock(ctx context.Context, key string) error

Lock acquires a distributed lock for key, blocking until it is obtained or ctx is cancelled. Acquisition is atomic: the lock object is created with a conditional (create-only) PutObject, so concurrent callers across instances can never both believe they hold the lock. Once held, the lock is refreshed in the background until Unlock so long-running issuance is not stolen.

func (*S3) Provision

func (s3 *S3) Provision(ctx caddy.Context) error

func (*S3) Stat

func (s3 *S3) Stat(ctx context.Context, key string) (certmagic.KeyInfo, error)

func (*S3) Store

func (s3 *S3) Store(ctx context.Context, key string, value []byte) error

func (*S3) Unlock

func (s3 *S3) Unlock(ctx context.Context, key string) error

func (*S3) UnmarshalCaddyfile

func (s3 *S3) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

type SecretBoxIO

type SecretBoxIO struct {
	SecretKey [32]byte
}

func NewSecretBoxIO

func NewSecretBoxIO(key [32]byte) *SecretBoxIO

func (*SecretBoxIO) ByteReader

func (sb *SecretBoxIO) ByteReader(msg []byte) Reader

func (*SecretBoxIO) IsValid

func (sb *SecretBoxIO) IsValid() bool

func (*SecretBoxIO) WrapReader

func (sb *SecretBoxIO) WrapReader(r io.Reader) io.Reader