s3cache

README

caddy-fs-s3cache

A Caddy virtual filesystem (caddy.fs.s3cache) that serves objects from an S3 bucket through a local disk cache. Built for file_server in front of a media bucket, where every request used to cost three S3 round trips and nothing was remembered between requests.

Why

file_server on a plain S3 filesystem (for example caddy.fs.s3, which wraps jszwec/s3fs) performs, per served file:

1. Stat → HeadObject
2. Open → GetObject
3. http.ServeContent seeks to the end and back → the running GetObject is aborted and a second GetObject is issued

Measured in a TYPO3 tenant in eu-north-1, that is 73–85 ms for a 900-byte image, 51 ms for a miss, against 3 ms from local disk. This module fetches an object with one GetObject, keeps the body on local disk, answers Stat, Open, Seek and ReadAt from that copy, and asks S3 again only after revalidate_after, with If-None-Match (a 304 moves no body).

Case	S3 requests	Typical latency (in-region)
hot object (within revalidate_after)	0	< 1 ms
stale object, unchanged upstream	1 (304)	one round trip
cold or changed object	1 (200)	one round trip + body
missing key (within negative_ttl)	1, then 0	one round trip, then < 1 ms
32 concurrent first requests for one key	1	shared

Usage

{
    filesystem media s3cache {
        bucket my-media-bucket
        region eu-north-1
        # endpoint http://s3:9000      # S3-compatible stores
        # use_path_style
        # prefix media/                # key prefix for every path
        cache_dir /var/cache/caddy/media
        cache_max_size 512MiB
        revalidate_after 60s
        negative_ttl 30s
        keepalive 25s                  # background HeadBucket; `off` to disable
        max_idle_conns 16
    }
}

example.org {
    handle /media/* {
        route {
            uri strip_prefix /media
            root * .
            fs media
            file_server {
                pass_thru
            }
            respond "Not Found" 404
        }
    }
}

Credentials come from the AWS SDK default chain: environment, shared config, web identity (IRSA / AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE), instance metadata. The module needs s3:GetObject on the keys and, for keepalive, s3:ListBucket on the bucket.

Options

Option	Default	Meaning
bucket	required	S3 bucket
region	SDK default	AWS region
endpoint	AWS	custom endpoint (RustFS, MinIO, …)
use_path_style	off	path-style addressing
prefix	none	prepended to every key; a trailing / is added if missing
cache_dir	$TMPDIR/caddy-fs-s3cache/<bucket>	body files; emptied when the module is provisioned
cache_max_size	512MiB	LRU bound on cached bytes (KiB/MiB/GiB/KB/MB/GB or bytes)
revalidate_after	60s	serve from cache this long before a conditional GET
negative_ttl	30s	remember a 404 this long
keepalive	25s	interval of a HeadBucket that keeps the TLS connection warm; off disables
max_idle_conns	16	pooled connections to S3

Semantics worth knowing

• Stat and Open are answered from the same cached copy, so Last-Modified and Caddy's ETag (mtime + size) follow the object's Last-Modified.
• A replaced object (same key, new ETag) becomes visible at the next revalidation, at most revalidate_after after the write. A deleted object turns into a 404 the same way, and its body is evicted.
• When S3 fails (network, 5xx, 403) and a cached copy exists, the copy is served and a warning is logged. Without a copy, the error propagates and file_server answers 500 rather than a misleading 404.
• The root . is a synthetic directory; everything else is treated as an object key. Paths that are not valid fs.FS names (.., leading /, trailing /) are rejected without touching S3.
• cache_dir must be private to one module instance. In Kubernetes an emptyDir is the right volume: the index lives in memory and starts empty, so the directory is wiped on provision.

Development

go test ./...
go run github.com/caddyserver/xcaddy/cmd/xcaddy@latest build v2.11.7 \
  --with github.com/ohartwig/caddy-fs-s3cache=. --output ./caddy

Unit tests run against an in-process fake S3 and count requests. For an end-to-end run, point endpoint at a local RustFS or MinIO.

Licence

MIT, see LICENSE.

Documentation

Overview

Package s3cache implements a Caddy virtual filesystem (`caddy.fs.s3cache`) that serves objects from an S3 bucket through a local disk cache.

It exists because `file_server` on a plain S3 filesystem costs three round trips per file (HEAD for Stat, GET for Open, and a second GET when http.ServeContent seeks back to the start) and nothing is remembered between requests. This module fetches an object once with a single GET, keeps it on local disk, answers Stat and Open from that copy, and revalidates with If-None-Match after a configurable interval. A hot object is served without any S3 request; a cold object costs one.

Index

• Constants
• type FS
  • func (FS) CaddyModule() caddy.ModuleInfo
  • func (f *FS) Cleanup() error
  • func (f *FS) Open(name string) (fs.File, error)
  • func (f *FS) Provision(ctx caddy.Context) error
  • func (f *FS) Stat(name string) (fs.FileInfo, error)
  • func (f *FS) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

Constants

const (
	DefaultCacheMaxSize    int64 = 512 << 20 // 512 MiB
	DefaultRevalidateAfter       = 60 * time.Second
	DefaultNegativeTTL           = 30 * time.Second
	DefaultKeepalive             = 25 * time.Second
	DefaultMaxIdleConns          = 16
)

Defaults applied by Provision when a field is zero.

Types

type FS

type FS struct {
	// Bucket is the S3 bucket name (required).
	Bucket string `json:"bucket,omitempty"`

	// Region is the AWS region of the bucket. Empty falls back to the SDK's
	// default resolution (AWS_REGION and friends).
	Region string `json:"region,omitempty"`

	// Endpoint overrides the S3 endpoint, for S3-compatible stores.
	Endpoint string `json:"endpoint,omitempty"`

	// UsePathStyle switches to path-style addressing (needed by most
	// S3-compatible test servers).
	UsePathStyle bool `json:"use_path_style,omitempty"`

	// Prefix is prepended to every key. A filesystem path `a/b.jpg` with
	// prefix `media/` reads the object `media/a/b.jpg`.
	Prefix string `json:"prefix,omitempty"`

	// CacheDir holds the cached object bodies. Default: a directory below
	// os.TempDir(). The directory is emptied when the module is provisioned,
	// because the in-memory index starts empty and would otherwise leak files.
	CacheDir string `json:"cache_dir,omitempty"`

	// CacheMaxSize bounds the cache in bytes; least recently used objects are
	// evicted first. Default 512 MiB.
	CacheMaxSize int64 `json:"cache_max_size,omitempty"`

	// RevalidateAfter is how long a cached object is served without asking S3
	// whether it changed. After that, the next access sends a conditional GET
	// (If-None-Match); a 304 costs one round trip and no body. Default 60s.
	RevalidateAfter caddy.Duration `json:"revalidate_after,omitempty"`

	// NegativeTTL is how long a missing key stays missing without asking S3
	// again. Default 30s.
	NegativeTTL caddy.Duration `json:"negative_ttl,omitempty"`

	// Keepalive is the interval of a background HeadBucket request that keeps
	// a TLS connection to S3 open, so the first request after a quiet period
	// does not pay for a new handshake. 0 disables it. Default 25s.
	Keepalive caddy.Duration `json:"keepalive,omitempty"`

	// MaxIdleConns bounds the pooled connections to S3. Default 16.
	MaxIdleConns int `json:"max_idle_conns,omitempty"`
	// contains filtered or unexported fields
}

FS is a Caddy virtual filesystem backed by an S3 bucket with a local disk cache.

func (FS) CaddyModule

func (FS) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (*FS) Cleanup

func (f *FS) Cleanup() error

Cleanup implements caddy.CleanerUpper.

func (*FS) Open

func (f *FS) Open(name string) (fs.File, error)

Open implements fs.FS.

func (*FS) Provision

func (f *FS) Provision(ctx caddy.Context) error

Provision implements caddy.Provisioner.

func (*FS) Stat

func (f *FS) Stat(name string) (fs.FileInfo, error)

Stat implements fs.StatFS.

func (*FS) UnmarshalCaddyfile

func (f *FS) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

UnmarshalCaddyfile implements caddyfile.Unmarshaler.

filesystem <name> s3cache {
    bucket <name>
    region <region>
    endpoint <url>
    use_path_style
    prefix <key-prefix>
    cache_dir <path>
    cache_max_size <bytes>
    revalidate_after <duration>
    negative_ttl <duration>
    keepalive <duration>|off
    max_idle_conns <n>
}