configawss3

README

config-aws-s3

Read and write a config file that lives in an AWS S3 bucket through config

Part of the phpboyscout Go toolkit — small, framework-free Go modules extracted from go-tool-base. Documented with the parent module at config.go.phpboyscout.uk

---

config reads and writes configuration through a small six-method filesystem interface. This module is an implementation of that interface backed by an S3 bucket, so a tool whose config.yaml lives as an object in S3 can load, watch and write it exactly as it would a local file — same codec, same precedence, same stage-and-commit write path.

It is a filesystem adapter, not a backend: it reads and writes a config file that happens to live in a bucket. It does not model object keys as config keys — that is the parameter-store shape config-aws-ssm implements, and a different thing.

You build and configure the *s3.Client — that is where every region, credential, endpoint and retry decision lives — and hand it in with the bucket name:

import (
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	cfg "gitlab.com/phpboyscout/go/config"
	configawss3 "gitlab.com/phpboyscout/go/config-aws-s3"
)

awsConfig, _ := config.LoadDefaultConfig(ctx)
client := s3.NewFromConfig(awsConfig)

store, err := cfg.NewStore(ctx,
	cfg.WithFiles(configawss3.Wrap(client, "my-bucket"), "app/config.yaml"))

The config.FS name ("app/config.yaml") is the object key, used verbatim against the bucket. Reading, hot-reload and writing all work as they do over the local filesystem.

Scoping a shared bucket

WithKeyPrefix joins a prefix onto each name to form the object key, so a bucket shared with unrelated objects can be carved out for configuration:

configawss3.Wrap(client, "my-bucket", configawss3.WithKeyPrefix("team/app"))
// the name "config.yaml" resolves to the object key "team/app/config.yaml"

With no prefix, the name is used as the key verbatim.

How a write commits — and the one caveat

config commits a write by staging the new content beside the target and renaming over it, so a reader never observes a half-written file. S3 has no atomic move, so Rename is copy-then-delete (CopyObject + DeleteObject). Both the target replacement and the initial put are atomic per object — the target is only ever swapped whole, never spliced, never lost — and because S3 is strongly read-after-write consistent (since Dec 2020) the new content is immediately visible.

The single imperfection is that a crash between the commit copy and its cleanup can leave the staged object behind. To keep that orphan findable, the staging object uses a deterministic, recognisable key: the resolved target key with a .config-stage suffix (for app/config.yaml, the stage is app/config.yaml.config-stage). This is a leak, never corruption and never a lost write — the target is always intact. Expire orphans with a bucket lifecycle rule matching the .config-stage suffix; the module does no opportunistic cleanup of prior orphans.

No directories, no POSIX mode

• MkdirAll is a no-op. Object stores have no directories; a key containing / is simply a flat key.
• WriteFile ignores its perm argument and Stat synthesises a fixed 0o644 mode with a nil Sys(). S3 has no POSIX permissions, so the mode is nominal and round-trips harmlessly through the Store's preserve-permissions step.

Watching

S3 has no native change feed, so watch is polling. Because every poll is a billed GetObject, the adapter implements config.PollIntervalHinter and hints a 60-second cadence rather than the core's 2-second default — bounding the bill at ~1 call/min while still catching a changed object within a minute. config.WithPollInterval(d) overrides the hint when passed.

S3-compatible stores (MinIO, Ceph, LocalStack)

Because the consumer owns the client's endpoint, the same adapter drives any S3-compatible store by pointing BaseEndpoint at it (usually with o.UsePathStyle = true). The adapter is oblivious to which it is talking to — the LocalStack integration suite in ./test/integration exercises exactly this path.

Injecting the client

Wrap takes the concrete *s3.Client — the six object operations map one-to-one onto its methods, so there is no narrow public interface to learn. Internally the adapter depends on a small unexported interface (the operations it calls), which is what lets the whole unit suite run against an in-memory fake object store with no AWS account.

What it costs

The config graph plus the AWS SDK for Go v2, pulling only the S3 service — eleven modules (aws-sdk-go-v2, service/s3, smithy-go and S3's required internals: eventstream, configsources, endpoints/v2, v4a, accept-encoding, checksum, presigned-url, s3shared), asserted by an allowlist test so an unforeseen transitive addition fails the build rather than arriving quietly. A consumer who also builds the client with config.LoadDefaultConfig pulls a larger credential-loading graph, but that is theirs, not the adapter's. The testcontainers-go / LocalStack integration suite is a test-only dependency and reaches no consumer.

License

See LICENSE.

Documentation

Overview

Package configawss3 adapts an AWS S3 bucket to the config.FS interface, so a tool whose configuration lives as an object in S3 can hand that bucket to a config Store and read, watch and write it exactly as it would a local file.

client := s3.NewFromConfig(awsCfg)
store, err := config.NewStore(ctx,
	config.WithFiles(configawss3.Wrap(client, "my-bucket"), "app/config.yaml"))

This is a filesystem adapter, not a backend: it reads and writes a config *file* that happens to live in a bucket, through the same codec and stage-and-rename write machinery a local file uses. It does not model object keys as config keys — that is the parameter-store shape config-aws-ssm implements, and a different thing.

What an object store makes different

S3 is not a POSIX filesystem, and the divergences are the substance:

• Rename is not atomic. S3 has no move, so Wrap's Rename is copy-then-delete (CopyObject then DeleteObject). The target is always replaced whole — a reader never sees a half-written object — but a crash between the copy and the delete leaves the staged object behind. To keep that orphan findable, the staged object uses a recognisable, target-derived key: the resolved target key with a ".config-stage" suffix, which a bucket lifecycle rule can expire. The leak is never corruption and never a lost write.

• There are no directories, so MkdirAll is a no-op.

• There is no POSIX mode, so WriteFile ignores its perm argument and Stat synthesises a fixed 0o644 regular-file mode with a nil Sys.

• There is no native change feed, so the Store polls. Because every poll is a billed GetObject, the adapter implements config.PollIntervalHinter and hints a 60-second cadence rather than the core's 2-second default; config.WithPollInterval still overrides it.

Because the consumer supplies the *s3.Client, they own its region, credentials, endpoint and retry policy — which is what lets the very same adapter drive an S3-compatible store (MinIO, Ceph, LocalStack) by pointing the client's BaseEndpoint at it.

Index

• Variables
• func FSFromConfig(cfg aws.Config, bucket string, opts ...Option) (config.FS, error)
• func Wrap(client *s3.Client, bucket string, opts ...Option) config.FS
• type Option
  • func WithKeyPrefix(prefix string) Option

Examples

• WithKeyPrefix
• Wrap

Variables

var ErrNoBucket = errors.NewSentinel("configawss3.no_bucket",
	"no bucket supplied; pass the bucket the configuration object lives in")

ErrNoBucket reports a rung called with no bucket name.

var ErrNoRegion = errors.NewSentinel("configawss3.no_region",
	"no AWS region in the supplied config; set one, or set AWS_REGION for the ambient rung")

ErrNoRegion reports a configuration that names no AWS region.

A config with no region names a bucket in an account nobody chose, so it is refused here rather than failing at the first request. There is no default: AWS documents none, and inventing one is a different act from adopting a provider's own — which is why go/awsclient refuses too.

Functions

func FSFromConfig

func FSFromConfig(cfg aws.Config, bucket string, opts ...Option) (config.FS, error)

FSFromConfig builds the S3 client from an aws.Config the caller resolved — a specific profile, an assumed role, a client pointed at MinIO or LocalStack — and returns a config.FS over the bucket.

s3.NewFromConfig does no I/O, returns no error and needs no close, so this rung constructs eagerly and reaching the bucket stays deferred to the first read.

Where the ambient rung is

Deliberately NOT here. Resolving the ambient AWS chain costs this module seven further dependencies — fewer than its Parameter Store and Secrets Manager siblings pay, because S3's larger service graph already carries internal/v4a, accept-encoding and presigned-url — but seven a consumer who builds their own client should not inherit.

It lives in the ambient subpackage instead, over go/awsclient:

import s3ambient "gitlab.com/phpboyscout/go/config-aws-s3/ambient"

fsys, err := s3ambient.Default(ctx, "my-bucket")

To share ONE resolved chain across several adapters — and across go/signing and go/encryption — resolve it with go/awsclient and pass the config here (spec 0012 L-4, L-5).

func Wrap

func Wrap(client *s3.Client, bucket string, opts ...Option) config.FS

Wrap adapts a configured *s3.Client and bucket to a read+write config.FS.

The client carries every credential, region, endpoint and retry decision; Wrap adds none — it only translates the six config.FS calls onto object operations. The returned value is neither a config.RealPather (an object key is not an operating-system path) nor a config.LinkReader (S3 has no symlinks), so the Store polls it — on the adapter's 60-second config.PollIntervalHinter hint.

Example

ExampleWrap reads a config object from an S3 bucket through a config Store. The consumer builds the *s3.Client — owning region, credentials and endpoint — and hands it to Wrap; the adapter only translates the filesystem calls onto object operations.

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	cfg "gitlab.com/phpboyscout/go/config"

	configawss3 "gitlab.com/phpboyscout/go/config-aws-s3"
)

func main() {
	ctx := context.Background()

	awsCfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		log.Fatal(err)
	}

	client := s3.NewFromConfig(awsCfg)

	store, err := cfg.NewStore(ctx,
		cfg.WithFiles(configawss3.Wrap(client, "my-bucket"), "app/config.yaml"))
	if err != nil {
		log.Fatal(err)
	}

	_ = store.View().GetString("server.host")
}

Types

type Option

type Option func(*adapter)

Option configures the adapter. See WithKeyPrefix.

func WithKeyPrefix

func WithKeyPrefix(prefix string) Option

WithKeyPrefix scopes the bucket by joining prefix onto each config.FS name to form the object key, so a bucket shared with unrelated objects can be carved out for configuration. The staging key a write commits through derives from the resolved (prefixed) target key, so prefix and staging compose cleanly. With no prefix, a name is used as the object key verbatim (spec D3).

Example

ExampleWithKeyPrefix scopes a bucket shared with unrelated objects: the config.FS name "config.yaml" resolves to the object key "team/app/config.yaml".

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	cfg "gitlab.com/phpboyscout/go/config"

	configawss3 "gitlab.com/phpboyscout/go/config-aws-s3"
)

func main() {
	ctx := context.Background()

	awsCfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		log.Fatal(err)
	}

	client := s3.NewFromConfig(awsCfg)

	store, err := cfg.NewStore(ctx,
		cfg.WithFiles(
			configawss3.Wrap(client, "my-bucket", configawss3.WithKeyPrefix("team/app")),
			"config.yaml"))
	if err != nil {
		log.Fatal(err)
	}

	_ = store.View().GetString("server.host")
}