configgcpgcs

README

config-gcp-gcs

Read and write a config file in a Google Cloud Storage bucket through config

Part of the phpboyscout Go toolkit — small, framework-free Go modules extracted from go-tool-base. Documented with the parent module at config.go.phpboyscout.uk

---

config reads and writes configuration files through a small six-method config.FS interface. config-gcp-gcs is an implementation of that interface backed by a Google Cloud Storage bucket, so a GCP-native workload — a Cloud Run service, a GKE pod, a batch job — can load, watch and write a config file that lives in GCS through the same codec and staging machinery a local file uses. A consumer who reads config from a local file, or from S3 or Azure, never compiles this module.

This reads a config file that happens to live in a bucket. It is not the object-keys-as-config-keys model — that is a different, backend-shaped thing.

Use it

You build and configure the *storage.Client — that is where every authentication, project, endpoint and retry decision lives — and hand it in with a bucket name. The adapter takes it from there:

import (
	"cloud.google.com/go/storage"
	"gitlab.com/phpboyscout/go/config"
	configgcpgcs "gitlab.com/phpboyscout/go/config-gcp-gcs"
)

client, err := storage.NewClient(ctx)          // auth, project, endpoint, retry — all yours
if err != nil { /* ... */ }
defer client.Close()

store, err := config.NewStore(ctx,
	config.WithFiles(configgcpgcs.Wrap(client, "my-bucket"), "app.yaml"))

A name passed to any method is the object key within the bucket (app.yaml, env/prod/app.yaml). An object store has a flat key namespace with / as a conventional separator, not real directories, so MkdirAll is a no-op and a nested key is written with no parent.

The whole exported surface is one function:

func Wrap(client *storage.Client, bucket string, opts ...Option) config.FS

What "atomic" honestly means over an object store

GCS is not a POSIX filesystem, and the adapter meets it honestly rather than pretending otherwise:

• WriteFile is atomic per object. The GCS Writer commits only when Close succeeds, so a concurrent reader sees either the whole previous object or the whole new one, never a half-written one. GCS is strongly read-after-write consistent, so the core's SHA-256 content-fingerprint conflict trap holds with no eventual-consistency caveat. (v0.1.0 relies on that shared machinery; GCS GenerationMatch preconditions are deferred to a later revision.)
• Rename uses the native server-side move (ObjectHandle.Move), a single-bucket atomic rename — one round-trip, no orphan-on-crash window, and the source is never left behind. The config write path stages content beside the target and renames over it; on GCS that rename is a real atomic move, not copy-then-delete.
• Stat synthesises an fs.FileInfo with real size and modification time, a nominal 0o644 mode (an object store has no POSIX permission bits) and Sys() == nil.
• MkdirAll returns nil. There are no directories to create.
• An absent object maps to fs.ErrNotExist, the one distinction the core branches on (a missing overlay is normal, a missing base file is fatal). Other errors — a 403, a 5xx, a quota error — are returned wrapped with the operation, key and bucket, not flattened onto an fs sentinel. A missing bucket fails loudly rather than being read as "optional source absent".

Watching costs money, so the poll cadence is conservative

A GCS object has no operating-system path, so config watches it by polling, and each poll is a billed object read. The adapter therefore implements config.PollIntervalHinter, returning 60 seconds, so the core's poll watcher adopts that cadence without you setting anything. config.WithPollInterval(d) still overrides the hint.

The base context

Every storage call needs a context.Context, but config.FS methods take none, so the adapter holds a base context that defaults to context.Background() and derives per-call contexts from it. Pass WithContext to tie the adapter's lifetime to a cancellable context:

configgcpgcs.Wrap(client, "my-bucket", configgcpgcs.WithContext(ctx))

Note that cancelling that context makes every subsequent read, write and stat fail — it is the lifetime of the adapter, not of a single call.

What it costs

This is the heaviest dependency graph in the adapter family: cloud.google.com/go/storage pulls the full first-party Google Cloud Go client stack — auth, IAM, the OpenTelemetry observability path, the s2a-go / SPIFFE mTLS path, and the gRPC / protobuf / genproto runtime. Measured against cloud.google.com/go/storage v1.64.0 plus the config graph, that is 54 third-party modules. An allowlist test (depfootprint_test.go) pins the exact set, so a version bump that grows the graph fails the build rather than arriving quietly. This is the honest cost of a first-party cloud SDK, and the whole reason GCS is its own module: a consumer who does not read config from GCS never compiles any of it.

Testing

The unit suite drives the whole adapter — every translation, the atomic-on-Close commit, the native move, the not-found mapping and the poll watcher — through a real config Store over an in-memory fake object store, with no network and no credentials.

An env-gated integration suite (INT_TEST_INTEGRATION=1, under test/integration) runs the adapter against fake-gcs-server in a container started by testcontainers-go, proving read, write-commit-on-close, stat, delete and not-found against the real protocol. The client there is pointed at the emulator per-client and hermetically via option.WithEndpoint(...) + option.WithoutAuthentication() (plus storage.WithJSONReads(), because fake-gcs-server serves reads only through the JSON API). fake-gcs-server does not yet implement the JSON objects.move endpoint the native Rename uses, so the real-move assertion skips against the emulator with that diagnosis; the move semantics are covered in full by the unit suite. testcontainers-go is a test-only dependency and reaches no consumer.

License

See LICENSE.

Documentation

Overview

Package configgcpgcs adapts a Google Cloud Storage bucket, reached through an already-configured storage.Client, to the config.FS interface, so a tool whose configuration file lives in a GCS bucket can load, watch and write it through the same codec and staging machinery a local file uses.

client, _ := storage.NewClient(ctx)
store, err := config.NewStore(ctx,
	config.WithFiles(configgcpgcs.Wrap(client, "my-bucket"), "app.yaml"))

A name passed to any method is the object key within the bucket (app.yaml, env/prod/app.yaml) — an object store has a flat key namespace with "/" as a conventional separator, not real directories, so MkdirAll is a no-op.

The consumer owns the client

Wrap takes a client the consumer has already built: authentication, project, endpoint, scopes and retry policy all live there, and the adapter re-decides none of it. It only translates the six config.FS calls onto the bucket's object handles. To point the adapter at the fake-gcs-server emulator, build the client with option.WithEndpoint(...) + option.WithoutAuthentication(); the adapter itself wires no endpoints.

What "atomic" honestly means over an object store

GCS is not POSIX, and this adapter meets it honestly:

• WriteFile is atomic per object: storage.Writer commits only when Close succeeds, so a concurrent reader sees either the whole previous object or the whole new one, never a half-written one. GCS is strongly read-after-write consistent, so the core's SHA-256 content-fingerprint conflict trap holds without an eventual-consistency caveat.
• Rename uses the native server-side storage.ObjectHandle.Move — a single-bucket atomic move — rather than copy-then-delete, so there is no orphan-on-crash window and the source is never left behind.
• Stat synthesises an fs.FileInfo with real size and modification time and a nominal 0o644 mode (an object store has no POSIX mode); Sys returns nil.
• MkdirAll returns nil: there are no directories to create.

Watching costs money, so the poll cadence is conservative

A GCS object has no operating-system path, so the adapter is not a config.RealPather: the core watches it by polling, and each poll is a billed object read. The adapter therefore implements config.PollIntervalHinter, returning 60 seconds, so the core's poll watcher adopts that cadence without the consumer setting anything. config.WithPollInterval still overrides it.

Example

Example shows the whole surface: build a storage client (auth, project and endpoint are the consumer's), wrap a bucket, and hand it to a config Store exactly as config.OS() would be. The Store then loads, watches and writes the object through the same codec a local file uses.

package main

import (
	"context"

	"cloud.google.com/go/storage"
	"gitlab.com/phpboyscout/go/config"

	configgcpgcs "gitlab.com/phpboyscout/go/config-gcp-gcs"
)

func main() {
	ctx := context.Background()

	client, err := storage.NewClient(ctx)
	if err != nil {
		return
	}
	defer func() { _ = client.Close() }()

	store, err := config.NewStore(ctx,
		config.WithFiles(configgcpgcs.Wrap(client, "my-bucket"), "app.yaml"))
	if err != nil {
		return
	}

	_ = store.View().GetString("server.host")
}

Index

• Constants
• Variables
• func Wrap(client *storage.Client, bucket string, opts ...Option) config.FS
• type Option
  • func WithContext(ctx context.Context) Option
• type OwnedFS
  • func Default(ctx context.Context, bucket string, opts ...Option) (*OwnedFS, error)
  • func FromOptions(ctx context.Context, bucket string, clientOpts []option.ClientOption, ...) (*OwnedFS, error)
  • func (f *OwnedFS) Close() error
  • func (f *OwnedFS) PollInterval() time.Duration

Examples

• Package

Constants

const PollInterval = 60 * time.Second

PollInterval is the cadence the adapter hints to the core's poll watcher. A poll is a billed object read, so this is deliberately far longer than the core's responsive local default; config.WithPollInterval overrides it.

Variables

var ErrNoBucket = errors.NewSentinel("configgcpgcs.no_bucket",
	"no bucket supplied; pass the bucket the configuration object lives in")

ErrNoBucket reports a rung called with no bucket name. A bucket name is globally unique in Google Cloud, so unlike Secret Manager and Parameter Manager this adapter needs no project — but it cannot invent the bucket.

Functions

func Wrap

func Wrap(client *storage.Client, bucket string, opts ...Option) config.FS

Wrap adapts a GCS bucket, reached through an already-configured client, to config.FS. The consumer owns the client (auth, project, endpoint, retry); the adapter owns only the six-method translation.

Every storage call needs a context.Context, but config.FS methods take none, so the adapter holds a base context that defaults to context.Background() and derives per-call contexts from it. Pass WithContext to tie the adapter's lifetime to a cancellable context — note that cancelling it makes every subsequent read fail.

Types

type Option

type Option func(*adapter)

Option configures the adapter returned by Wrap.

func WithContext

func WithContext(ctx context.Context) Option

WithContext sets the base context every storage call derives from, in place of the default context.Background(). Cancelling it makes every subsequent adapter operation fail.

type OwnedFS

type OwnedFS struct {
	config.FS
	// contains filtered or unexported fields
}

OwnedFS is a config.FS that built its own storage client and can release it.

The obligation here is real but weaker than its Secret Manager sibling's. storage.Client is HTTP-backed and its documentation says Close "need not be called at program exit", where secretmanager.Client is gRPC-backed and says it "must be Closed". So a one-shot CLI can ignore this entirely, and a long-lived service that builds and discards Stores should hand it over with config.WithCloser, so closing the Store releases it:

fsys, err := configgcpgcs.Default(ctx, "my-bucket")
if err != nil { return err }

store, err := config.NewStore(ctx,
	config.WithFiles(fsys, "config.yaml"), config.WithCloser(fsys))
if err != nil { return err }
defer store.Close()

Wrap returns a plain config.FS, because there the consumer built the client and still owns it — the obligation follows who built it, not which rung (spec 0012 L-7, R2).

func Default

func Default(ctx context.Context, bucket string, opts ...Option) (*OwnedFS, error)

Default builds a storage client from Application Default Credentials — the metadata server, a workload identity, GOOGLE_APPLICATION_CREDENTIALS, or gcloud's own login — and returns a filesystem over the bucket.

This is the zero-conf rung, and it is the simplest in the family: a bucket name is globally unique, so unlike config-gcp-secret and config-gcp-parameter there is no project or location to supply. ADC names the principal, the bucket names itself.

The returned filesystem owns the client and can be Closed; see OwnedFS.

func FromOptions

func FromOptions(
	ctx context.Context, bucket string, clientOpts []option.ClientOption, opts ...Option,
) (*OwnedFS, error)

FromOptions builds a storage client from client options the caller holds — an emulator endpoint, an explicit credentials file, or a credential resolved once with go/gcpclient and shared across the GCP siblings — and returns a filesystem over the bucket that owns it.

The returned filesystem owns the client and can be Closed; see OwnedFS.

func (*OwnedFS) Close

func (f *OwnedFS) Close() error

Close releases the storage client this filesystem built. It is safe to call more than once.

func (*OwnedFS) PollInterval

func (f *OwnedFS) PollInterval() time.Duration

PollInterval forwards the owned filesystem's cadence hint. Without it the core would poll a billed object read at its two-second file default.