mamori

package module
v1.1.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 20, 2026 License: MIT Imports: 23 Imported by: 0

README

守り  mamori

Typed, watchable config & secrets for Go

Load configuration and secrets from anywhere into validated Go structs - and keep them reconciled at runtime, without a restart.

Go Reference CI Go Report Card

mamori (守り - Japanese for protection / safeguard) is an embedded Go library that loads application configuration and secrets from heterogeneous sources - environment, files, AWS Secrets Manager, Vault, GCP, Azure, Kubernetes, Consul, and more - into typed, validated structs, and keeps them reconciled at runtime. When a source value changes, mamori detects it, re-validates the whole configuration, and - only if the new snapshot is valid - atomically swaps it in and hands your application a diff-aware callback so it can react (rotate a DB pool, rebuild a client) without restarting.

Think: External Secrets Operator's provider model, one layer down - as a library inside your process instead of an operator inside your cluster.

Why

The primitives exist, but nobody composed them. runtimevar watches one variable but has no struct composition or validation. Viper/koanf do multi-source config but treat secrets and rotation as afterthoughts. The AWS caching client and Vault's LifetimeWatcher refresh one provider each, in silos. So every production Go service hand-rolls a ConfigManager with a ticker, a mutex, and a prayer. mamori is that glue, done once, with a provider ecosystem and a conformance kit.

Install

go get github.com/xavidop/mamori

env: and file:// work out of the box. Cloud providers are separate modules so the core has zero cloud-SDK dependencies:

go get github.com/xavidop/mamori/providers/aws     # aws-sm://  aws-ps://
go get github.com/xavidop/mamori/providers/vault   # vault://
go get github.com/xavidop/mamori/providers/k8s     # k8s-secret://  k8s-cm://
# ... gcp, azure, consul, doppler, onepassword, sops

Quick start

type Config struct {
    // A secret string from AWS Secrets Manager (redacted in logs by default)
    DBPassword secret.String `source:"aws-sm://prod/db#password"`

    // Plain config from the environment, with a default and validation
    LogLevel   string        `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"`
    Workers    int           `source:"env:WORKERS"   default:"4"    validate:"gte=1,lte=256"`

    // A file-backed value, hot-reloaded via fsnotify
    TLSCert    []byte        `source:"file:///etc/tls/tls.crt"`

    // A nested struct decoded from one JSON secret
    Redis      RedisConfig   `source:"aws-sm://prod/redis" flatten:"json"`
}

// One-shot load
cfg, err := mamori.Load[Config](ctx)

// Or: watch and reconcile at runtime
w, err := mamori.Watch[Config](ctx,
    mamori.OnChange(func(ev mamori.Change[Config]) {
        if ev.Changed("DBPassword") {
            pool.Rotate(ev.New.DBPassword.Reveal())
        }
    }),
    mamori.OnError(func(err error) { metrics.Inc("config_error") }),
)
defer w.Close()

cfg := w.Get() // lock-free snapshot; always the last *valid* config

What makes it different

  • Typed & tag-driven - one struct, multiple sources, generics API (Load[T] / Watch[T]).
  • Reconciled at runtime - native watch where the backend supports it (Kubernetes informers, Consul blocking queries, fsnotify), polling with jitter everywhere else, and lease-aware pre-expiry refresh for Vault.
  • Atomic & validated - an update that fails validation is rejected; Get() keeps returning the last good config. Config never enters a broken state mid-flight.
  • Coalesced - bursts of field changes within a debounce window produce a single Change event.
  • Secret hygiene by default - secret.String / secret.Bytes redact themselves in String(), fmt, MarshalJSON, and slog. Only the explicit, greppable Reveal() exposes the value. A shipped go vet analyzer (reconcilevet) flags sensitive refs assigned to plain string fields.
  • Pluggable - providers register with the database/sql pattern; a providertest conformance kit guarantees they all behave identically.

Providers

Module Schemes Watch
core (built-in) env: · dotenv:// · file:// · exec: (opt-in) fsnotify (file/dotenv) · poll (env/exec)
providers/aws aws-sm:// · aws-ps:// poll
providers/gcp gcp-sm:// poll
providers/azure azure-kv:// poll
providers/vault vault:// lease-aware poll (NotAfter)
providers/k8s k8s-secret:// · k8s-cm:// native (watch API)
providers/consul consul:// native (blocking queries)
providers/doppler doppler:// poll
providers/onepassword op:// poll
providers/sops sops:// fsnotify
providers/postgres postgres:// native (LISTEN/NOTIFY)
providers/mysql mysql:// poll
providers/sqlite sqlite:// fsnotify
providers/mongodb mongodb:// native (change streams)
providers/dynamodb dynamodb:// poll
providers/redis redis:// native (keyspace notifications)
providers/etcd etcd:// native (watch API)
providers/firestore firestore:// native (snapshot listeners)
providers/firebase-rc firebase-rc:// poll
providers/firebase-rtdb firebase-rtdb:// native (streaming)
providers/s3 s3:// poll (ETag)
providers/gcs gcs:// poll (generation)
providers/azblob azblob:// poll (ETag)
providers/cosmos cosmos:// poll (ETag)
providers/launchdarkly launchdarkly:// native (streaming)
providers/unleash unleash:// poll
providers/flagsmith flagsmith:// poll
providers/configcat configcat:// poll
providers/split split:// poll
providers/growthbook growthbook:// poll
providers/flipt flipt:// poll
providers/goff goff:// (GO Feature Flag) poll

Every provider that passes the providertest conformance kit earns a badge. See each module's README for auth and ref grammar.

Middleware

Providers compose because they share one interface:

mamori.WithProvider(
    middleware.Cache(5*time.Minute,
        middleware.Audit(logger,
            middleware.Failover(primary, replica))))

Cache, Audit, Failover, RateLimit, and Prefix (multi-tenant namespace rewriting) ship in middleware/.

Documentation

Project layout

This is a multi-module monorepo. The core (github.com/xavidop/mamori) depends only on go-playground/validator, go-viper/mapstructure, and fsnotify. Each provider is its own module with its own release cadence, so a cloud SDK never leaks into your build unless you use that provider.

Contributing

Contributions welcome - new providers especially. See CONTRIBUTING.md and the Write a provider guide. A provider that passes providertest and the conformance kit gets listed here.

License

MIT © Xavier Portilla Edo

Documentation

Overview

Package mamori loads application configuration and secrets from heterogeneous sources (environment, files, cloud secret managers, Vault, Kubernetes, ...) into typed, validated Go structs, and keeps them reconciled at runtime.

When a source value changes, mamori detects it, re-validates the whole configuration, and - only if the new snapshot is valid - atomically swaps it in and notifies the application with a diff-aware callback so it can react (rotate a database pool, rebuild a client, ...) without restarting.

Loading

Define a struct whose fields carry a `source` tag describing where each value comes from, then call Load: 

type Config struct {
    DBPassword secret.String `source:"aws-sm://prod/db#password"`
    LogLevel   string        `source:"env:LOG_LEVEL" default:"info"`
    Workers    int           `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"`
}

cfg, err := mamori.Load[Config](ctx)

Watching

Watch performs an initial fail-fast load and then keeps the configuration reconciled, delivering validated, diff-aware updates: 

w, err := mamori.Watch[Config](ctx,
    mamori.OnChange(func(ev mamori.Change[Config]) {
        if ev.Changed("DBPassword") {
            pool.Rotate(ev.New.DBPassword.Reveal())
        }
    }),
)
defer w.Close()
cfg := w.Get() // lock-free snapshot; always the last valid config

Providers

Sources are pluggable via the Provider SPI and registered with Register using the database/sql pattern. The core module has zero cloud-SDK dependencies; each cloud provider ships as its own module under providers/.

Index

Constants

This section is empty.

Variables

View Source 
var ErrNotFound = errors.New("mamori: not found")

ErrNotFound is the sentinel error providers wrap (or return) when a referenced value does not exist. Consumers and mamori itself test for it with errors.Is(err, ErrNotFound); mamori applies defaults / optional handling only for not-found, never for other errors.

Functions

func Load 

func Load[T any](ctx context.Context, opts ...Option) (T, error)

Load resolves all refs of T once, applies defaults, validates, and returns the typed config. It fails fast: on any resolve or validation error it returns a non-nil error and the zero value of T; partial config is never returned.

func Register 

func Register(p Provider)

Register makes a provider available under its Scheme(), following the database/sql driver pattern. It is normally called from a provider package's init function. Register panics if a provider for the same scheme is already registered (fail fast at process init) or if p is nil.

func RegisteredSchemes 

func RegisteredSchemes() []string

RegisteredSchemes returns the sorted list of registered provider schemes. It is useful for diagnostics and for the docs/provider gallery.

func SelectKey 

func SelectKey(data []byte, key string) ([]byte, error)

SelectKey extracts a single field named key from a JSON object payload, for refs of the form scheme://path#key. If key is empty, data is returned unchanged. String values are returned unquoted; objects, arrays, numbers, and booleans are returned as their JSON encoding. If the key is absent, SelectKey returns an error wrapping ErrNotFound.

Provider authors should call SelectKey with ref.Key after fetching the raw payload, so that `#key` selection behaves identically across all providers.

func VersionHash 

func VersionHash(b []byte) string

VersionHash produces a stable version string from bytes. Provider authors can use it to synthesize a Value.Version when the backend has no native revision identifier, giving mamori cheap change detection without a byte comparison.

Types

type BatchProvider 

type BatchProvider interface {
	Provider
	// ResolveBatch resolves all refs, returning a map keyed by each input Ref's
	// Raw string (equivalently Ref.String()). A ref that is not found should be
	// omitted from the map (mamori applies the default) rather than causing the
	// whole batch to fail. Ref itself is not comparable (it holds url.Values), so
	// the raw tag string is used as the key.
	ResolveBatch(ctx context.Context, refs []Ref) (map[string]Value, error)
}

BatchProvider is an optional interface for providers that can resolve many refs in one backend call (SM BatchGetSecretValue, one file read for many keys). mamori uses it automatically when available.

type Change 

type Change[T any] struct {
	Old    T
	New    T
	Fields []FieldChange
}

Change is delivered to OnChange when a reconciled, re-validated update is applied. Old and New are immutable full snapshots; Fields lists what changed.

func (Change[T]) Changed 

func (c Change[T]) Changed(path string) bool

Changed reports whether the field at the given dotted path is among the changed fields in this event.

type Clock 

type Clock interface {
	// Now returns the current time.
	Now() time.Time
	// NewTimer returns a timer that fires once after d.
	NewTimer(d time.Duration) *Timer
	// NewTicker returns a ticker that fires every d.
	NewTicker(d time.Duration) *Ticker
	// After is shorthand for NewTimer(d).C.
	After(d time.Duration) <-chan time.Time
}

Clock abstracts time so the reconciler can be tested deterministically. All time-dependent engine code uses a Clock rather than the time package directly. The default is the system clock; pass a different one with WithClock.

func SystemClock 

func SystemClock() Clock

SystemClock returns the default, real-time Clock.

type FakeClock 

type FakeClock struct {
	// contains filtered or unexported fields
}

FakeClock is a manually-driven Clock for deterministic tests. Advance moves time forward, firing any timers/tickers whose deadline is reached.

func NewFakeClock 

func NewFakeClock(start time.Time) *FakeClock

NewFakeClock returns a FakeClock started at the given time (or a fixed epoch if start is the zero value).

func (*FakeClock) Advance 

func (c *FakeClock) Advance(d time.Duration)

Advance moves the fake clock forward by d, firing any timers and tickers whose deadlines fall within the interval, in chronological order.

func (*FakeClock) After 

func (c *FakeClock) After(d time.Duration) <-chan time.Time

After registers a timer and returns its channel.

func (*FakeClock) NewTicker 

func (c *FakeClock) NewTicker(d time.Duration) *Ticker

NewTicker registers a periodic ticker.

func (*FakeClock) NewTimer 

func (c *FakeClock) NewTimer(d time.Duration) *Timer

NewTimer registers a one-shot timer.

func (*FakeClock) Now 

func (c *FakeClock) Now() time.Time

Now returns the fake current time.

type FieldChange 

type FieldChange struct {
	Path       string // dotted field path, e.g. "Redis.Password"
	OldVersion string
	NewVersion string
}

FieldChange records one field that changed between two snapshots.

type Meter 

type Meter interface {
	// RecordResolve reports a provider resolve: its scheme, duration, and error
	// (nil on success).
	RecordResolve(scheme string, dur time.Duration, err error)
	// RecordRefresh reports that a watched value changed and was reconciled.
	RecordRefresh(scheme string)
	// RecordWatchError reports a provider watch-channel error.
	RecordWatchError(scheme string)
}

Meter is the minimal metrics sink mamori emits to. It is deliberately tiny so the core module takes no OpenTelemetry dependency; the x/otel module provides an adapter that implements this interface on top of an OTel meter. Pass one with WithMeter. All methods must be safe for concurrent use.

type Option 

type Option func(*options)

Option configures Load and Watch.

func OnChange 

func OnChange[T any](fn func(Change[T])) Option

OnChange installs the callback invoked (on a single, serialized goroutine) for each applied update. It is typed to the same T passed to Watch.

func OnError 

func OnError(fn func(error)) Option

OnError installs a callback for runtime resolve/validation/stale errors.

func WithBackoff 

func WithBackoff(base, max time.Duration) Option

WithBackoff configures per-ref exponential backoff on resolve failure.

func WithClock 

func WithClock(c Clock) Option

WithClock overrides the clock, primarily for deterministic tests.

func WithDebounce 

func WithDebounce(d time.Duration) Option

WithDebounce sets the coalescing window for change events (default 500ms). A per-field `?debounce=` ref option overrides this for that field.

func WithDecodeHook added in v1.1.0 

func WithDecodeHook(h mapstructure.DecodeHookFunc) Option

WithDecodeHook adds a mapstructure decode hook applied when decoding a flatten:"json|yaml|env" payload into a nested struct. Hooks run after the built-in secret/duration hook, in the order registered, so you can convert custom field types (a time.Time layout, a net.IP, an enum, ...).

func WithExecProvider 

func WithExecProvider() Option

WithExecProvider enables the exec: provider for this Load or Watch call only. It is not registered globally; you must opt in explicitly.

func WithJitter 

func WithJitter(f float64) Option

WithJitter sets the poll jitter fraction (0..1); a value of 0.2 randomizes each interval by ±20% to avoid thundering herds.

func WithMeter 

func WithMeter(m Meter) Option

WithMeter installs a metrics sink (see the x/otel module for an OTel adapter).

func WithPollInterval 

func WithPollInterval(d time.Duration) Option

WithPollInterval sets the fallback poll interval for non-watchable providers.

func WithProvider 

func WithProvider(p Provider) Option

WithProvider registers a provider for this call only, taking precedence over the global registry for its scheme.

func WithQueueDepth 

func WithQueueDepth(n int) Option

WithQueueDepth bounds the OnChange dispatch queue; when full, the oldest event is dropped (default 16).

func WithStale 

func WithStale(maxAge time.Duration) Option

WithStale escalates staleness to a hard error: if a ref cannot be refreshed for longer than maxAge, OnError receives a *StaleError.

func WithTracer 

func WithTracer(t Tracer) Option

WithTracer installs a tracing sink (see the x/otel module for an OTel adapter).

func WithValidator 

func WithValidator(v Validator) Option

WithValidator overrides the default (go-playground/validator) validator.

type Provider 

type Provider interface {
	// Scheme returns the URL scheme this provider handles, e.g. "aws-sm".
	Scheme() string
	// Resolve fetches the current Value for ref. It must return an error that
	// satisfies errors.Is(err, ErrNotFound) when the referenced value does not
	// exist, so defaults and optional fields can be applied.
	Resolve(ctx context.Context, ref Ref) (Value, error)
}

Provider resolves refs of a single scheme into Values. It is the minimum a source must implement. Providers should be safe for concurrent use.

type ProviderError 

type ProviderError struct {
	Scheme string
	Ref    string
	Err    error
}

ProviderError wraps an error from a specific provider resolve, tagging it with the scheme and ref for diagnostics and metrics. It is delivered to OnError for runtime resolve failures.

func (*ProviderError) Error 

func (e *ProviderError) Error() string

func (*ProviderError) Unwrap 

func (e *ProviderError) Unwrap() error

Unwrap allows errors.Is/As to reach the underlying error (e.g. ErrNotFound).

type Ref 

type Ref struct {
	// Scheme selects the provider, e.g. "aws-sm", "vault", "env", "file".
	Scheme string
	// Path is the provider-specific location of the value, e.g. "prod/db",
	// "kv/data/api", "/etc/tls/tls.crt", or "LOG_LEVEL".
	Path string
	// Key selects a single field from a structured payload (the URL fragment,
	// i.e. the part after '#'). It is empty when no key is requested.
	Key string
	// Opts holds provider-specific options parsed from the query string, plus a
	// small set of core-recognized options (debounce, optional, version).
	Opts url.Values
	// Raw is the original, unparsed tag value, retained for error messages.
	Raw string
}

Ref is a parsed reference to a value in a provider. It is produced from the `source` struct tag by ParseRef. The general grammar is: 

<scheme>://<path>[#<key>][?<opt>=<v>&...]

Opaque schemes such as env: and exec: take everything after the colon as the Path (no "//" authority section): 

env:LOG_LEVEL
exec:echo hello

func ParseRef 

func ParseRef(tag string) (Ref, error)

ParseRef parses a `source` tag value into a Ref. It returns an error for an empty tag or a tag without a scheme.

The grammar is scheme-agnostic and, per the mamori spec, places the optional #key fragment BEFORE the optional ?opts query (the reverse of a standard URL). Parsing is therefore done by hand rather than via net/url: 

scheme://path[#key][?opts]   (hierarchical: aws-sm, vault, file, ...)
scheme:path[#key][?opts]     (opaque: env, exec)

func (Ref) Opt 

func (r Ref) Opt(name string) string

Opt returns the first value for the named option, or "" if unset.

func (Ref) String 

func (r Ref) String() string

String renders the Ref back into its canonical tag form. Query options are omitted if empty. It is primarily useful for diagnostics.

type StaleError 

type StaleError struct {
	Ref string
	Err error
}

StaleError is returned/delivered when a value has exceeded the configured WithStale max age without a successful refresh.

func (*StaleError) Error 

func (e *StaleError) Error() string

func (*StaleError) Unwrap 

func (e *StaleError) Unwrap() error

type Ticker 

type Ticker struct {
	C <-chan time.Time
	// contains filtered or unexported fields
}

Ticker is a Clock-provided periodic ticker. Its channel C receives ticks.

func (*Ticker) Stop 

func (t *Ticker) Stop()

Stop halts the ticker. It does not close C.

type Timer 

type Timer struct {
	C <-chan time.Time
	// contains filtered or unexported fields
}

Timer is a Clock-provided one-shot timer. Its channel C receives the time it fires. Stop prevents a not-yet-fired timer from firing.

func (*Timer) Stop 

func (t *Timer) Stop() bool

Stop halts the timer, returning true if it had not yet fired.

type Tracer 

type Tracer interface {
	// StartResolve begins a span for a resolve and returns a derived context plus
	// a finish function to be called with the resolve error (nil on success).
	StartResolve(ctx context.Context, scheme, ref string) (context.Context, func(err error))
}

Tracer is the minimal tracing sink mamori emits to (see Meter for the no-dep rationale). Pass one with WithTracer.

type Update 

type Update struct {
	// Value is the new value. Valid only when Err is nil.
	Value Value
	// Err carries a transient watch error. The channel remains open; mamori
	// surfaces the error and keeps the last-good value.
	Err error
}

Update is a single event delivered on a WatchableProvider's channel.

type ValidationError 

type ValidationError struct {
	Err error
}

ValidationError wraps a validation failure. When an updated snapshot fails validation the update is rejected atomically and this error is delivered to OnError; Get continues to return the last valid config.

func (*ValidationError) Error 

func (e *ValidationError) Error() string

func (*ValidationError) Unwrap 

func (e *ValidationError) Unwrap() error

type Validator 

type Validator interface {
	// Validate returns nil if v is valid, or a non-nil error describing the
	// failure (which mamori wraps in *ValidationError).
	Validate(v any) error
}

Validator validates a fully-decoded config value. It is invoked on the initial load and on every reconciled update. The default implementation wraps go-playground/validator, reading `validate` struct tags. Supply an alternative with WithValidator.

type Value 

type Value struct {
	// Bytes is the raw resolved payload.
	Bytes []byte
	// Version is a provider-supplied revision identifier (Secrets Manager
	// VersionId, Vault version, a file mtime+size hash, ...). It enables cheap
	// change detection: mamori treats a changed Version as a changed value
	// without comparing bytes. If empty, mamori falls back to byte comparison.
	Version string
	// Sensitive marks the value as secret, driving redaction downstream. It is
	// set by secret-bearing providers and by the decode layer for
	// secret.String / secret.Bytes fields.
	Sensitive bool
	// NotAfter, when non-zero, is the time at which this value is known to expire
	// (e.g. a Vault lease). mamori schedules a refresh before this instant rather
	// than waiting for the next poll tick.
	NotAfter time.Time
	// Metadata carries optional provider-specific annotations. It must never
	// contain the secret payload.
	Metadata map[string]string
}

Value is what a Provider returns for a resolved Ref. It carries the raw bytes plus the metadata mamori needs for change detection, redaction, and lease-aware refresh.

type WatchableProvider 

type WatchableProvider interface {
	Provider
	// Watch returns a channel of Updates for ref. The channel is closed when the
	// watch ends (including on ctx cancellation). Transient errors are delivered
	// as Updates with a non-nil Err; channel closure signals termination.
	Watch(ctx context.Context, ref Ref) (<-chan Update, error)
}

WatchableProvider is an optional interface for providers with native change notification (Vault leases, Kubernetes informers, Consul blocking queries). Providers without native watch support are wrapped by mamori in a polling adapter - provider authors must never fake a Watch with an internal ticker.

type Watcher 

type Watcher[T any] struct {
	// contains filtered or unexported fields
}

Watcher holds the reconciled configuration of type T and manages the background watch goroutines. Obtain one from Watch and always Close it.

func Watch 

func Watch[T any](ctx context.Context, opts ...Option) (*Watcher[T], error)

Watch performs an initial, fail-fast Load of T and then keeps it reconciled at runtime, delivering validated, diff-aware updates to OnChange. It returns after the initial configuration is resolved (OnChange fires only on subsequent changes).

func (*Watcher[T]) Close 

func (w *Watcher[T]) Close() error

Close cancels all provider watches, drains the callback queue, and returns.

func (*Watcher[T]) Get 

func (w *Watcher[T]) Get() T

Get returns the current configuration snapshot. It is lock-free and always returns the last valid configuration (never a partially-updated or validation-failing one).

Source Files

View all Source files

Directories

Path Synopsis
examples
basic command
Command basic demonstrates mamori: loading typed config from the environment and a file, then watching for changes and reacting without a restart.
 Command basic demonstrates mamori: loading typed config from the environment and a file, then watching for changes and reacting without a restart.
Package middleware provides composable Provider decorators: Cache, Audit, Failover, RateLimit, and Prefix.
 Package middleware provides composable Provider decorators: Cache, Audit, Failover, RateLimit, and Prefix.
Package providertest is the mamori provider conformance kit.
 Package providertest is the mamori provider conformance kit.
Package secret provides wrapper types for sensitive configuration values that redact themselves everywhere a value is normally rendered - String, fmt, encoding/json, and log/slog - so a secret cannot leak through a stray log line or error message.
 Package secret provides wrapper types for sensitive configuration values that redact themselves everywhere a value is normally rendered - String, fmt, encoding/json, and log/slog - so a secret cannot leak through a stray log line or error message.
tools
reconcilevet module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.