README
¶
config-etcd
etcd as a config layer — with real compare-and-swap and a real change feed
Part of the phpboyscout Go toolkit — small, framework-free Go modules extracted from go-tool-base. Documented with the parent module at config.go.phpboyscout.uk
A prefix of an etcd v3 cluster becomes an ordinary
config layer: full precedence,
per-key merge, provenance, safe hot-reload and writes.
import (
"gitlab.com/phpboyscout/go/config"
configetcd "gitlab.com/phpboyscout/go/config-etcd"
clientv3 "go.etcd.io/etcd/client/v3"
)
client, err := clientv3.New(clientv3.Config{Endpoints: []string{"localhost:2379"}})
if err != nil {
return err
}
store, err := config.NewStore(ctx,
config.WithFiles(fsys, "/etc/app.yaml"), // defaults on disk
config.WithBackend(configetcd.FromClient(client, "app/")), // etcd outranks them
)
Keys split on / into the tree, with the prefix stripped: app/server/port
becomes server.port. Explain("server.port") will name etcd as the source.
What makes it different from the other backends
Most of this family polls and cannot compare-and-swap. etcd does both properly, which is why it is read and write from v0.1.0.
| Compare-and-swap | Real. Every write is guarded by the ModRevision its key held at load, so a change that landed since your code decided to write is refused rather than overwritten. |
| Atomic across keys | A batch is one etcd transaction: every operation applies, or none does. AtomicMultiKey is true, which almost nothing else in the family can claim. |
| Native watch | A real subscription, not a poll. An explicit poll interval is accepted and ignored — there is nothing to pace. |
| No gap on start | The watch replays from the revision captured at load, so a change landing between the read and the watch attaching is delivered rather than missed. |
It is not a secrets store, and there is no option to pretend otherwise
Sensitive is false and there is deliberately no WithSensitive().
Kubernetes keeps Secret objects in etcd, so people demonstrably store secrets
there. But an etcd prefix has no audit trail of who read a value, no leasing or
rotation, and no separation between the credential that reads configuration and
the one that reads secrets — which is exactly what
config-vault and the other
Phase B adapters provide. A flag that made etcd look like a secrets store would
make the weaker choice the easier one.
If you have secrets in etcd and cannot move them, wrap this backend in
config.Filtered (which forwards Capabilities) or contribute those keys
through a backend that does declare itself sensitive.
Leased keys can vanish
etcd keys can carry a lease and disappear when it expires. They are read like any other key: the value contributes while it exists, and once the lease lapses the key is simply absent at the next load.
That is not a special case — it is what a layer losing a key already means, and merge, provenance and shadowing all handle it. The consequence is worth stating though: a configuration layer over a leased prefix can lose keys without anyone writing anything. If that is surprising, use an unleased prefix.
Structured values
etcd stores bytes, so a value is a scalar string by default and the View's typed accessors coerce it. Where a value is itself a JSON or YAML document, hand in a codec:
configetcd.FromClient(client, "app/", configetcd.WithValueCodec(configjson.Codec{}))
A value that decodes to a mapping becomes a subtree; anything the codec rejects stays a scalar, so a prefix mixing flat keys and blobs reads correctly. This module takes no codec dependency of its own.
Batch size
No cap is imposed here. etcd bounds both transaction operation count
(--max-txn-ops, 128 by default) and request size (--max-request-bytes), both
are server settings a client cannot know, and both are reported
distinguishably — so an oversized batch fails with etcd's own error naming which
limit it hit:
configetcd: committing 200 operation(s): etcdserver: too many operations in txn request
A guessed client-side cap would be wrong in both directions: too low on a tuned cluster, too high on a default one.
What it costs
| Modules added | 25 — 15 for the etcd client, 10 for the config graph |
| Requires | config v0.13.0+ and etcd v3 |
The etcd client is the same weight as Consul's, and less than half of the
Kubernetes client-go — the measurement that decided this adapter was worth
building and that config-k8s was not. An allowlist test pins it, so a
transitive upgrade cannot widen it quietly.
Install
go get gitlab.com/phpboyscout/go/config-etcd
Documentation
- How to read and write etcd — the task guide
- How dynamic backends work — what every remote backend shares
- Spec 0001 — the decisions, and what was rejected
- API reference
Licence
MIT — see LICENSE.
Documentation
¶
Overview ¶
Package configetcd contributes configuration from an etcd v3 cluster as a first-class config layer: precedence, per-key merge, provenance, safe hot-reload and compare-and-swap writes, exactly like a file layer.
A prefix scopes the backend to one namespace of the key space; the keys beneath it, split on "/", become the layer's nested tree. etcd stores bytes, so a value is a scalar string by default and the View's typed accessors coerce it (GetInt("server.port") parses "8080"). A value that is itself a JSON or YAML document is decoded into a subtree when a codec is supplied with WithValueCodec — see the config-etcd spec, D4.
The cluster is reached through the narrow KV interface, which Wrap adapts from a configured *clientv3.Client. Injecting the client keeps every endpoint, credential and TLS decision with the consumer and lets the whole unit suite run against a fake, needing no etcd.
etcd is the family's second native-watch backend after Consul, and the only one whose compare-and-swap covers a whole batch: a transaction is atomic across every key it touches, so config.Capabilities.AtomicMultiKey is true.
Index ¶
- Constants
- Variables
- func FromClient(client *clientv3.Client, prefix string, opts ...Option) config.Backend
- func FromConfig(cfg clientv3.Config, prefix string, opts ...Option) (config.Backend, error)
- func New(kv KV, prefix string, opts ...Option) config.Backend
- type Cmp
- type KV
- type Op
- type Option
- type Pair
Constants ¶
const SourceKind = config.SourceKind("etcd")
SourceKind identifies an etcd layer in provenance. A value read from here reports as coming from "etcd:<prefix>" rather than from a file.
Variables ¶
var ErrNoEndpoints = errors.New("configetcd: no etcd endpoints configured; set Endpoints on the config")
ErrNoEndpoints reports a config that names no etcd endpoint.
etcd has no ambient convention to fall back on — see FromConfig — so an empty Endpoints slice cannot be filled in from the environment and would fail at the first request instead of here.
Functions ¶
func FromClient ¶
FromClient is the common path: New over a Wrap-ped etcd client, so a consumer writes configetcd.FromClient(client, "app/") without touching the narrow interface.
func FromConfig ¶ added in v0.2.0
FromConfig builds the etcd client from a config the caller assembled — the endpoints, dial timeout, credentials and TLS — and returns a backend over it.
clientv3.New is non-blocking: it validates the config and prepares the gRPC connection without dialling, so a failure here is a malformed configuration rather than an unreachable cluster. Reaching etcd is deferred to Load, which has a context to bound it.
There is no ambient rung here, deliberately ¶
Every other remote adapter in this family offers a zero-conf Default() over its SDK's own ambient convention — LoadDefaultConfig for AWS, DefaultAzureCredential for Azure, Application Default Credentials for GCP, DefaultConfig for Vault and Consul. etcd has none: clientv3 has no DefaultConfig, and its only environment variable is ETCD_CLIENT_DEBUG, a debug flag rather than an endpoint or a credential.
A Default() here would therefore have to *invent* the convention, and adopting a provider's documented default is a different act from inventing one. It would also mean a configuration store quietly connecting to localhost:2379 and serving a tool's settings from an unintended source — a failure that presents as wrong values rather than as a connection error.
The question is deferred rather than closed: config spec 0012 R1 revisits it once every other adapter has been upgraded and the full set of shapes is visible, so it is settled once rather than one adapter at a time.
Types ¶
type Cmp ¶
Cmp is one compare-and-swap guard: the key must still be at the ModRevision it held at Load. Revision zero requires the key be absent — that is how etcd spells "does not exist" — so a create cannot silently overwrite a key someone else added in the meantime.
type KV ¶
type KV interface {
// Get returns every pair under prefix and the store revision identifying
// that read. The revision is what the write and watch paths build on, so it
// must be the header revision etcd reported for this read, not a fresh
// fetch: the watch replays from it to close the gap between reading and
// attaching, and Verify compares against it.
Get(ctx context.Context, prefix string) (pairs []Pair, revision int64, err error)
// Txn applies ops as one transaction, guarded by cmps. ok is false, with a
// nil error, when a comparison failed — a key moved since Load. etcd applies
// the whole set or none of it.
Txn(ctx context.Context, cmps []Cmp, ops []Op) (ok bool, err error)
// Watch calls onChange for every change under prefix, replaying from
// fromRevision so a change that landed between Get and the watch attaching
// is still delivered. The returned stop ends the watch.
Watch(ctx context.Context, prefix string, fromRevision int64, onChange func()) (stop func(), err error)
}
KV is the slice of etcd this adapter uses, behind an interface it owns so a fake drives the unit suite and the real client is adapted by Wrap.
type Op ¶
Op is one write in a transaction: a put or a delete of a full etcd key. The guard lives in the accompanying Cmp rather than on the op, matching etcd's own If/Then split.
type Option ¶
type Option func(*backend)
Option configures a backend.
func WithValueCodec ¶
WithValueCodec decodes each etcd value through codec: a value that decodes to a mapping becomes a subtree at its key's path, and a value the codec rejects — a bare scalar, or bytes that are not a document — stays a scalar string, so a prefix mixing flat keys and object blobs reads correctly. Omitted, every value is a scalar string.
codec is any config.Codec — the interface the sibling format adapters implement — so a JSON-blob store is read with configjson.Codec{} and a YAML-blob store with the core's, and this module takes no codec dependency of its own.
Writes always target flat etcd keys; writing into a value that a codec decoded from a blob is not supported and lands as a sibling flat key.
type Pair ¶
type Pair struct {
// Key is the full etcd key, prefix included.
Key string
// Value is the raw bytes etcd holds.
Value []byte
// ModRevision is the store revision at which this key was last modified.
// It is the per-key version compare-and-swap guards on.
ModRevision int64
}
Pair is one etcd key/value, with the revision the write path compares against.
Source Files