etcdstorage

README

caddy-storage-etcd

A Caddy storage module that keeps certificates, keys and the other assets Caddy manages in etcd, so that several Caddy instances share them, along with the locks that stop them from all renewing the same certificate at once.

xcaddy build --with github.com/alien43/caddy-storage-etcd

Configuration

{
	storage etcd {
		endpoints etcd1:2379 etcd2:2379 etcd3:2379
		prefix    /caddy
	}
}

The endpoints can also go on the first line: storage etcd etcd1:2379 etcd2:2379.

JSON:

"storage": {
	"module": "etcd",
	"endpoints": ["etcd1:2379", "etcd2:2379", "etcd3:2379"],
	"prefix": "/caddy"
}

Option	Default
endpoints	(required)	etcd members, as host:port or URLs. The client fails over between them.
prefix	/caddy	The etcd key prefix this module owns.

Caddy can't start or reload while etcd is unreachable. It needs its storage then, as with any storage module. Running instances keep serving.

How it works

• Values live at <prefix>/data/<key>, where the key is certmagic's slash-separated path. A directory is any key prefix ending at a slash; it exists while something lies below it. Each value starts with a 9-byte header: a format byte and the time it was stored, because etcd keeps no modification time and Caddy asks for one.
• Locks are etcd mutexes under <prefix>/locks/, each on a lease of its own. Waiters are served first come, first served, and Lock returns when the context ends. The client keeps a held lock's lease alive. If the holder dies, the lease expires after 15 seconds and the lock goes with it, so there's no stale lock to clean up.

Moving existing certificates in

Caddy can copy its storage between modules:

caddy storage export --config old.json --output certs.tar   # config with the old storage
caddy storage import --config new.json --input certs.tar    # config with storage etcd

The archive holds private keys; delete it afterwards.

Limits

• No TLS or authentication to etcd yet: use it on a network you trust, or put a firewall in front of etcd's client port.
• A value can't exceed etcd's request size limit (1.5 MiB by default). Certificates and keys are a few kilobytes.
• If etcd is unreachable for longer than the lock lease, a held lock lapses while its holder still works. The worst case is two instances ordering the same certificate.

Testing

The test needs an etcd it may write to. It works under a prefix of its own and removes it afterwards:

podman run --rm -d -p 127.0.0.1:52379:2379 quay.io/coreos/etcd:v3.6.14 etcd \
	--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://127.0.0.1:2379
ETCDSTORAGE_TEST_ENDPOINTS=127.0.0.1:52379 go test -race ./...

License

MIT

Documentation

Overview

Package etcdstorage is a Caddy storage module that keeps certificates, keys and the other assets Caddy manages in etcd, so that several Caddy instances can share them.

Under the configured prefix, values live below <prefix>/data/ and locks below <prefix>/locks/. Keys are certmagic's slash-separated paths. A "directory" is any key prefix that ends at a slash, and it exists while some key lies below it.

Locks are etcd mutexes held on leases. Waiters are served first come, first served; a held lock is kept alive by the client; and if its holder dies, the lease expires and takes the lock with it.

Index

• type Storage
  • func (*Storage) CaddyModule() caddy.ModuleInfo
  • func (s *Storage) CertMagicStorage() (certmagic.Storage, error)
  • func (s *Storage) Cleanup() error
  • func (s *Storage) Delete(ctx context.Context, key string) error
  • func (s *Storage) Exists(ctx context.Context, key string) bool
  • func (s *Storage) List(ctx context.Context, dir string, recursive bool) ([]string, error)
  • func (s *Storage) Load(ctx context.Context, key string) ([]byte, error)
  • func (s *Storage) Lock(ctx context.Context, name string) error
  • func (s *Storage) Provision(ctx caddy.Context) error
  • func (s *Storage) Stat(ctx context.Context, key string) (certmagic.KeyInfo, error)
  • func (s *Storage) Store(ctx context.Context, key string, value []byte) error
  • func (s *Storage) String() string
  • func (s *Storage) Unlock(_ context.Context, name string) error
  • func (s *Storage) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

Types

type Storage

type Storage struct {
	// Endpoints are the etcd members to talk to, as host:port or URLs.
	Endpoints []string `json:"endpoints,omitempty"`

	// Prefix is the etcd key prefix this module owns. Default: /caddy.
	Prefix string `json:"prefix,omitempty"`
	// contains filtered or unexported fields
}

Storage is the caddy.storage.etcd module.

func (*Storage) CaddyModule

func (*Storage) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (*Storage) CertMagicStorage

func (s *Storage) CertMagicStorage() (certmagic.Storage, error)

CertMagicStorage returns the module itself: it is the storage.

func (*Storage) Cleanup

func (s *Storage) Cleanup() error

Cleanup releases the locks this instance still holds and disconnects. Caddy calls it when the config that loaded the module is unloaded.

func (*Storage) Delete

func (s *Storage) Delete(ctx context.Context, key string) error

Delete removes key, or, if key is a directory, everything below it.

func (*Storage) Exists

func (s *Storage) Exists(ctx context.Context, key string) bool

Exists reports whether key is a stored key or a directory.

func (*Storage) List

func (s *Storage) List(ctx context.Context, dir string, recursive bool) ([]string, error)

List returns the entries directly below dir, keys and directories, or, if recursive, every key below it.

func (*Storage) Load

func (s *Storage) Load(ctx context.Context, key string) ([]byte, error)

Load returns the value at key.

func (*Storage) Lock

func (s *Storage) Lock(ctx context.Context, name string) error

Lock takes the lock for name, waiting as long as ctx allows.

func (*Storage) Provision

func (s *Storage) Provision(ctx caddy.Context) error

Provision connects to etcd, and fails if it can't be reached.

func (*Storage) Stat

func (s *Storage) Stat(ctx context.Context, key string) (certmagic.KeyInfo, error)

Stat describes key, which may be a stored key or a directory.

func (*Storage) Store

func (s *Storage) Store(ctx context.Context, key string, value []byte) error

Store puts value at key, replacing any value already there.

func (*Storage) String

func (s *Storage) String() string

String identifies the storage in logs.

func (*Storage) Unlock

func (s *Storage) Unlock(_ context.Context, name string) error

Unlock releases the lock for name, taken by Lock on this instance.

func (*Storage) UnmarshalCaddyfile

func (s *Storage) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

UnmarshalCaddyfile sets up the storage from Caddyfile tokens:

storage etcd [<endpoints...>] {
	endpoints <endpoints...>
	prefix    <prefix>
}