etcd

README

caddy-etcd

This is a clustering plugin for Caddy that will store Caddy-managed certificates and any other assets in etcd rather than the filesystem. It implements a virtual filesystem on top of etcd storage in order to allow multiple instances of caddy to share configuration information and TLS certificates without having to share a filesystem. You must have already set up your own etcd cluster for this plugin to work.

Configuration

Caddy clustering plugins are enabled and configured through environment variables. The table below lists the available options, but to enable this plugin you must first set CADDY_CLUSTERING="etcd".

Environment	Function	Default
CADDY_CLUSTERING_ETCD_SERVERS	A comma or semicolon separated list of etcd servers for caddy to connect to. The servers must be specified as a full URL including scheme, e.g.: https://127.0.0.1:2379.	http://127.0.0.1:2379
CADDY_CLUSTERING_ETCD_PREFIX	A prefix that will be added to each Caddy-managed file to separate it from other keys you have in your etcd cluster	/caddy
CADDY_CLUSTERING_ETCD_TIMEOUT	The timeout for locks on Caddy resources. In the event of a failure or network issue, the lock on a particular resource will timeout after this value, allowing another operation to try to write that value. Must be expressed as a Go-style duration, like 5m, 30s.	5m
CADDY_CLUSTERING_ETCD_CADDYFILE	The plugin includes a Caddyfile loader that will read Caddyfile configuration from <KeyPrefix>/caddyfile. If this file exists in etcd, it will be used as the Caddyfile configuration. This environment variable allows you to bootstrap a clustered configuration from an existing Caddyfile on disk. When set, it will load this file and store it in etcd for other cluster members to use. If both etcd contains Caddyfile configuration and a Caddyfile exists on disk, the configuration in etcd will be used.
CADDY_CLUSTERING_ETCD_CADDYFILE_LOADER	To disable loading/storing Caddyfile configuration in etcd, set this to "disable"	enable

Building Caddy with this Plugin

This plugin requires caddy to be built with go modules. It cannot be built by the current build server on caddyserver.com because it does not support modules. This project uses mage to build the caddy binary. To build, first download mage:

go get -u github.com/magefile/mage

You must have the following binaries available on your system to run the build:

go >= 1.11
sed

Then build by running

mage build

See the magefile.go for other customizations, such as including other plugins in your custom build.

Testing

This project uses go modules and must be tested with the -mod vendor flag in order to use vendored dependencies that have been modified to work with caddy.

go test -v -cover -race -mod vendor

Roadmap

• etcd mutual TLS support
• incremental Caddyfile configuration - allow new keys inserted under <KeyPrefix>/caddyfile/ to modify the running Caddy configuration (e.g., add a new site by writing to etcd under /caddy/caddyfile/mysite with just the site's configuration)
• make plugin buildable on caddyserver.com

Documentation

Overview

package etcd adds clustering capabilities to caddy to store all Caddy-managed certificates and caddyfile to etcd

Index

• func FilterExactPrefix(prefix string, cut string) func(client.Node) bool
• func FilterPrefix(prefix string, cut string) func(client.Node) bool
• func FilterRemoveDirectories() func(client.Node) bool
• func IsFailedChecksumError(e error) bool
• func IsNotExistError(e error) bool
• func Load(servertype string) (caddy.Input, error)
• func NewCluster() (certmagic.Storage, error)
• type Cluster
  • func (c Cluster) Delete(key string) error
  • func (c Cluster) Exists(key string) bool
  • func (c Cluster) List(prefix string, recursive bool) ([]string, error)
  • func (c Cluster) Load(key string) ([]byte, error)
  • func (c Cluster) Lock(key string) error
  • func (c Cluster) Stat(key string) (certmagic.KeyInfo, error)
  • func (c Cluster) Store(key string, value []byte) error
  • func (c Cluster) Unlock(key string) error
• type ClusterConfig
  • func NewClusterConfig(opts ...ConfigOption) (*ClusterConfig, error)
• type ConfigOption
  • func ConfigOptsFromEnvironment() (opts []ConfigOption)
  • func WithCaddyFile(s string) ConfigOption
  • func WithDisableCaddyfileLoad(s string) ConfigOption
  • func WithPrefix(s string) ConfigOption
  • func WithServers(s string) ConfigOption
  • func WithTimeout(s string) ConfigOption
• type FailedChecksum
  • func (e FailedChecksum) Error() string
• type Lock
• type Metadata
  • func NewMetadata(key string, data []byte) Metadata
• type NotExist
  • func (e NotExist) Error() string
• type Service
  • func NewService(c *ClusterConfig) Service

Functions

func FilterExactPrefix

func FilterExactPrefix(prefix string, cut string) func(client.Node) bool

FilterExactPrefix returns only terminal nodes (files) with the exact path prefix. For example, for two files `/one/two/file.txt` and `/one/two/three/file.txt` only the first would be returned for a prefix of `/one/two`.

func FilterPrefix

func FilterPrefix(prefix string, cut string) func(client.Node) bool

FilterPrefix is a filter to be used with List to return only paths that start with prefix. If specified, cut will first trim a leading path off the string before comparison.

func FilterRemoveDirectories

func FilterRemoveDirectories() func(client.Node) bool

FilterRemoveDirectories is a filter to be used with List to remove all directories (i.e., nodes that contain only other nodes and no value)

func IsFailedChecksumError

func IsFailedChecksumError(e error) bool

IsFailedChecksumError checks to see if error is of type FailedChecksum

func IsNotExistError

func IsNotExistError(e error) bool

IsNotExistError checks to see if error is of type NotExist

func Load

func Load(servertype string) (caddy.Input, error)

Load satisfies the caddy.Input interface to return the contents of a Caddyfile in the following order: (1) any caddy files that are loaded in etcd at key: /<keyprefix>/caddyfile (2) a caddyfile that is set using CADDY_CLUSTERING_ETCD_CADDYFILE (3) other configured caddyfile loaders, including the default loader

func NewCluster

func NewCluster() (certmagic.Storage, error)

NewCluster returns a cluster plugin that reads from the environment to configure itself

Types

type Cluster

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster implements the certmagic.Storage interface as a cluster plugin

func (Cluster) Delete

func (c Cluster) Delete(key string) error

Delete fulfills the certmagic.Storage interface and deletes the node located at key along with any associated metadata.

func (Cluster) Exists

func (c Cluster) Exists(key string) bool

Exists fulfills the certmagic.Storage interface. Exists returns true only if the there is a terminal node that exists which represents a file in a filesystem.

func (Cluster) List

func (c Cluster) List(prefix string, recursive bool) ([]string, error)

List fulfills the certmagic.Storage interface and lists all nodes that exist under path `prefix`. For recursive queries, it returns all keys located at subdirectories of `prefix`. Otherwise, it only returns terminal nodes that represent files present at exactly the patch `prefix`.

func (Cluster) Load

func (c Cluster) Load(key string) ([]byte, error)

Load fulfills the certmagic.Storage interface. Each load operation retrieves the value associated at the file node and checks it against the hash stored in the metadata node associated with the file. If the node does not exist, a `NotExist` error is returned. Data corruption found via a hash mismatch returns a `FailedChecksum` error.

func (Cluster) Lock

func (c Cluster) Lock(key string) error

Lock fulfills the certmagic.Storage Locker interface. Each etcd operation gets a lock scoped to the key it is updating with a customizable timeout. Locks that persist past the timeout are assumed to be abandoned.

func (Cluster) Stat

func (c Cluster) Stat(key string) (certmagic.KeyInfo, error)

Stat fulfills the certmagic.Storage interface and returns metadata about existing nodes. When the key represents a file in the filesystem, it returns metadata about the file. For directories, it traverses all children to determine directory size and modified time.

func (Cluster) Store

func (c Cluster) Store(key string, value []byte) error

Store fulfills the certmagic.Storage interface. Each storage operation results in two nodes added to etcd. A node is created for the value of the file being stored. A matching metadata node is created to keep details of creation time, SHA1 hash, and size of the node. Failures to create both nodes in a single transaction make a best effort at restoring the pre-transaction state.

func (Cluster) Unlock

func (c Cluster) Unlock(key string) error

Unlock fulfills the certmagic.Storage Locker interface. Locks are cleared on a per path basis.

type ClusterConfig

type ClusterConfig struct {
	KeyPrefix        string
	ServerIP         []string
	LockTimeout      time.Duration
	CaddyFile        []byte
	CaddyFilePath    string
	DisableCaddyLoad bool
}

ClusterConfig maintains configuration information for cluster resources such as etcd server instances

func NewClusterConfig

func NewClusterConfig(opts ...ConfigOption) (*ClusterConfig, error)

NewClusterConfig returns a new configuration with options passed as functional options

type ConfigOption

type ConfigOption func(c *ClusterConfig) error

ConfigOption represents a functional option for ClusterConfig

func ConfigOptsFromEnvironment

func ConfigOptsFromEnvironment() (opts []ConfigOption)

ConfigOptsFromEnvironment reads environment variables and returns options that can be applied via NewClusterConfig

func WithCaddyFile

func WithCaddyFile(s string) ConfigOption

WithCaddyFile sets the path to the bootstrap Caddyfile to load on initial start if configuration information is not already present in etcd. The first cluster instance will load this file and store it in etcd. Subsequent members of the cluster will prioritize configuration from etcd even if this file is present. This function will not error even if the Caddyfile is not present. If a caddyfile cannot be read from etcd, from this file, or from the default loader, caddy will start with an empty default configuration.

func WithDisableCaddyfileLoad

func WithDisableCaddyfileLoad(s string) ConfigOption

WithDisableCaddyfileLoad will skip all attempts at loading the caddyfile from etcd and force caddy to fall back to other enabled caddyfile loader plugins or the default loader

func WithPrefix

func WithPrefix(s string) ConfigOption

WithPrefix sets the etcd namespace for caddy data. Default is `/caddy`. Prefixes are normalized to use `/` as a path separator.

func WithServers

func WithServers(s string) ConfigOption

WithServers sets the etcd server endpoints. Multiple endpoints are assumed to be separated by a comma, and consist of a full URL, including scheme and port (i.e., http://127.0.0.1:2379) The default config uses port 2379 on localhost.

func WithTimeout

func WithTimeout(s string) ConfigOption

WithTimeout sets the time locks should be considered abandoned. Locks that exist longer than this setting will be overwritten by the next client that acquires the lock. The default is 5 minutes. This option takes standard Go duration formats such as 5m, 1h, etc.

type FailedChecksum

type FailedChecksum struct {
	Key string
}

FailedChecksum error is returned when the data retured by Load does not match the SHA1 checksum stored in its metadata node

func (FailedChecksum) Error

func (e FailedChecksum) Error() string

type Lock

type Lock struct {
	Token    string
	Obtained string
	Key      string
}

Lock is a clients lock on updating keys. When the same client requests multiple locks, the lock is extended. Assumes that one client does not try to set the same key from different go routines. In this case, a race condition exists and last write wins.

type Metadata

type Metadata struct {
	Path      string
	Size      int
	Timestamp time.Time
	Hash      [20]byte
	IsDir     bool
}

Metadata stores information about a particular node that represents a file in etcd

func NewMetadata

func NewMetadata(key string, data []byte) Metadata

NewMetadata returns a metadata information given a path and a file to be stored at the path. Typically, one metadata node is stored for each file node in etcd.

type NotExist

type NotExist struct {
	Key string
}

NotExist is returned when a key lookup fails when calling Load or Metadata

func (NotExist) Error

func (e NotExist) Error() string

type Service

type Service interface {
	Store(key string, value []byte) error
	Load(key string) ([]byte, error)
	Delete(key string) error
	Metadata(key string) (*Metadata, error)
	Lock(key string) error
	Unlock(key string) error
	List(path string, filters ...func(client.Node) bool) ([]string, error)
	// contains filtered or unexported methods
}

Service is a low level interface that stores and loads values in Etcd

func NewService

func NewService(c *ClusterConfig) Service

NewService returns a new low level service to store and load values in etcd. The service is designed to store values with associated metadata in a format that allows it to fulfill with the Certmagic storage interface, effectively implementing simple filesystem semantics on top of etcd key/value storage. Locks are acquired before writes to etcd and the library will make its best attempt at rolling back transactions that fail. Concurrent writes are blocking with exponential backoff up to a reasonable time limit. Errors are logged, but do not guarantee that the system will return to a coherent pre-transaction state in the presence of significant etcd failures or prolonged unavailability.