caddydicer

README

caddy-dicer

An ingress for Dicer: a Caddy plugin that sends requests to Dicer instances, chosen by their labels.

The instances can carry their own routing, so the Caddyfile does not change as they come and go:

:80 {
	reverse_proxy {
		dynamic dicer
	}
}

$ dicer run -d --label caddy.host=shop.example.com --label caddy.port=80 nginx:1.27

Every running instance labelled caddy.host=shop.example.com gets a share of the requests for shop.example.com, on its port 80. Start another with the same labels, and it gets requests as soon as it is running. Stop one, and it stops getting them.

A route can also choose its instances with a label selector:

app.example.com {
	reverse_proxy {
		dynamic dicer app=web 8080
	}
}

How it works

The plugin adds a dynamic upstream source for Caddy's reverse_proxy, http.reverse_proxy.upstreams.dicer. It connects to the Dicer daemon, lists its instances, and lists them again whenever the daemon reports an event about an instance. Caddy matches each request against the instances the plugin already knows about, so a request never waits on the daemon.

An instance is routed to when it is:

• running, with an address on its network
• healthy, if it has a health check. One whose check has no verdict yet, such as in its start period, is not routed to yet.
• selected by the route for the request (see below)

On a route with a selector, a request goes to the instances whose labels match it. On a route without one, it goes to the instances whose caddy.host label lists the request's host. Either way, an instance with a caddy.host label only gets requests for the hosts it lists.

When a route has no such instance, Caddy sends its requests to an instance that is waking up or can be woken, if there is one (see Scaling to zero). Otherwise it answers 503 Service Unavailable. Caddy logs each instance that joins or leaves the rotation.

If the daemon goes away, its instances keep running, and Caddy keeps routing to the instances it last knew of until the daemon is back.

Labels

Label	Meaning
caddy.host	The hosts the instance serves, separated by commas, such as shop.example.com,www.shop.example.com. Hosts are matched case-insensitively, ignoring any port.
caddy.port	The guest port the instance serves on. It overrides the route's port.

Give an instance its labels when you create it, with --label, or with labels in a compose file:

$ dicer run -d --name web-1 --label app=web --label tier=frontend nginx:1.27

Anyone who can create or label an instance on the daemon can claim a host this way. On a route with no selector, an instance labelled caddy.host=shop.example.com joins that host's rotation, and the permission module allows a certificate for it. That suits a daemon with a single operator. Where others can run instances too, route with a selector instead.

Where requests go

An instance's port is its caddy.port label, or else the route's port.

• If the instance publishes that port on the host, such as with -p 8001:80, requests go to the published port. If no port is named, they go to the only TCP port the instance publishes. A connection to a published port wakes an instance on standby.
• Otherwise, they go to the instance's own address on its Dicer network. Caddy must be able to reach that network. The simplest way is to run Caddy on the daemon's host.
• If no port is named and the instance publishes no TCP port, or more than one, it gets no requests.

A published port is reached at:

• the address it is published on, if one is given, such as -p 192.0.2.10:8001:80
• for a port published on every address, the published_port_host option, if it is set
• otherwise, the host part of the address option, if the daemon is reached over TCP at an address that is not loopback
• otherwise, the gateway address of the instance's network, which is one of the host's own addresses. This is the case for a daemon reached through its socket, or at a loopback address.

Set published_port_host when the address option does not name the daemon's host: when the API is reached through a proxy or load balancer on another machine, or by a name that resolves to a loopback address.

It is never reached at localhost, because the daemon does not forward published ports from loopback. For the same reason, a port published only on a loopback address is not used, and requests go to the instance's own address instead.

Scaling to zero

An instance given --standby-after is put on standby once it has been idle that long, and is woken again by a connection to one of its published ports. The plugin keeps track of such instances while they are on standby, so a route whose instances have all gone on standby still answers. While none of a route's instances is routable, Caddy sends its requests to one of them instead:

1. one whose health check has no verdict yet, such as one just woken
2. otherwise, the first instance on standby, in name order, that publishes the port its requests go to

When Caddy connects to the published port of an instance on standby, the daemon wakes it. This usually takes about a second. The daemon then passes the connection on to the guest, so the first request is slow rather than refused. Requests then go to the woken instance and, once it is found healthy, to every routable instance as usual.

Only one instance is woken at a time. The route's other instances stay on standby until a request reaches them, which happens only while none of the route's instances is routable. If the route needs more, start them with dicer start.

An instance that should scale to zero needs a published port. A connection to its own address does not wake it, so an instance on standby with no published port is not routed to. Each instance needs a host port of its own, and a fork publishes none unless given one with -p. An instance put on standby with dicer standby and no --standby-after is not routed to either, because a connection does not wake it.

HTTPS for the hosts in labels

A site with no hostnames, such as https://, cannot have its certificates obtained in advance. Caddy gets each one on demand instead, the first time its host is requested. The plugin's permission module allows a certificate only for a host that an instance's caddy.host label lists, whatever state the instance is in:

{
	on_demand_tls {
		permission dicer
	}
}

https:// {
	tls {
		on_demand
	}
	reverse_proxy {
		dynamic dicer
	}
}

permission dicer takes the same daemon options as dynamic dicer. Give the two the same values, and they share a connection to the daemon. Each host must still resolve to Caddy's address, so that the certificate authority can reach Caddy to validate it.

Requirements

• A Dicer daemon
• Caddy 2.11 or later, built with this plugin
• Access to the daemon's API (see Access to the daemon)

Install

Build Caddy with the plugin with xcaddy:

$ xcaddy build --with github.com/kazysgurskas/caddy-dicer
$ ./caddy list-modules | grep dicer
http.reverse_proxy.upstreams.dicer
tls.permission.dicer

Configuration

Caddyfile

reverse_proxy {
	dynamic dicer [<selector> [<port>]] {
		address             <target>
		token               <token>
		selector            <selector>
		port                <port>
		published_port_host <host>
		tls {
			ca          <file>
			server_name <name>
		}
	}
}

• selector chooses the instances, by their labels. Without one, requests are routed by the instances' caddy.host labels. Quote a selector that has spaces in it.
• port is the guest port the instances serve on. An instance's caddy.port label overrides it. Without either, requests go to the only port an instance publishes.
• address is the daemon's API, as a gRPC target. Default: the local daemon's socket, unix:///run/dicer/dicer.sock. A socket may also be unix-abstract:NAME. A daemon's TCP listener is host:port.
• token is the token to call a TCP listener with, which it requires. The socket takes none. Keep it out of the Caddyfile with a placeholder: {file./etc/caddy/dicer-token} or {env.DICER_TOKEN}. Surrounding whitespace, such as a file's last newline, is dropped. The token carries the daemon's certificate fingerprint, which is how Caddy checks the daemon.
• tls is only for a daemon whose certificate an authority signed, rather than the one it makes for itself. ca is that authority, and server_name is the name the certificate must carry, if it is not the host in address.
• published_port_host is the host at which Caddy reaches the ports that instances publish on every address. By default it is worked out from address, as Where requests go describes.

address takes placeholders too, such as {env.DICER_ADDRESS}. Each setting may be given once. The ca file is read each time Caddy loads its configuration, so reload Caddy after replacing it.

The rest of reverse_proxy works as usual: a load balancing policy, retries and passive health checks apply to the instances the plugin finds.

shop.example.com {
	reverse_proxy {
		dynamic dicer "app=shop,tier in (frontend,edge),!canary" 8080
		lb_policy least_conn
		lb_retries 2
	}
}

canary.shop.example.com {
	reverse_proxy {
		dynamic dicer app=shop,canary 8080
	}
}

The permission module takes only the daemon settings:

on_demand_tls {
	permission dicer {
		address             <target>
		token               <token>
		published_port_host <host>
		tls {
			ca          <file>
			server_name <name>
		}
	}
}

JSON

{
  "handler": "reverse_proxy",
  "dynamic_upstreams": {
    "source": "dicer",
    "selector": "app=web",
    "port": 8080,
    "address": "unix:///run/dicer/dicer.sock"
  }
}

With tls, it is an object of ca and server_name. In JSON, the permission module goes in the TLS app's automation.on_demand.permission, as {"module": "dicer", "address": ...}.

Selectors

A selector is written as Kubernetes writes label selectors:

Selector	Matches an instance whose
app=web, app==web	label app is web
tier!=canary	label tier is not canary, or which has no tier label
tier in (frontend,edge)	label tier is one of these
tier notin (canary)	label tier is none of these, or which has no tier label
public	has a label public, whatever its value
!internal	has no label internal

Separate requirements with commas: an instance must meet all of them. A key or a value is a run of characters other than whitespace, commas, parentheses, = and !.

Access to the daemon

Caddy only needs to list instances and networks, and to follow instance events. Give it a token with just those scopes, through the daemon's TCP listener. It lists networks only to learn their gateway addresses. When the listener is on a loopback address, Caddy reaches ports published on every address at those gateways. A token for a listener on any other address does not need networks:read.

On the daemon's host, turn the listener on for the loopback address only, in /etc/dicerd/config.yaml:

server:
  listen: 127.0.0.1:7443

then restart the daemon and make the token:

$ sudo systemctl restart dicerd
$ sudo dicer token create caddy --scopes instances:read,events:read,networks:read |
    sudo install -m 600 -o caddy -g caddy /dev/stdin /etc/caddy/dicer-token

and point the plugin at it:

dynamic dicer app=web 8080 {
	address 127.0.0.1:7443
	token   {file./etc/caddy/dicer-token}
}

The socket works too, with no token, for a Caddy whose user is in the dicer group. Prefer the token: the socket gives whatever can use it as much as root on the host, and the token only lets Caddy read.

Development

The style of the code is set out in CLAUDE.md.

$ make test     # the tests, with the race detector
$ make lint     # golangci-lint
$ make fmt      # format the code
$ make build    # Caddy with the plugin, into bin/caddy
$ make help     # every target

The tests run against a fake daemon that serves Dicer's API on a Unix socket, so they need no Dicer.

End-to-end tests

The end-to-end tests run on a host with Dicer, reached over SSH: another machine, or a virtual machine on this one. They copy this repository there, build Caddy with the plugin, boot five instances of a small web server, run Caddy in front of them, and send requests through it from here:

• three instances behind one site, which must share its requests evenly
• one instance each behind two more sites, which must get all of theirs
• a site whose selector matches nothing, which must answer 503
• an instance that is stopped, paused, started, or deleted, which must leave or join its site's rotation without Caddy being reloaded
• a site whose only instance is on standby, which a request must wake through its published port, 18090
• a site with no selector, which must route by the instances' caddy.host labels, and answer 503 for a host that no instance lists

$ make test-e2e CADDY_DICER_E2E_HOST=lima@192.168.1.27 CADDY_DICER_E2E_SSH_PORT=2222

The host needs dicerd running, Go, git, rsync and curl, and sudo without a password for the SSH user. If the daemon serves its TCP listener, Caddy reaches it with a read-only token the tests make. Otherwise Caddy runs as root and uses the socket. Each run labels what it makes with its own ID and removes it all at the end.

Variable	Default
CADDY_DICER_E2E_HOST		Required: user@host
CADDY_DICER_E2E_SSH_PORT	22
CADDY_DICER_E2E_KERNEL_ARGS	console=ttyAMA0 reboot=k panic=1 on arm64	Given to every instance
CADDY_DICER_E2E_RESTART_DAEMON		Also restart dicerd mid-test. Its instances keep running.
CADDY_DICER_E2E_KEEP		Leave the instances and Caddy running afterwards

On an arm64 host, the instances are given console=ttyAMA0, because Dicer's Cloud Hypervisor default, console=ttyS0, names no device there and its guests do not boot.

License

MIT. See LICENSE.

Documentation

Overview

Package caddydicer routes Caddy's reverse proxy to Dicer instances chosen by their labels: an ingress for the virtual machines a Dicer daemon runs.

It adds two modules. http.reverse_proxy.upstreams.dicer is a dynamic upstream source. A route names a label selector and a port, and its requests go to that port on every instance whose labels match:

app.example.com {
	reverse_proxy {
		dynamic dicer app=web 8080
	}
}

Alternatively, instances can carry their routing in the caddy.host and caddy.port labels. A route with no selector then sends each request to the instances that serve its host:

:80 {
	reverse_proxy {
		dynamic dicer
	}
}

The second module, tls.permission.dicer, allows on-demand TLS certificates only for the hosts those labels list.

Build Caddy with the plugin using xcaddy:

xcaddy build --with github.com/kazysgurskas/caddy-dicer

See Upstreams for what is routed to, PermissionByLabel for certificates, and the selector package for the syntax of a selector.

Index

• type Daemon
• type PermissionByLabel
  • func (PermissionByLabel) CaddyModule() caddy.ModuleInfo
  • func (p *PermissionByLabel) CertificateAllowed(_ context.Context, name string) error
  • func (p *PermissionByLabel) Cleanup() error
  • func (p *PermissionByLabel) Provision(ctx caddy.Context) error
  • func (p *PermissionByLabel) UnmarshalCaddyfile(d *caddyfile.Dispenser) error
  • func (p *PermissionByLabel) Validate() error
• type TLSConfig
• type Upstreams
  • func (Upstreams) CaddyModule() caddy.ModuleInfo
  • func (u *Upstreams) Cleanup() error
  • func (u *Upstreams) GetUpstreams(r *http.Request) ([]*reverseproxy.Upstream, error)
  • func (u *Upstreams) Provision(ctx caddy.Context) error
  • func (u *Upstreams) UnmarshalCaddyfile(d *caddyfile.Dispenser) error
  • func (u *Upstreams) Validate() error

Types

type Daemon

type Daemon struct {
	// The daemon's address, as a gRPC target: unix:///path/to/socket, or
	// host:port for its TCP listener. Default: the local daemon's socket,
	// unix:///run/dicer/dicer.sock. Placeholders are replaced, as in token.
	Address string `json:"address,omitempty"`

	// The token for the daemon's TCP listener, which requires one. The
	// socket needs none. Placeholders are replaced, so that the token can
	// be kept out of the configuration: {env.DICER_TOKEN}, or
	// {file./etc/caddy/dicer-token}. Surrounding whitespace, such as a
	// file's last newline, is dropped.
	Token string `json:"token,omitempty"`

	// How to verify a daemon whose certificate a certificate authority
	// signed. A daemon's self-signed certificate needs none of this: the
	// token carries its fingerprint.
	TLS *TLSConfig `json:"tls,omitempty"`

	// The host at which Caddy reaches the ports that instances publish on
	// every address. Default: the host in address, for a TCP listener at
	// an address that is not loopback, or otherwise the gateway address of
	// the instance's network. Set it when address reaches the daemon through a
	// proxy on another machine, or by a name that resolves to a loopback
	// address. Placeholders are replaced, as in token.
	PublishedPortHost string `json:"published_port_host,omitempty"`
}

Daemon identifies the Dicer daemon a module follows and how to reach it. Upstreams and PermissionByLabel embed it.

type PermissionByLabel

type PermissionByLabel struct {
	Daemon
	// contains filtered or unexported fields
}

PermissionByLabel is a permission module for on-demand TLS. It allows a certificate only for a host listed in a Dicer instance's caddy.host label, whatever state the instance is in, so a site that routes by those labels does not need to name its hosts:

{
	on_demand_tls {
		permission dicer
	}
}

https:// {
	tls {
		on_demand
	}
	reverse_proxy {
		dynamic dicer
	}
}

It follows the daemon as Upstreams does, and shares the connection with every Upstreams that names the same daemon.

func (PermissionByLabel) CaddyModule

func (PermissionByLabel) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (*PermissionByLabel) CertificateAllowed

func (p *PermissionByLabel) CertificateAllowed(_ context.Context, name string) error

CertificateAllowed returns nil if an instance's caddy.host label lists name, and an error wrapping caddytls.ErrPermissionDenied otherwise.

func (*PermissionByLabel) Cleanup

func (p *PermissionByLabel) Cleanup() error

Cleanup releases the connection to the daemon, which closes once no configuration uses it.

func (*PermissionByLabel) Provision

func (p *PermissionByLabel) Provision(ctx caddy.Context) error

Provision connects to the daemon, unless another module already has, and waits briefly for its instances to be listed.

func (*PermissionByLabel) UnmarshalCaddyfile

func (p *PermissionByLabel) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

UnmarshalCaddyfile sets up the module from Caddyfile tokens:

permission dicer {
	address             <target>
	token               <token>
	published_port_host <host>
	tls {
		ca          <file>
		server_name <name>
	}
}

func (*PermissionByLabel) Validate

func (p *PermissionByLabel) Validate() error

Validate checks the configuration.

type TLSConfig

type TLSConfig struct {
	// The PEM file of the certificate authority that signed the daemon's
	// certificate. Default: the system's trusted authorities. It is read
	// each time the configuration is loaded, so a reload picks up a change.
	CA string `json:"ca,omitempty"`

	// The name the daemon's certificate must carry. Default: the host in
	// the daemon's address.
	ServerName string `json:"server_name,omitempty"`
}

TLSConfig is how to verify a daemon whose certificate a certificate authority signed. A daemon's self-signed certificate needs none of it: the token carries its fingerprint.

type Upstreams

type Upstreams struct {
	Daemon

	// Which instances to route to, as a label selector such as
	// "app=web,tier in (frontend,edge)". Without one, a request goes to the
	// instances whose caddy.host label lists its host.
	Selector string `json:"selector,omitempty"`

	// The guest port the instances serve on. An instance's caddy.port label
	// overrides it. Without either, requests go to the only port an
	// instance publishes, and an instance that publishes none, or several,
	// gets none.
	Port int `json:"port,omitempty"`
	// contains filtered or unexported fields
}

Upstreams routes to Dicer instances chosen by their labels. It is a dynamic upstream source for reverse_proxy.

A route chooses its instances with a selector, such as "app=web". Instances can also declare which requests they serve, so the Caddyfile does not have to. An instance labelled caddy.host=shop.example.com only gets requests for that host, and on a route with no selector it gets every request for it. Its caddy.port label overrides the route's port.

If an instance publishes its guest port on the host, requests go to the published port. If neither the route nor the instance's labels name a port, they go to the only port the instance publishes. A connection to a published port wakes an instance on standby. An instance that publishes no port gets requests at its own address on its network, which Caddy must be able to reach: run it on the daemon's host.

It routes to instances that are running, have an address, and, if they have a health check, have been found healthy. When a route has no such instance, it routes to one instance with standby_after instead. It prefers one whose health check has no verdict yet, such as one just woken. Otherwise it picks the first instance on standby, in name order, that publishes the port its requests go to. A route whose instances have all gone on standby therefore wakes one at a time.

It only needs to list instances and networks, and to follow instance events. Over the daemon's TCP listener, give it a token with just those scopes:

dicer token create caddy --scopes instances:read,events:read,networks:read

It lists networks only to learn their gateway addresses, where it reaches ports published on every address when the listener is on a loopback address. A token for a listener on any other address does not need networks:read.

The instances are kept up to date from the daemon's events, so a request never waits on the daemon. While the daemon cannot be reached, the instances it last reported are routed to.

func (Upstreams) CaddyModule

func (Upstreams) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (*Upstreams) Cleanup

func (u *Upstreams) Cleanup() error

Cleanup releases the connection to the daemon, which closes once no configuration uses it.

func (*Upstreams) GetUpstreams

func (u *Upstreams) GetUpstreams(r *http.Request) ([]*reverseproxy.Upstream, error)

GetUpstreams returns an upstream for each routable instance that serves r, in name order. If there is none, it returns one for the route's fallback instance, if any. It returns no upstreams, and no error, when no instance serves r.

func (*Upstreams) Provision

func (u *Upstreams) Provision(ctx caddy.Context) error

Provision connects to the daemon, unless another module already has, and waits briefly for its instances to be listed.

func (*Upstreams) UnmarshalCaddyfile

func (u *Upstreams) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

UnmarshalCaddyfile sets up the module from Caddyfile tokens:

dynamic dicer [<selector> [<port>]] {
	address             <target>
	token               <token>
	selector            <selector>
	port                <port>
	published_port_host <host>
	tls {
		ca          <file>
		server_name <name>
	}
}

A selector with spaces, such as one with "in (a, b)", is quoted. With no selector, requests are routed by the instances' caddy.host labels.

func (*Upstreams) Validate

func (u *Upstreams) Validate() error

Validate checks the configuration.