homelab-horizon

README

Homelab Horizon

A self-contained homelab management tool for WireGuard VPN, split-horizon DNS, reverse proxy, and service monitoring. Single binary, runs on Ubuntu/Debian.

Website: homelab-horizon.iodesystems.com · the official, original project — accept no clones.

The Problem

Running a homelab with external access means juggling multiple systems that don't talk to each other:

• SSL Certificate Sprawl: Managing 12+ individual certificates, each with their own renewal schedule
• Internal SSL Headaches: HTTPS doesn't work from inside your network because certs are tied to external IPs, so you're stuck with HTTP internally or browser warnings
• Unnecessary Public Exposure: OAuth callbacks and other endpoints need valid SSL, forcing you to expose internal-only services to the internet just to get certificates
• Manual DNS Management: Updating Route53 or other DNS providers by hand every time your IP changes or you add a service
• Broken Internal Resolution: Your domains work from the internet but timeout when you're on your own network (the classic split-horizon DNS problem)
• WireGuard Friction: Every new device needs a config file, QR code, and manual peer setup on the server
• Scattered Configuration: HAProxy configs, DNS records, WireGuard peers, and SSL certs all managed separately with no unified view

The Solution

Homelab Horizon consolidates all of this into a single web UI:

• Consolidated Certs: Wildcard certs only cover one level (*.example.com won't cover app.sub.example.com), so we make it easy to add extra SANs like *.vpn.example.com - all visible and editable in the UI, and you can inspect exactly what each cert covers. No more mystery broken SSL.
• HTTPS Everywhere: Same SSL cert works internally - no more HTTP fallbacks or certificate warnings on your LAN
• Automatic DNS Sync: Add a service, DNS records update automatically (Route53, Cloudflare, Name.com, and more)
• Split-Horizon Built-in: Services resolve to internal IPs on your network/VPN, external IPs from the internet
• Self-Service VPN: Generate invite links - users scan a QR code and they're connected
• Unified Dashboard: See all your services, their health status, DNS records, and SSL certificates in one place

Screenshots

Dashboard

Services

Service Detail

Deleting a service

Deleting doesn't silently strand the state a service didn't own — the zone SubZone that gave its domain HTTPS, and the record published at your DNS provider. Both are listed, and the delete won't proceed until you say whether to retract them.

DNS

Zones hz publishes to, and per zone every record live at the provider labelled by who owns it — derived from a service, declared on the zone, observed (not hz's, never rewritten), or tombstoned (deletion pending).

Domains

Every domain hz knows, with internal DNS, external DNS, proxy and HTTPS state side by side — plus the SSL coverage gaps and the zone records it publishes.

Observability

Declare hosts and exporter jobs, then hand Prometheus a generated scrape config.

Ports

Reserved ports per host, and the denylist hz ports next skips when allocating.

Port Map

Settings

VPN MFA captive portal

What an MFA-jailed peer sees when it asks for any other service — the request is redirected to the portal, and nothing else on the network answers until a second factor is accepted. See VPN MFA.

These two come from bin/e2e, not make screenshots: the portal identifies the caller by WireGuard source IP, so the only honest way to photograph a jailed peer's view is to be one. The hermetic container has no WireGuard.

The VPN and Checks pages are captured too (docs/screenshots/{vpn,checks}.png) but aren't shown here — the hermetic container has no live WireGuard peers and can't reach the documentation IPs it checks, so both render as empty or all-red and would misrepresent the pages. dns-zone.png (the per-zone record table) is likewise captured but not shown: the mock provider serialises the zone's SOA value as a Python repr, which looks like an hz bug and isn't one.

Regenerate with make screenshots — boots a hermetic Docker container (daemons off, RFC 5737 documentation IPs, no outbound network) and captures these with Playwright. See docs/take-screenshots.mjs.

DNS_FIXTURE=1 ./bin/screenshots additionally boots a mock Route53 (motoserver/moto) seeded with a zone, so the DNS pages capture real records instead of an empty state. Opt-in: it needs the aws CLI and pulls a ~400MB image. The DNS shots above were taken with it.

Features

• Auto-Heal: Detects and installs missing dependencies on a fresh Ubuntu system
• WireGuard VPN Management: Create clients, generate QR codes, manage peers
• VPN MFA: Optional per-peer TOTP or passkey (codes recommended for full-tunnel peers — why). Peers without a verified session are jailed to a captive portal until they authenticate
• Split-Horizon DNS: Internal DNS via dnsmasq, external DNS via Route53, Name.com, Cloudflare, and more
• Reverse Proxy: HAProxy with automatic Let's Encrypt wildcard SSL certificates
• Static Sites: Serve a folder of files as a service — hz hosts it directly, HAProxy routes to it with the same auto SSL/DNS
• Service Monitoring: Health checks with ntfy push notifications
• Outside-In Checks (hz-probe): A small agent on a host outside the homelab probes your public DNS, HTTPS and latency. hz polls it, so hz stays unreachable from the internet and nothing about the private network leaves the box
• Prometheus Discovery: Declare hosts and exporter jobs; hz serves a generated scrape config and probes every target
• Port Allocation: Server-authoritative reservations and denylist, so hz ports next never hands out a port something already uses
• Operator CLI (hz): Service CRUD, HTTPS per domain, hosts, exporters, sync — with a preview of what a mutation actually changes
• Honest Deletes: Deleting a service reports the SubZone and DNS record it would strand, and makes you choose
• Unsynced-Change Tracking: hz pending and a badge in the UI show what's edited but not yet published
• Self-Service Onboarding: Users redeem invite tokens to get VPN configs
• IP Banning: Per-service IP bans with timeout support
• Rolling Deploys: Blue-green deployment support with hz-client CLI
• Multi-Instance HA: Run two boxes for automatic config replication, cert failover, and round-robin DNS
• MCP Server: Machine-readable API for AI-assisted management

Quick Start

Kick the tires (Docker)

cd examples/simple
./setup.sh            # generate WG keys and config
docker compose up -d  # start HZ

Open http://localhost:8090 and log in with the admin token:

docker exec hz cat /etc/homelab-horizon/config.json.token

Install from a release

ARCH=linux-amd64   # or linux-arm64, linux-armv7
curl -fSL -o hz.tar.gz \
  https://github.com/IodeSystems/homelab-horizon/releases/latest/download/homelab-horizon-$ARCH.tar.gz
tar -xzf hz.tar.gz
sudo ./homelab-horizon-$ARCH        # installs itself, then serves

One file: the admin UI is compiled into the binary, so there is nothing else to place. Override with STATIC_DIR or ui_dir only if you want hz to serve a frontend of your own.

Bare metal install (from source)

# Build (requires Go 1.25+ and Node.js)
make

# Run as root (WireGuard, dnsmasq, HAProxy, iptables, ports 80/443, systemd)
sudo ./homelab-horizon

On first run, the binary:

1. Copies itself to /usr/local/bin/
2. Installs a systemd service
3. Writes an admin token to /etc/homelab-horizon/config.json.token

With auto_heal: true, it detects and installs missing packages (wireguard-tools, iproute2, haproxy, dnsmasq) via apt-get.

Pass config via environment for Docker or backup/restore workflows:

sudo HZ_CONFIG='{"listen_addr":":8080","auto_heal":true,...}' ./homelab-horizon

Growing with you

HZ scales from a single box to a redundant pair without reconfiguration. When you're ready, add a second instance:

• Same-subnet (two boxes in one rack or VPC): low complexity, shared VPN range, LAN replication
• Cross-site (two locations or availability zones): WireGuard tunnel between peers, disjoint VPN ranges, full geo-redundancy

cd examples/ha-same-subnet && ./setup.sh && docker compose up -d
cd examples/ha-site-to-site && ./setup.sh && docker compose up -d

See the High Availability section and examples/ for details.

Setup Guide

Step 1: Get a Domain

You need a domain where you control DNS. Supported providers:

• AWS Route53
• Cloudflare
• Name.com
• DigitalOcean
• Hetzner
• Gandi
• Google Cloud DNS
• DuckDNS

Step 2: Configure Your Router

1. Static DHCP: Give the Homelab Horizon device a fixed IP
2. DNS Server: Point network DNS to the Homelab Horizon device
3. Port Forwarding:
   • 51820/UDP - WireGuard VPN
   • 80/TCP - HTTP (Let's Encrypt challenges)
   • 443/TCP - HTTPS (reverse proxy)

Step 3: Configure Zones & Services

1. Add a DNS zone with your domain and provider credentials
2. Add services — each gets a domain, internal DNS, optional external DNS, and optional HAProxy backend
3. Click "Sync DNS, SSL & HAProxy" to apply everything

Zones & Domains

A zone represents a domain you own (e.g., example.com) and connects it to your DNS provider. Once a zone is configured, you can add services under it with any subdomain.

Wildcard SSL

Each zone automatically gets a wildcard SSL certificate (*.example.com) via Let's Encrypt DNS-01 challenges. This means any service like grafana.example.com or wiki.example.com gets valid HTTPS with no per-service cert management.

For deeper subdomains, add sub-zones. For example, adding "vpn" as a sub-zone to example.com gets you a *.vpn.example.com wildcard — so VPN client names like carl.vpn.example.com also get valid SSL.

Adding Services

Once your zone is set up, adding a service is straightforward:

• Name: human-readable identifier (e.g., grafana)
• Domains: one or more FQDNs under your zone (e.g., grafana.example.com)
• Internal DNS: the LAN IP that VPN/local clients should resolve to (e.g., 192.168.1.50)
• External DNS: enables public DNS records pointing to your public IP (auto-detected)
• Proxy: HAProxy backend (host:port) — can be a LAN service or an external host

Services don't have to be on your local network. The proxy backend can point to any reachable host:port — a Raspberry Pi on your LAN, a VM in the cloud, or a container on the same machine.

Managing services from the CLI (hz)

hz is an operator CLI for driving a whole instance from your workstation — service list/show/create/edit/delete plus a global sync. It's distinct from the per-service hz-client (which is service-token scoped, for deploys/bans/site uploads): hz authenticates with the instance admin token.

Install it straight from the instance (detects your OS/arch, drops the binary in ~/.local/bin or /usr/local/bin, and — if you pass a token — writes ~/.hz_config):

curl -fsSL $HZ_URL/admin/hz/install | HZ_HOST=$HZ_URL HZ_TOKEN=<admin-token> bash

The binaries are served by the instance itself (no GitHub needed). Omit HZ_TOKEN to install the binary only, or build locally with make build-hz. Either way hz reads ~/.hz_config:

{ "host": "http://192.168.1.89:8080", "token": "<admin-token>" }

(HZ_HOST/HZ_TOKEN env or --host/--token flags override the file.)

hz service list                      # table of all services
hz service show grafana --json       # one service
hz setup                             # interactive questionnaire -> create + sync
hz service create --name ebb --domain ebb.example.com \
    --backend 192.168.1.76:8300 --internal-only --health-check /healthz --sync
hz service edit ebb --public         # only the flags you pass change
hz service create --name shop --domain lan.example.com \
    --domains-https www.example.com --backend 192.168.1.76:8080
                                     # lan.example.com stays HTTP, www.example.com gets HTTPS
hz service edit ebb --https --confirm # HTTPS on every domain of the service
hz domain list                       # every domain: service, zone, HTTPS coverage, cert
hz domain ssl add ebb.example.com    # give one domain HTTPS
hz domain ssl rm ebb.example.com --confirm
                                     # drop it back to plain HTTP
hz service delete ebb                # reports what the delete would strand, then refuses
hz service delete ebb --delete-orphans --sync
                                     # ...also retract the SubZone + published DNS record
hz sync --wait                       # trigger a global sync, block until done
hz pending                           # show unsynced config changes
hz ports list --host 192.168.1.76    # reserved ports on a host + suggested free ports
hz ports next --host 192.168.1.76 --count 100
                                     # next free port range (safe band 20000+, common dev ports skipped)
hz schema service                    # dump the request schema (reflected from apitypes)

hz --help lists every command and flag; hz schema service prints the exact JSON request shape the server accepts (generated from the shared internal/apitypes structs, so it never drifts).

Deleting is the one mutation that can leave state behind, because a service doesn't own everything it depends on. The zone SubZone giving its domain HTTPS lives on the zone, and the record published at your DNS provider lives at the provider — neither disappears with the service. hz service delete prints both and stops until you pick --delete-orphans or --keep-orphans:

$ hz service delete grafana
Deleting "grafana" leaves behind:

  ! https grafana.example.com            SubZone "grafana" on zone example.com — keeps a cert SAN and an http->https redirect for a host nothing serves
  ! dns   grafana.example.com            A 198.51.100.10 at the DNS provider — stays live and keeps resolving after the delete
  . dns   grafana.example.com            dnsmasq A 192.0.2.50 — removed automatically on the next sync

  ! needs a decision   . goes away on sync   = shared, left alone

error: this delete strands 2 item(s) listed above — re-run with --delete-orphans or --keep-orphans

Coverage inherited from a wildcard SubZone, and any SubZone another service still uses, are reported as shared and never offered for deletion. A service that strands nothing deletes without asking anything.

Static Sites

A service can serve a folder of files instead of proxying to a backend. Set proxy.static_root to an absolute directory (mutually exclusive with proxy.backend):

{
  "name": "docs",
  "domains": ["docs.example.com"],
  "external_dns": { "ttl": 300 },
  "proxy": { "static_root": "/var/lib/homelab-horizon/docs" }
}

HAProxy can't serve a directory itself, so hz runs a small internal file server (loopback-only, port static_serve_port, default 8091) and routes the service's domains to it by Host header. Static services inherit wildcard SSL, split-horizon DNS, and the internal_only restriction exactly like proxied ones.

hz runs as root, but the file server does not: it runs as a separate child process dropped to the unprivileged nobody user, so it physically cannot read files nobody can't — even a bug in the handler can't leak root-only secrets. (If hz can't drop privileges, it refuses to serve static rather than serve as root.) The served directory must therefore be readable by nobody.

On top of that, the file server is deliberately strict:

• Bound to 127.0.0.1 only — never directly reachable off-box.
• Every file open is pinned inside static_root via os.Root; ../ and symlinks cannot escape the directory.
• Dotfiles and dot-directories (.git, .env, .ssh) are never served.
• Directories are never listed — a directory serves its index.html or returns 404.
• static_root cannot be the filesystem root or a system directory (/etc, /root, /proc, …) — checked even through symlinks.
• Content-Type is set explicitly from the file extension (no content sniffing), and X-Content-Type-Options: nosniff is sent on every response.
• Errors render a standard hz error page, or the site's own 404.html / 5xx.html if present. (A wholly missing/unreadable root can't read its own error page, so that case always shows the built-in page.)

Point static_root at a directory containing only files you intend to publish.

For single-page apps, set "spa": true so a browser refresh on a client-side route (a path with no file extension) serves index.html instead of 404:

{ "name": "app", "domains": ["app.example.com"],
  "proxy": { "static_root": "/var/lib/homelab-horizon/app", "spa": true } }

Deploying a static site

Upload a directory with hz-client (the same token-authed client used for rolling deploys; grab the snippet + token from the service's Integration panel in the UI):

export HZ_TOKEN=<service token>  HZ_URL=https://hz.example.com
curl -sO "$HZ_URL/admin/haproxy/hz-client" && chmod +x hz-client

./hz-client site push ./public             # upload as a new release, atomic swap
./hz-client site push ./public --validate  # dry run: extract + validate, no swap
./hz-client site releases                  # list retained releases
./hz-client site rollback                  # revert to the previous release

Deploys are atomic: the upload is extracted into a fresh release directory, then static_root (an hz-managed symlink) is repointed in a single rename — requests never see a half-written site. The last few releases are retained for rollback. Uploads are received by the root process, validated (no path traversal, no symlinks, size/file caps), and the files are owned by nobody so the unprivileged file server can read them.

This is how the project hosts its own landing page (docs/): a static service on the public domain, served by hz, with auto SSL.

Observability

hz already knows every host and backend it routes to, so it can hand Prometheus a scrape config instead of you maintaining one by hand. Declare the boxes it doesn't route to (a NAS, a DB server), add exporter jobs, and pull the result.

hz host add --name nas --ip 192.0.2.100 --label role=storage
hz exporter add --job node --mode port --port 9100          # node_exporter on every known host
hz exporter add --job postgres --mode static --target 192.0.2.110:9187
hz exporter list                                            # jobs, then live targets with up/down

Three ways a job generates targets:

Mode	Targets
port	one port expanded across hosts (--host '*' = every host hz knows)
service	one target per service backend (per slot for blue-green), for services not already opted in
static	the explicit --target list

Per-service metrics are opt-in and probed before they're served, so a service only appears once hz has actually seen its endpoint respond:

hz service edit grafana --metrics --metrics-path /metrics --sync

The generated config is served at /integration/prometheus/scrape.yaml and /integration/prometheus/targets.json, authorized by a scrape token that is separate from the admin token — a scraper never holds admin rights. The Observability page has copy-run snippets for both a one-time pull and a cron that keeps it current.

Ports

Picking a backend port by hand is how you end up with two services on 8080. hz keeps a server-authoritative map of what's reserved — derived from service backends, HAProxy and WireGuard — plus a built-in denylist of common ports and whatever ranges you exclude yourself.

hz ports next --host 192.0.2.50              # next free port, denylist applied
hz ports next --host 192.0.2.50 --count 2    # a blue-green pair
hz ports list --host 192.0.2.50              # what's reserved, and what's free

port_exclusions in the config adds your own ranges on top of the built-in list. The Ports page shows both tabs — reservations per host, and the exclusions that allocation skips.

Port forwards (UDP / TCP)

HAProxy carries HTTP and TLS. It cannot carry UDP, so QUIC, WebTransport, game and voice traffic needs a plain layer-4 forward on the gateway. A service's forwards list is that forward, managed by hz like everything else — no hand rules to drift or to lose on a reboot.

hz service edit sprink --forward udp:4433:192.168.1.76:4433   # add, or replace udp/4433
hz service edit sprink --remove-forward udp:4433
hz service show sprink                                        # lists Forwards:

"forwards": [
  {"proto": "udp", "port": 4433, "backend": "192.168.1.76:4433", "name": "webtransport"}
]

port is the public port on the gateway; backend is ip:port on the gateway's LAN. The router still has to forward that port to the gateway — hz does not configure the router.

What gets installed

For the example above, with eth0 as the default-route interface:

-t nat    -A PREROUTING     -m addrtype --dst-type LOCAL -j HZ-PREROUTING
-t nat    -A POSTROUTING    -j HZ-POSTROUTING
-t filter -A FORWARD        -j HZ-FORWARD
-t nat    -A HZ-PREROUTING  -p udp --dport 4433 -j DNAT --to-destination 192.168.1.76:4433
-t nat    -A HZ-POSTROUTING -d 192.168.1.76/32 -o eth0 -p udp --dport 4433 -m conntrack --ctstate DNAT -j MASQUERADE
-t filter -A HZ-FORWARD     -d 192.168.1.76/32 -i eth0 -p udp --dport 4433 -m conntrack --ctstate DNAT -j ACCEPT
-t filter -A HZ-FORWARD     -s 192.168.1.76/32 -o eth0 -p udp --sport 4433 -m conntrack --ctstate DNAT -j ACCEPT

Rule	Why
PREROUTING --dst-type LOCAL jump	Only packets addressed to the gateway itself are rewritten, never transit traffic. That covers the internet (via the router), LAN clients, and VPN peers, which resolve the service name to the gateway through split-horizon DNS.
DNAT	Sends the flow to the backend.
MASQUERADE	The backend shares a LAN with the router. Without it the backend replies straight to the router, which drops the reply. The backend therefore sees the gateway as the client address.
HZ-FORWARD accept, -i eth0	Accepts the forwarded flow in both directions before FORWARD's DROP policy. Traffic from VPN peers arrives on wg0 and is left to WG-FORWARD, so VPN profiles and the MFA jail still decide what a peer can reach.
--ctstate DNAT	Only flows that HZ-PREROUTING DNATed match. A LAN host that routes through the gateway gets nothing extra.

The jumps sit in FORWARD, not Docker's DOCKER-USER. Horizon inserts them at position 1. Docker's chains (DOCKER-USER, DOCKER-FORWARD) match only Docker bridge interfaces and return everything else to FORWARD. That is why WG-FORWARD, whose jump sits below Docker's, works on a Docker host today. DOCKER-USER exists only while Docker is installed and running. Putting the rules there would make forwards depend on Docker.

The three HZ-* chains belong to horizon and are rebuilt atomically when they drift. When a forward changes, those chains are rebuilt. When the last forward is removed, the three jumps are deleted and the chains are removed. A host with no forwards never gets the chains or the jumps.

What is refused

• Ports 22, 53, 80 and 443, horizon's own port, HAProxy's ports (including metrics), and the WireGuard port. Refused on either protocol: the DNAT matches everything addressed to the gateway on that port, so forwarding 22 would take the gateway's SSH.
• A backend that is not an IPv4 address inside the gateway's LAN CIDR, and a backend that is the gateway itself. The return path (MASQUERADE out of, and accept in from, the default-route interface) only works for LAN hosts.
• The same proto/port on two services.

The generator checks the reserved ports and the LAN containment itself, not only the API. A hand-edited config, or one synced from an HA peer on a different subnet, cannot install an unsafe forward. The forward is skipped. No rule is ever added to INPUT, no chain horizon does not own is flushed, and no policy is changed.

When it applies

The iptables reconciler installs forwards on its 60-second tick, or immediately with Settings → IPTables → Reconcile now. hz sync does not install them. The pending-changes view lists forward edits with the rest of a service's changes. Forwards are IPv4 only. hz ports list and hz ports next treat both the gateway port and the backend port as reserved, for every protocol.

Metrics

hz serves its own Prometheus exposition at /metrics, guarded by the same admin-or-scrape-token check as the discovery endpoints — it names every peer's MFA posture and where the gateway is soft.

It covers what only hz can answer, and deliberately not host metrics:

Area	Examples
VPN	hz_vpn_peers, hz_vpn_peers_recently_handshaked
MFA	hz_vpn_mfa_jailed_peers, hz_vpn_mfa_active_sessions, hz_vpn_mfa_enrolled_peers{factor}, hz_vpn_mfa_active_exceptions
Edge	hz_haproxy_backend_up{backend}, hz_banned_ips
DNS	hz_dnsmasq_cache_{hits,misses,insertions,evictions}_total, hz_dnsmasq_upstream_{queries,failures}_total{server}
Drift	hz_iptables_rules{state} — sustained unknown or stale means something is editing your firewall
Host	hz_time_synchronised, hz_pending_updates{kind}, hz_apt_lists_age_seconds, hz_certificate_expiry_days{domain}
Controls	hz_control_state{control,requirement}

PCI scope

PCI applies to the cardholder data environment — the services that store, process or transmit account data, plus what connects to them. Not the whole box. Mark them:

"services": [
  { "name": "shop",    "pci_scope": "cde" },
  { "name": "grafana", "pci_scope": "connected" },
  { "name": "wiki" }                              // default: out of scope
]

Scoped services get hz_service_control_state{service,scope,control,requirement} and a per-service table on the dashboard. Unscoped services emit nothing — "not evaluated" must never look like "compliant" — and the default is out, because keeping a service out of the CDE is the cheapest compliance there is.

What hz can observe from the edge:

Control	Requirement	Means
not_internet_exposed	1.3.1	the service is restricted to the local network (proxy.internal_only)
tls_covered	4.2.1	a live certificate covers every domain it answers on
served_over_https	4.2.1	TLS is actually being served, not merely a cert existing on disk
cert_not_expiring	4.2.1	no covering certificate lapses within 30 days
backend_not_cleartext_offhost	4.2.1	hz reaches the backend on loopback rather than in cleartext across a network

What hz cannot see, and does not claim: Requirement 3 (stored account data), 6.2 (secure development), key management, and access control inside the application. hz is an edge control — it can evidence Requirements 1 and 4 well, parts of 10 and 11, and nothing about what an app does with a card number once it has one.

TLS floor

haproxy_tls_min_version sets ssl-min-ver on every bind, defaulting to TLSv1.2 — PCI DSS 4.2.1 has prohibited TLS 1.0 and 1.1 since 2018, and 1.2 is still what a long tail of clients speaks, so defaulting to 1.3 would break them silently. Set "TLSv1.3" to raise it. Ciphers are Mozilla's intermediate profile: forward secrecy and AEAD only.

Setting a lower floor is allowed and reported honestly — hz_control_state shows tls_min_version as not met rather than the config quietly disagreeing with the dashboard.

hz_control_state reports whether a configurable security control is in its hardened setting — vpn_mfa_no_admin_bypass, vpn_mfa_session_bounded, and so on, labelled with the PCI DSS requirement each speaks to. It describes hz's configuration only. Whether that satisfies a requirement is an assessor's judgement over a defined scope, which is why nothing here is named hz_pci_compliant.

Everything else on the box

hz doesn't reimplement what already exists — it owns, installs, or detects:

• HAProxy — hz generates its config, so it switches on HAProxy's built-in exporter (haproxy_metrics_port, default 8405, 0 disables), restricted to RFC1918. No extra process; prometheus-haproxy-exporter would scrape the stats socket from outside for less detail.
• dnsmasq — hz reads dnsmasq's own CHAOS counters (hits.bind, misses.bind, servers.bind, …) directly and publishes them above. That's the same source google/dnsmasq_exporter uses, but it isn't packaged for Debian or Ubuntu, so hz couldn't install it through the vetted allowlist it uses for everything else.
• node-exporter — hz does not compete with it for CPU/memory/disk. Set node_exporter_enabled and hz installs it; if you installed it yourself hz notices on its next health tick and switches the flag on for you. Either way it's folded into the scrape config hz serves as an ordinary node job over every known host.
• Your services — anything declaring integrations.metrics is probed and published in /integration/prometheus/{scrape.yaml,targets.json}.

So a central Prometheus points at one discovery endpoint and gets hz, HAProxy, node-exporter and every compatible service, without per-host scrape config.

Grafana dashboard

Observability → Grafana dashboard generates a ready-to-import dashboard for everything above: VPN and MFA posture, the security controls, edge health, iptables drift, plus dnsmasq and host panels when those are present. Copy it, then in Grafana pick Dashboards → New → Import, paste, and choose your Prometheus data source.

It's generated per deployment rather than shipped as a static file — a box without node-exporter doesn't get a row of "No data" CPU panels, and one without dnsmasq doesn't get cache-hit graphs that will never fill in. A dashboard with dead panels teaches people to ignore panels.

VPN MFA

WireGuard has no second factor of its own. A peer either holds a valid key or it doesn't, and the Noise handshake has no interactive step to hang a prompt on. hz adds one after the tunnel comes up: a peer with no verified session still completes its handshake, but is jailed — confined to the Horizon portal until it enters a TOTP code.

That is a real distinction worth understanding before relying on it. This gates what a peer can reach, not whether it can connect. A stolen key still brings up a tunnel; it just lands somewhere with nothing in it.

Enable under Settings → VPN Multi-Factor Authentication, or set vpn_mfa_enabled in config.json.

What a jailed peer can reach

Nothing but the portal — enforced in three places, because no single one of them covers the whole path:

Layer	Where	Blocks
WG-INPUT	iptables, jumped from INPUT -i wg0	everything addressed to the gateway itself, except Horizon's port, HAProxy's ports, and DNS. That means sshd, exporters, and anything else bound to the WireGuard address.
WG-FORWARD	iptables, per-peer DROP	everything transiting the gateway to the LAN
mfa_jailed	HAProxy ACL, source list at <haproxy dir>/mfa-jailed.lst	every vhost except the portal; the rest redirect to <kiosk_url>/app/mfa

The HAProxy half is not decoration. HAProxy fronts every other service and originates those backend connections itself, so a jail that only covered WG-FORWARD would hand a jailed peer the whole LAN through the proxy. Conversely HAProxy never sees traffic aimed straight at sshd — that is WG-INPUT's job. Traffic to the gateway's own address is delivered locally and never traverses FORWARD at all, which is exactly why the second chain exists.

With MFA off, WG-INPUT is empty and the ACL file is empty: no behaviour changes for anyone not using this.

Sessions

Verifying a code opens a session for a duration the operator allows (vpn_mfa_durations, default 2h/4h/8h/forever). Expiry is pruned on a 60s tick, so a session outlives its nominal end by up to a minute.

Per-peer controls live on the VPN page, on each peer's row — revoke session re-jails a peer immediately, and grant session opens an 8h one without a code, for when someone has lost their authenticator. The enable toggle and the allowed durations are in Settings → VPN Multi-Factor Authentication.

forever is a real session, not an exemption — revoking still applies.

Enforcement scope

vpn_mfa_scope decides whether admins are exempt:

Scope	Behaviour
admins-exempt (default)	Peers in vpn_admins are never jailed, so an operator can't lock themselves out.
all	Nobody is exempt, admins included. Required by PCI DSS 8.5.1, which permits no standing bypass for any user.

Promoting or demoting an admin takes effect immediately, in both directions.

Switching to all is refused — with the peers named — if any VPN admin has neither a TOTP secret nor a passkey, since those are exactly the accounts that would lose a bypass with nothing to replace it. Pass "force": true if that's intended.

Exceptions

The only bypass all scope allows is a time-limited, reasoned exception:

curl -b cookie -X POST -H 'Content-Type: application/json' \
  -d '{"name":"laptop","duration":"4h","reason":"lost phone, replacement Tuesday"}' \
  http://<hz>:8080/api/v1/mfa/exception

Duration and reason are both mandatory, the maximum is 7 days, and there is no permanent form — a bypass nobody has to renew is the thing all scope exists to remove. Grants and revocations are logged at WARN so they're greppable during an assessment, and live exceptions are listed in Settings → VPN MFA.

Headless peers (NAS, printers, site-to-site links) can never complete a portal. Under all scope they need a standing decision rather than a renewed exception: scope them out of the CDE, or treat them as system accounts with documented compensating controls (PCI DSS 8.6).

Locked out? docs/mfa-lockout-recovery.md — keep a copy somewhere reachable without the VPN. Short version: the jail is scoped -i wg0, so LAN SSH and the admin UI over the LAN are unaffected, and a jailed peer can still reach the portal to enrol.

Choosing a factor

A peer may hold a TOTP secret, one or more passkeys, or both; any one of them clears the jail, and the session policy is identical either way.

Passkeys require a secure context, so they appear only when kiosk_url is https. When it isn't, the portal says why rather than offering a button that cannot work. RP ID is the kiosk hostname — change that hostname later and every enrolled passkey is orphaned.

⚠️ Full-tunnel peers: phone-scanned passkeys do not work

If your peers are on the full-tunnel profile, steer them to authenticator codes. The QR-on-desktop, scan-with-phone flow is WebAuthn hybrid transport, and it is not what it looks like: the QR carries no challenge and the phone never contacts hz. It bootstraps an encrypted tunnel between the two devices through a relay service on the public internet, run by Google or Apple. A jailed full-tunnel peer routes everything through WireGuard, the jail drops it, and the browser cannot reach that relay — so the ceremony stalls with no useful error.

Unaffected: lan-access and vpn-only peers, whose ordinary traffic never enters the tunnel and who therefore reach the relay over their own connection.

Always fine, on any profile, because nothing leaves the machine: authenticator codes (TOTP), a passkey built into the browsing device (Touch ID, Windows Hello), or a USB/NFC security key.

hz knows each peer's profile, so the portal shows this warning to affected peers before they enroll — but it cannot detect which kind of passkey someone is about to reach for, so the warning is advisory, not a block.

Enrollment

First contact offers both factors. TOTP shows a QR plus the secret in text; the QR is generated in your browser, so the secret is never sent anywhere but to the peer it belongs to.

The TOTP secret is displayed exactly once. A peer that loses it before adding it to an authenticator needs an admin to hit reset TOTP on its row in the VPN page. That reset clears every factor including passkeys — an operator resetting a peer is normally responding to a lost device, and leaving a registered passkey behind would let that device keep clearing the jail.

A peer can remove its own passkeys from the portal, which is how you retire one device while still holding another.

Requirements and failure modes

• kiosk_url must route to a service with proxy.self. That is what makes the portal a vhost HAProxy can exempt. If its host doesn't resolve to a portal backend, hz logs a warning and falls back to a plain 403 instead of a redirect — deliberately, because redirecting to a host that isn't the portal would loop forever for every jailed peer.
• The portal lives at /app/mfa, not /mfa; the UI is a SPA mounted under /app/.
• Jailed peers get DNS to the gateway (udp/tcp 53). Without it the portal can't resolve by name and a jailed tunnel reads as broken rather than locked.
• The jail is not a login gate. It restricts reachability once connected. Revoking a peer's access still means removing its key.

Testing it

make e2e boots a throwaway multipass VM with real WireGuard, iptables and HAProxy, builds a peer and a stand-in LAN host as network namespaces, and asserts what a peer can actually reach while jailed, once verified, and as an admin. Multipass rather than Docker because hz drives systemctl and systemd-run, which need a real PID 1.

METRICS=1 ./bin/e2e additionally installs real dnsmasq and real node-exporter in the VM and checks hz reads and merges them — the format risk unit tests can't cover, since a test double only proves hz parses what hz expects.

PASSKEY=1 ./bin/e2e additionally reconfigures the VM onto https with a self-signed cert and drives a full WebAuthn ceremony against Chrome's virtual authenticator — real credentials, real signatures, verified by hz — asserting that registration alone does not open a session and that asserting does.

KEEP=1 ./bin/e2e leaves the VM up; REUSE=1 re-runs the assertions against it. To click through the portal yourself from another machine, the VM needs to be on your LAN rather than multipass's host-local network:

BRIDGED=1 BRIDGE_IFACE=<iface> ./bin/e2e

BRIDGE_IFACE is worth preferring over multipass's local.bridged-network setting, which is global to the host and silently applies to every --bridged launch by any project until someone unsets it. Either way, bridging enslaves a physical interface to a new bridge, which can briefly drop the link — don't do it on a box you're administering remotely. bin/e2e refuses to configure any of this for you and says why.

High Availability

Run two HZ instances for automatic failover. No orchestrator, no election, no shared state — just two boxes.

Capabilities: config replication, cert renewal failover, round-robin DNS, read-only guard on the spare.

Failover: when the primary dies permanently, remove it from the spare's fleet peer list. The spare detects it has no primary to follow and promotes itself — no SSH, no config editing, no restart.

How it works

1. One primary, one spare. The primary is the single config writer. The spare pulls config every 30s, validates, and applies changes.
2. Cert ownership is deterministic. Each SSL domain is assigned to one peer via consistent hashing. If the owner dies, ownership shifts automatically and the survivor renews.
3. DNS has both IPs. Round-robin DNS gives clients both addresses. Browsers retry on TCP failure — failover is automatic.
4. Edit on the primary. The spare's UI shows a read-only banner. Mutating API calls return 403 with the primary's ID.
5. Promotion is automatic. Remove the dead primary from the spare's peer list — the spare promotes itself on the next sync cycle.

Two topologies

	Same-subnet	Site-to-site
Use case	Two boxes in one DC/VPC	Two boxes at different locations
Fleet comms	LAN IP	WireGuard tunnel IP
VPN range	Shared /24	Disjoint /24 per site
Complexity	Low	Medium (pre-configure s2s tunnel)

Try it

# Same-subnet HA (startup scenario)
cd examples/ha-same-subnet && ./setup.sh && docker compose up -d

# Site-to-site HA (homelab scenario)
cd examples/ha-site-to-site && ./setup.sh && docker compose up -d

Each example includes a test.sh that verifies startup, replication, guard middleware, and failover. See docs/common-scenarios.md for the full story.

Fleet config

Add to each instance's config.json:

{
  "peer_id": "hz1",
  "config_primary": true,
  "peers": [
    { "id": "hz2", "wg_addr": "10.0.0.2:8080" }
  ]
}

The spare mirrors this with "config_primary": false and marks the primary peer with "primary": true.

Architecture

                    Internet
                       |
           +-----------+-----------+
           |                       |
    Remote VPN Clients      Public HTTPS Traffic
    (phone, laptop, etc.)   (grafana.example.com)
           |                       |
           | :51820/UDP            | :80/:443
           v                       v
       +-------+              +--------+
       |Router |------------->| Router |
       +-------+              +--------+
           |                       |
           v                       v
    +----------------------------------------------+
    |            Homelab Horizon                    |
    |                                              |
    |  WireGuard ---- dnsmasq ---- HAProxy + SSL   |
    |  (VPN server)  (split DNS)  (reverse proxy)  |
    +----------------------------------------------+
           |              |              |
           v              v              v
    +-----------+  +-----------+  +-----------+
    | Local VPN |  | LAN       |  | External  |
    | Clients   |  | Services  |  | Services  |
    | (on-site) |  |           |  |           |
    +-----------+  | grafana   |  | cloud-app |
                   | :3000     |  | :8080     |
                   | nextcloud |  +-----------+
                   | :8080     |
                   | NAS :445  |
                   +-----------+

VPN clients can connect from anywhere — inside your network or remotely over the internet. Services can be local LAN hosts (e.g., 192.168.1.50:3000) or external targets (e.g., a cloud VM). HAProxy terminates SSL and proxies to the configured backend, wherever it lives.

Split-Horizon DNS

The same domain resolves differently depending on where you are:

Location	DNS Resolution	Path
On VPN (remote)	Internal IP (e.g., 192.168.1.50)	VPN tunnel -> direct to service
On VPN (local)	Internal IP (e.g., 192.168.1.50)	Direct to service
Local Network	Internal IP (via dnsmasq)	Direct to service
Public Internet	Your Public IP	Router -> HAProxy -> Service

This means grafana.example.com works with valid HTTPS from everywhere — your couch, your phone on cellular, or the public internet.

Building

Quick Build (current platform)

make

Cross-Platform Builds

Build for all supported platforms:

make build-all

Or build for specific targets:

make build-linux-amd64   # Linux x86_64 (most servers/VMs)
make build-linux-arm64   # Raspberry Pi 4/5, modern ARM64 servers
make build-linux-arm     # Raspberry Pi 2/3, older 32-bit ARM

Binaries are output to dist/.

Create Release Archives

make release

Creates .tar.gz archives for each platform in dist/.

Tests

make check   # gofmt, go vet, golangci-lint
go test ./...
make e2e     # VPN MFA jail, end to end in a multipass VM (see VPN MFA)

Manual Build (without Make)

# Current system
CGO_ENABLED=0 go build -o homelab-horizon ./cmd/homelab-horizon

# Raspberry Pi 4/5 (ARM64)
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o homelab-horizon-arm64 ./cmd/homelab-horizon

# Raspberry Pi 2/3 (32-bit ARM)
CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o homelab-horizon-armv7 ./cmd/homelab-horizon

Note: CGO_ENABLED=0 creates a fully static binary with no external dependencies.

Configuration

Configuration is stored in JSON (with // comment support). Locations searched (in order):

1. /etc/homelab-horizon/config.json
2. /etc/homelab-horizon.json
3. ./config.json
4. ./homelab-horizon.json

Alternatively, pass the full config as JSON via the HZ_CONFIG environment variable.

Example Configuration

{
  "listen_addr": ":8080",
  "auto_heal": true,

  "wg_interface": "wg0",
  "wg_config_path": "/etc/wireguard/wg0.conf",
  "server_endpoint": "vpn.example.com:51820",
  "vpn_range": "10.100.0.0/24",
  "dns": "10.100.0.1",

  "dnsmasq_enabled": true,
  "haproxy_enabled": true,
  "ssl_enabled": true,

  "zones": [
    {
      "name": "example.com",
      "zone_id": "Z1234567890",
      "dns_provider": {
        "type": "route53",
        "aws_profile": "default"
      },
      "ssl": {
        "enabled": true,
        "email": "admin@example.com"
      },
      "sub_zones": ["vpn"]
    }
  ],
  "services": [
    {
      "name": "grafana",
      "domains": ["grafana.example.com"],
      "internal_dns": { "ip": "192.168.1.50" },
      "external_dns": { "ttl": 300 },
      "proxy": {
        "backend": "192.168.1.50:3000",
        "health_check": { "path": "/api/health" }
      }
    }
  ],
  "ntfy_url": "https://ntfy.sh/my-homelab-alerts"
}

Web Interface

Page	Description
/app/dashboard	Overview dashboard
/app/services	Service management — domains, DNS, proxy, health status
/app/domains	Every domain's DNS/proxy/HTTPS state, SSL gaps, and zone records
/app/vpn	VPN client management — create clients, QR codes, invites
/app/bans	IP ban management
/app/checks	Health check status and notifications
/app/observability	Prometheus topology — hosts, exporter jobs, scrape config wiring
/app/ports	Port reservations per host and the allocation denylist
/app/settings	Zones, HAProxy, SSL, health checks, system health, hz CLI install

DNS Providers

Configure your provider in the zone's dns_provider block:

Provider	Type	Required Fields
AWS Route53	route53	aws_profile or aws_access_key_id + aws_secret_access_key
Cloudflare	cloudflare	cloudflare_api_token
Name.com	namecom	namecom_username + namecom_api_token
DigitalOcean	digitalocean	api_token
Hetzner	hetzner	api_token
Gandi	gandi	api_token
Google Cloud DNS	googlecloud	gcp_project (+ optional gcp_service_account_json)
DuckDNS	duckdns	api_token

Single sign-on (OIDC)

hz can hand authentication to an OpenID Connect provider. Local accounts and the admin token keep working alongside it — deliberately, because hz is the edge: the outage that takes the provider down is the one where an operator most needs to sign in.

Configure it in Settings → Users → Single sign-on, which also shows the redirect URI to give the provider and has a Test discovery button. The equivalent config, for anyone who prefers the file:

"oidc": {
  "enabled": true,
  "issuer": "https://id.example.com",
  "client_id": "…",
  "client_secret": "…",
  "name": "Company SSO",
  "allowed_email_domains": ["example.com"],
  "required_claims": {"hd": ["example.com"]},
  "auto_provision": false
}

admin_url must be https: the redirect URI is derived from it, and is <admin_url>/api/v1/auth/oidc/callback. Register exactly that with the provider. Everything else — endpoints, JWKS — comes from discovery.

Who gets in

Four gates, each optional, evaluated in this order:

Setting	Refuses when
allowed_email_domains	the email claim is unverified, or its domain is not listed
required_claims	a named claim is missing, or carries none of the listed values
allowed_groups	groups_claim carries none of these
admin_groups	groups_claim carries none of these (hz has one role, so a non-admin is refused rather than given a session that authenticates nothing)

Groups alone cannot gate every provider. Google Workspace sends no group claims by default, so allowed_groups would never match; the domain gate is what that provider does assert. Conversely a domain alone is weak — a consumer account can carry a company address, and a multi-tenant provider may pass an email nobody verified — so on any provider that is not exclusively yours, pair it with required_claims (Workspace: hd).

An unverified email is refused outright rather than matched, which is the entire point of the gate.

Providers

hz speaks OpenID Connect and nothing else: give it an issuer, and discovery supplies the endpoints and keys. Anything with a .well-known works — Authentik, Keycloak, Authelia, Pocket ID, Zitadel.

If you do not already run one, Zitadel is a good choice and is what this project's authors run. It self-hosts from one container against Postgres, and it federates Google, Microsoft and the rest, so hz keeps a single issuer while people sign in with whatever their company uses. Two things to know before you start: v4 defaults to a separate login container served under /ui/v2/login/, which needs a proxy that can route by path — hz routes by host, so either disable that feature at instance creation (ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_REQUIRED=false) or put a path router in front of it. And its API needs proto h2 upstream, which is what --backend-proto h2 is for.

GitHub is not supported for signing in, and cannot be without new code: GitHub has no OIDC provider for users — no issuer to redirect a person to, no ID token, no standard claims. (It does run one for CI job identity at token.actions.githubusercontent.com, which is machine identity for workflows and cannot back a login page.) The usual answer is to let your IdP federate GitHub and keep hz pointed at the IdP — the same way this deployment federates Google Workspace through Zitadel rather than teaching hz about Google.

Accounts

auto_provision is off by default, and that is a real decision. hz has one privilege level: admin. With a domain gate and auto-provision on, every account in that domain becomes an administrator of your gateway. With it off, someone must already have an account for SSO to attach to, and the account list stays deliberate.

Identity is the subject, never the username or email — those are what a provider lets people change. On first sign-in hz matches the derived username (preferred_username, else the local part of a verified email, else the subject), then stores the subject so the link survives a rename on either side.

DNS checks: answering vs forwarding

The System Health tab reports these separately, because they fail separately and the fixes differ:

Row	Question	Fails when
Answering DNS on <addr>	is a socket replying there	dnsmasq is down, or bind-dynamic never picked that address up
Resolves <probe> via <addr>	can it answer a name it does not serve itself	upstreams unreachable, no-resolv with no server=, or no route out

Both are checked on every address dnsmasq should answer on — the LAN address and the WireGuard gateway address. dnsmasq binds those separately, so it can stop answering on the VPN address while the LAN address stays perfect; VPN clients are then the only ones with no DNS, and nothing else notices.

The forwarding probe resolves example.com by default (IANA-reserved, stable A records). Override with dns_probe_name in config.json. The probe must be a name hz does not serve: a name with an address= line is answered from the config and never touches an upstream, so the check would pass on a box with no internet at all. hz compares the probe against every service domain and zone and refuses to run the check rather than report a meaningless green row.

Upstreams themselves are upstream_dns (default 1.1.1.1, 8.8.8.8, 8.8.4.4). dnsmasq is not given strict-order, so it picks whichever upstream has been answering fastest rather than always preferring the first.

Publishing one path of an internal service

internal_only is all or nothing: the service is reachable from the local networks and the VPN, and nowhere else. public_paths carves exceptions:

hz service edit index \
    --internal-only \
    --public-path /api/packages/iodesystems/debian/ --sync

emits one rule:

http-request deny deny_status 403 if host_index !local_access !{ path_beg /api/packages/iodesystems/debian/ }

Denied unless the source is local or the path matches. Several prefixes AND as negated terms, so any one of them is enough to let a request through.

Use it for the endpoint a machine outside the network must reach while the rest of the service stays inside — a package repository a cloud host installs from, a webhook a provider posts to. The service keeps one backend and one hostname; this is not path routing (two backends under one name), which hz does not do.

Two things this does not give you. It is not authentication — if the backend does not demand credentials on those paths, neither does hz. And a prefix is a tree: /api/packages/ exposes everything beneath it, so name the narrowest prefix that works, with the trailing slash, or /api/packages/x-debian-secret rides along with /api/packages/x-debian.

Backend protocol (h2c)

By default HAProxy talks HTTP/1.1 to every backend, which is what nearly every service wants. A backend that requires HTTP/2 — a gRPC service, or anything built on Connect-RPC — needs proto h2 on its server line:

hz service create --name id --domain id.example.com \
    --backend 127.0.0.1:20005 --backend-proto h2 --internal-only --sync
hz service edit id --backend-proto ""      # back to HTTP/1.1

In the UI it is a select under Timeouts in the service editor.

Why this is per service and not the default: cleartext HTTP/2 is used by prior knowledge. HAProxy sends an HTTP/2 preface immediately, with no negotiation and no fallback, so pointing it at an HTTP/1.1 backend does not degrade — it fails, and it looks like the service is down. The health check travels the same way. WebSocket backends are another reason to leave it off unless asked for: WebSockets over an HTTP/2 upstream need extended CONNECT (RFC 8441) support on both sides.

Only h2 is accepted; anything else is rejected at validation (proxy.backend_proto), and it cannot be combined with a static-folder or self service, whose backend is hz's own HTTP/1.1 server.

Health Checks

Services with HAProxy backends automatically get health checks. Configure ntfy URL to receive push notifications when services go down.

Check types:

• ping: TCP connect to common ports (80, 443, 22)
• http: HTTP GET expecting 200 response
• tls: full handshake per served domain, hourly — catches a certificate that stopped covering a name, and one about to expire

All of these run on hz, and answer "can this box reach the service". That is not the question your users ask.

Reserved slots

Hardware is finite and not everything runs at once. A service marked dormant is a slot held open on purpose: the DNS record, the certificate coverage and the HAProxy entry all stay, and nothing is expected to answer behind them.

What goes quiet is backend reachability — the svc: check and the outside vantage's HTTPS probe, both of which would otherwise report a 503 that is the intended state. What stays checked is DNS and TLS, because those are the reservation: the name still resolving to the right place and the certificate still valid are exactly what has to be intact on the day you bring the slot back, and both fail silently otherwise.

It is not the same as deleting the service, and not the same as disabling a check. Deleting gives the name up; disabling a check says "stop telling me about this"; dormant says "this is deliberately not running", which is a fact about the service worth showing next to it.

hz service edit vay --dormant --dormant-reason "waiting on the new box"
hz service edit vay --dormant=false        # bring it back

Or the Reserved slot switch in the service editor.

Outside-in checks (hz-probe)

hz-probe answers the other question: can the internet reach it. It runs on a host outside the homelab — a cheap VPS — and probes your public names for DNS, HTTPS and latency from there. Results land on the Checks page beside the local ones, tagged with the vantage they came from.

The agent reports in; nothing dials it. It needs no public address, no open port and no certificate of its own, so it works behind NAT or on a VM whose IP changes. hz accepting an authenticated POST adds no exposure — it arrives on the same public edge these checks exist to verify. The only facts that cross the wire are ones the public internet already holds: the hostnames hz serves, and the public IP they should resolve to. No backends, no LAN CIDRs, no VPN ranges.

The other direction is still available (HZ_PROBE_PULL=1), where hz dials the agent. It costs a public address, an inbound firewall rule, a self-signed certificate the agent serves, and a re-pin every time that address changes. Worth it only if you specifically want hz to hold no outbound dependency.

The agent probes on its own schedule and buffers what it saw, so the poll after an hz outage returns the outage rather than a gap in the history.

Protocol — the agent reports, hz answers with a target set only when the agent is holding the wrong one:

agent -> hz   I am GCP, holding target set (none). Here are 0 results.
hz -> agent   took 0. Here is set 4f2a = [api.example.com] -> 203.0.113.10.
              Report every 300s.
agent -> hz   holding 4f2a. Here are 2 results.
hz -> agent   took 2.

The version is a hash of the set, so a new domain in hz means a new version, means the agent is handed it on its next report. Nothing to redeploy.

hz acknowledges a count, and the agent drops exactly that many — so a failed or half-processed report is retried rather than lost. That is what makes an hz outage a delay instead of a hole: the agent keeps probing and flushes the backlog when hz answers again.

Add one from the UI. Checks → Outside vantages → Add vantage mints a token and hands you a one-liner for the outside host:

curl -fsSL https://hz.example.com/admin/hz-probe/install \
  | HZ_PROBE_TOKEN=<minted> sudo -E bash

That downloads the agent from this hz instance, writes the token, installs a hardened systemd unit and starts it. There is nothing to paste back — the agent's first report carries the install grant, and hz turns that into a registered vantage named by the agent. It appears in the list within a minute.

In pull mode there is more to do: the installer also generates a self-signed certificate and prints a URL to paste, and Test connection shows you the certificate the agent presented so you can press Pin it. Nothing is trusted because a test reported it — trust on first use is only trust if somebody says yes.

Serving the binary needs a server built with -tags hzembed (make hz-embed cross-compiles both clients first); without it the installer reports that the build has no embedded clients.

Where the installer lives. It is served on the public-facing hostname (kiosk_url), not the admin one — a host outside the network is exactly who fetches it, so it belongs on the vhost whose threat model already assumes anonymous access, and the admin name stays narrow. On the admin hostname these routes 404. With no kiosk_url configured there is no second vhost and any host serves them.

The script is anonymous; the binary is not. A 7MB download reachable by anyone is a free bandwidth tap and tells a stranger the exact build you are running, so the binary requires an install grant — the token hz minted into the command, good for an hour, or the admin token for the hz CLI. The host fetching it still needs no account and no prior relationship with hz: the credential is the one already in the command somebody pasted.

Or do it by hand, which is the same steps without hz in the loop:

make build-probe-all                       # dist/hz-probe-linux-{amd64,arm64,armv7}
scp dist/hz-probe-linux-amd64 vps:/usr/local/bin/hz-probe

# On the VPS:
hz-probe gen-cert --host 198.51.100.7   # skip if the host has a real certificate
sudo hz-probe install                   # mints a token, writes + starts the unit
sudo hz-probe fingerprint               # the pin_sha256 value

install writes an unprivileged, hardened systemd unit (DynamicUser, the token and key reaching it through systemd credentials rather than loosened file modes), validates it with systemd-analyze verify, enables it, and prints the ready-made remote_probes entry. hz-probe show-systemd prints the unit without installing anything.

Serve TLS. The token crosses the public internet on every poll. With a domain, use a real certificate; on a bare IP, gen-cert writes a self-signed one and hz pins it by fingerprint — a stronger guarantee than a public CA gives for this one connection.

The installer is the one time anything talks to hz. It is a bootstrap an operator is sitting in front of, not the steady state: what it installs holds no address for hz, and the agent it starts never dials anything.

Or configure it directly in config.json:

"remote_probes": [
  {
    "name": "vps-nyc",
    "url": "https://198.51.100.7:8443",
    "token": "SHARED_TOKEN",
    "enabled": true,
    "poll": 60,                            // how often hz asks
    "probe": 60,                           // how often the agent probes
    "resolvers": ["1.1.1.1:53", "8.8.8.8:53"],
    "pin_sha256": "…"                      // for a self-signed agent cert
  }
]

This block is local to the host, never synced to HA peers: the token is a credential for an agent this peer chose. Name resolvers explicitly if you care which ones agree — a VPS's system resolver is usually a caching forwarder with a view of its own.

Keeping agents current. hz tells each agent which build it holds on every report. An agent on an older one logs a warning and shows a needs update chip on its row — a vantage running an old binary still reports, so nothing else about it looks wrong, which is exactly why it needs saying.

The update itself is a daily root timer (hz-probe-update.timer) running hz-probe update: it asks hz for the version, downloads it if that differs, checks the new binary runs, swaps it in and restarts. The agent does none of this — it runs as a DynamicUser and cannot write its own binary or restart its unit, deliberately. An agent that fetched and executed what hz sent it would turn a compromised hz into code execution on a host outside your network. hz is a source of bytes over a verified connection, never an instruction.

Re-running the install command does the same thing immediately, and --no-auto-update skips the timer.

The Checks page lists each vantage with its own state, which the check rows cannot carry: a vantage hz has never reached produces no rows at all, and that reads identically to one nobody configured.

Reading the history

The history panel groups rows by where the check ran — hz first, then each vantage — because inside and outside do not have to agree, and when they disagree, which one is wrong is the whole diagnosis. Each group gets one strip showing the worst status across it; only checks that actually changed state get their own row, and everything steady collapses behind a count. Latency is plotted on a log scale with decade gridlines, since a local TCP connect is single-digit milliseconds and an HTTPS request from another continent is hundreds.

The server sends this bucketed and run-length encoded (?buckets=N, default 120), so a check that was up the whole window is one run rather than one sample per column. Twenty-five checks across three vantages is 28 SVG nodes and about 17 KB.

Rows it produces, all prefixed ext::

Row	Means
ext:vps-nyc:agent	whether hz can reach the agent at all
ext:vps-nyc:dns:api.example.com	what the name resolves to from outside, against what hz expects
ext:vps-nyc:https:api.example.com	status code, latency, and days left on the certificate

The agent row is the one that keeps the rest honest — without it, a dead agent reads as every target frozen on its last known status.

SSL Certificates

Wildcard certificates are automatically obtained via Let's Encrypt using DNS-01 challenges. A background sweep runs every 12 hours and renews certificates within 30 days of expiry — no operator action needed.

In an HA fleet, cert renewal is deterministic: each domain is assigned to one peer. If that peer dies, ownership shifts automatically and the survivor renews. Non-owners pull certs from the owner.

Certificates cover:

• *.example.com (base zone)
• *.vpn.example.com (sub-zones you configure)

Requirements

• Ubuntu/Debian Linux
• Go 1.25+ and Node.js (for building from source)
• Root access — needed for WireGuard, dnsmasq, HAProxy, iptables, systemd service management, and binding ports 80/443

Runtime packages (auto-installed when auto_heal is enabled):

• iproute2 - Network interface management
• wireguard-tools - VPN management
• haproxy - Reverse proxy (when haproxy_enabled)
• dnsmasq - Internal DNS (when dnsmasq_enabled)
• iptables - NAT masquerading
• qrencode - VPN client QR codes

License

MIT