dn-prometheus-sd

README

dn-prometheus-sd

Prometheus service discovery for Defined Networking (Managed Nebula) networks.

dn-prometheus-sd fetches your hosts from the Defined Networking API and writes Prometheus file_sd_configs target files based on the tags assigned to each host. Tag a host svc:node-exporter in the DN admin panel and it shows up in your node scrape job — no Prometheus reload, no config edits.

How it works

1. Fetch all hosts from GET https://api.defined.net/v2/hosts.
2. For each configured tag→job mapping, collect matching hosts and build IP:port targets from their Nebula addresses.
3. Write one file_sd JSON file per job into the output directory, atomically.
4. Exit (one-shot mode, for cron or a systemd timer), or sleep and repeat (daemon mode via -interval).

The tool does not generate prometheus.yml. You keep defining jobs there — scrape interval, metrics path, scheme, TLS — and point each job at the generated file:

scrape_configs:
  - job_name: 'node'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/node.json']

Prometheus watches file_sd files with inotify and picks up changes immediately, without a reload.

Failure model

If a run fails — network error, non-200 response, bad JSON — the tool touches nothing: existing files stay in place, the error is logged, and it exits non-zero (daemon mode logs and retries). Prometheus keeps scraping the last-written files, so a broken run degrades to "targets frozen", never "targets gone".

As an extra guard, if the API returns zero hosts while existing target files are non-empty, the run is treated as a failure — an API that suddenly reports an empty network is more likely broken than a network that suddenly lost every host. Disable with -empty-guard=false if you really do want an empty network to empty your targets.

Installation

$ go install github.com/johnmaguire/dn-prometheus-sd@latest

Or build from a checkout:

$ go build -o dn-prometheus-sd .

Configuration

# /etc/dn-prometheus-sd.yaml
output_dir: /etc/prometheus/targets
address_family: prefer-ipv4   # prefer-ipv4 | ipv4 | ipv6

mappings:
  - tag: svc:node-exporter
    job: node
    port: 9100
  - tag: svc:windows-exporter
    job: windows
    port: 9182
  - tag: svc:incus
    job: incus
    port: 8444
  - tag: nebula:metrics
    job: nebula
    port: 9133
  - tag: svc:smartctl-exporter
    job: smart
    port: 9633
  - tag: svc:traefik
    job: traefik
    port: 8090
  - tag: svc:cadvisor
    job: cadvisor
    port: 8080
  - tag: svc:smokeping
    job: smokeping-prober
    port: 9374
  # One tag can feed multiple jobs (different ports on the same hosts).
  - tag: svc:immich
    job: immich_api
    port: 8081
  - tag: svc:immich
    job: immich_microservices
    port: 8082

Key	Description
output_dir	Directory the target files are written to. Must be dedicated to this tool: after each successful run, *.json files that don't correspond to a configured job are deleted, so removing a mapping removes its file.
address_family	prefer-ipv4 (default) picks the host's IPv4 address, falling back to IPv6. ipv4 and ipv6 are strict: hosts lacking an address in that family are skipped with a warning. IPv6 targets are bracketed ([fdef::1]:9100).
mappings	List of tag→job mappings, matched against full tags (svc:node-exporter, nebula:metrics, …) — no prefix convention is imposed.
mappings[].tag	Tag to match, e.g. svc:node-exporter.
mappings[].job	Output file name: targets are written to <output_dir>/<job>.json.
mappings[].port	Port appended to each matching host's address.
mappings[].labels	Optional labels merged into every target group of this job.

A host may match many mappings, and the same tag may appear in multiple mappings (to feed one tag into several jobs). Tags without a mapping are ignored. Hosts with isBlocked: true are skipped; lighthouses and relays are included like any other host — tags decide.

Each mapping with zero matching hosts still writes a valid empty file ([]), so Prometheus never complains about a missing file.

Output format

One file per job, one target group per host, so per-host labels work:

[
  {
    "targets": [
      "192.168.128.15:9100"
    ],
    "labels": {
      "dn_name": "demeter.example.com"
    }
  }
]

Every target group carries a dn_name label with the Defined Networking host name; per-mapping labels are merged on top. Output is deterministic (sorted by host name, stable key order) and written atomically (temp file + rename). Writes are skipped entirely when content is unchanged, so file mtimes only move when targets actually change.

Use dn_name to get readable instance labels instead of raw IP:port:

scrape_configs:
  - job_name: 'node'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/node.json']
    relabel_configs:
      - source_labels: [dn_name]
        target_label: instance

API token

Create an API key in the Defined Networking admin panel with the hosts:list scope — nothing more. The tool reads it from the DN_API_TOKEN environment variable, or from a file via -token-file (handy for systemd LoadCredential). It is never read from the config file.

v1 vs v2 API gotcha: GET /v1/hosts returns HTTP 200 with an empty data array for hosts on v2 (dual-stack) networks — it looks like a permission problem but isn't. This tool exclusively uses GET /v2/hosts, which returns all hosts. Keep that in mind if you ever poke the API by hand.

Usage

$ dn-prometheus-sd -config /etc/dn-prometheus-sd.yaml                # one-shot
$ dn-prometheus-sd -config /etc/dn-prometheus-sd.yaml -interval 5m   # daemon
$ dn-prometheus-sd -config /etc/dn-prometheus-sd.yaml -dry-run       # print, don't write
$ dn-prometheus-sd -version

Flag	Description
-config	Path to the YAML config file (required).
-interval	Refresh interval. 0 (default) runs once and exits — use cron or a systemd timer.
-dry-run	Print would-be file contents plus a diff against the current files; write nothing.
-token-file	Read the API token from this file instead of DN_API_TOKEN.
-empty-guard	Refuse to act when the API returns zero hosts but existing files have targets (default true).
-log-level	debug, info, warn, or error (default info). Logs go to stderr in log/slog text format.

Exit codes: 0 on success, non-zero on any failure in one-shot mode. Daemon mode logs errors and retries on the next tick.

Running under systemd

One-shot service plus timer (recommended over daemon mode — you get retries, logging, and scheduling from systemd for free):

# /etc/systemd/system/dn-prometheus-sd.service
[Unit]
Description=Generate Prometheus file_sd targets from Defined Networking
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/dn-prometheus-sd -config /etc/dn-prometheus-sd.yaml -token-file ${CREDENTIALS_DIRECTORY}/dn-api-token
LoadCredential=dn-api-token:/etc/dn-prometheus-sd.token
User=prometheus
Group=prometheus

# Hardening
ProtectSystem=strict
ReadWritePaths=/etc/prometheus/targets
PrivateTmp=true
NoNewPrivileges=true

# /etc/systemd/system/dn-prometheus-sd.timer
[Unit]
Description=Refresh Prometheus targets from Defined Networking

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min
RandomizedDelaySec=30s

[Install]
WantedBy=timers.target

$ systemctl enable --now dn-prometheus-sd.timer

License

MIT

Documentation

Overview

Command dn-prometheus-sd generates Prometheus file_sd target files from Defined Networking (Managed Nebula) host tags.