Promsaint
Nagios -> Prometheus Alertmanager integration
When transitioning from Nagios to Prometheus it is useful to be able to maintain the alerts configured in nagios. Promsaint allows you to push alerts generated in Nagios into the Prometheus Alertmanager so that you can have one consistent alerting and control plane.
Names
"Prometheus Ain't Gonna Insist On Sainthood"
Promsaint is a portmanteau of Prometheus and the original name for Nagios NetSaint.
Building
A Makefile is provided with this project to make it extremely easy to just clone and build all the binaries
make
Running the Daemon
./promsaint
Usage of ./bin/promsaint:
-alertmanager string
Alertmanager host (default "http://localhost:9093")
-forgetage duration
How long to persist an alert after nagios forgets about it before removing it (default 1m0s)
-generator_url string
Breakdown Source for nagios (default "https://nagios.example.com/nagios/cgi-bin/status.cgi")
-listen string
HTTP endpoint (default ":8080")
-log.format value
Set log formatter
-log.level value
Set log level
-pruneage duration
How old of alerts to keep around (default 1m0s)
-pruneinterval duration
How often to prune the database (default 2m0s)
-publishinterval duration
How often to publish the database (default 1m0s)
-publishminimum duration
Minimum time to wait between alertmanager pushes (default 5s)
Docker
Alternately you can start an Promsaint and Alertmanager instance automatically using docker-compose
docker-compose up
Running the CLI
The primary interaction method from Nagios to the Promsaint Daemon is done through the included promsaint-cli.
-hostalert
This is a host alert (default: false)
-servicealert
This is a service alert (default: false)
-notify
Value of notify label (default: blackhole)
-actifid
Value of $_HOSTACTIFID$ (default: empty)
-ressource
Value of $SERVICEDESC$ (default: empty)
-Env
Value of $_HOSTENV$ (default: empty)
-datetime
Value of "$SHORTDATETIME$" (default: empty)
-ntype
Value of $NOTIFICATIONTYPE$ PROBLEM/ACKNOWLEDGEMENT/RECOVERY (default: empty)
-state
Value of $HOSTSTATE$: UP/DOWN or $SERVICESTATE$: CRITICAL/WARNING/UNKNOWN/OK (default: empty)
-severity
Value of $_HOSTSEVERITY$ or "$_SERVICESEVERITY$": MINOR/MAJOR/CRITICAL (default: empty)
-host
Value of $HOSTALIAS$ of firing alert (default: empty)
-service
Value of $SERVICEDESC$ of firing alert (default: empty)
-msg
Value of $SERVICEOUTPUT$ (default: empty)
-note
Value of $HOSTNOTESURL$ or $SERVICENOTESURL$ (default: empty)
-promsaint
Url of running promsaint Daemon to post to (default: http://localhost:8080)
Host Alerts
./promsaint-cli -promsaint URL -hostalert -notify linux -ntype PROBLEM -state DOWN -host "test.lan" -env "dev" -severity "MAJOR" -msg "The host is down" -actifid "1234567" -datetime "10-13-2000 00:30:28" -note "Mor info on this host"
And the recovery
./promsaint-cli -promsaint http://URL:8080 -hostalert -notify linux -ntype RECOVERY -state UP -host "test.lan"
Service Alert
./promsaint-cli -promsaint http://URL:8080 -servicealert -notify linux -ntype PROBLEM -state WARNING -host "test.lan" -env "dev" -severity "MAJOR" -note "Mor info on this service" -service "Memory Usage" -msg "90% Memory Usage" -actifid "1234567" -datetime "10-13-2000 00:30:28"
Recovery
./promsaint-cli -servicealert -notify linux -ntype RECOVERY -state OK -host "test.lan" -service "Memory Usage"
Nagios Configuration
Included in this repo is a sample config file that use can use to start sending alerts from Nagios into Alertmanager.
promsaint.cfg
Drop that file in your Nagios config directory and include the users in contact sections of your configured services and after a quick reload you should see alerts appearing in Alertmanager!
Smoke Tests
An end-to-end testing tool is provided promsaint-smoke
which will create a testing alert in promsaint and check alertmanager to ensure that it was created there. At this point it will resolve the alert.
./promsaint-smoke
Systemd
Included in this repo the service files for systemd
promsaint-devl.service & promsaint-prod.service
Modify the ExecStart command for your needs and drop these files in /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start promsaint-devl.service
sudo systemctl start promsaint-prod.service
sudo systemctl enable promsaint-devl.service
sudo systemctl enable promsaint-prod.service
Multiple Alertmanagers
Starts two promsaint servers listening on different ports and point them to their respective alertmanager:
bin/promsaint -alertmanager "http://alertmanager1:9093" -listen ":8080"
bin/promsaint -alertmanager "http://alertmanager1:9093" -listen ":8081"
In Nagios, create a contact and command set for each alertmanager. See promsaint.cfg