Prometheus-Configmanager
Prometheus Configmanager consists of two HTTP-based configuration services for Prometheus and Alertmanager configurations. Both Prometheus and Alertmanager use yaml files configuration, and are only modifiable by directly rewriting these files and then sending a request to the respective service to reload the configuration files. Configmanager provides an HTTP API to modify and reload these configuration files remotely (alertmanager.yml and alert rules files used by prometheus).
Multi-tenancy
Both configmanagers are built with multiple tenants in mind. API paths require a tenant_id
which uniquely identifies a tenant using the system. While multiple tenants operate on the same configuration, there is no worry about conflict as every alerting rule, routing receiver, or other component is kept distinct by using the tenant ID.
The basic way of providing multitenancy in prometheus components is by using labels. For example, in a multitenant alertmanager-configurer setup, each alert is first routed on the tenancy label, and then the routing tree is distinct for each tenant. With prometheus, alerting rules can be restricted so that each rule can only be triggered by metrics which have the label {tenancyLabel: tenant_id}
.
Prometheus
Command line Arguments:
-port string
Port to listen for requests. Default is 9100 (default "9100")
-prometheusURL string
URL of the prometheus instance that is reading these rules. Default is prometheus:9090 (default "prometheus:9090")
-multitenant-label string
The label name to segment alerting rules to enable multi-tenant support, having each tenant's alerts in a separate file. Default is tenant (default "tenant")
-restrict-queries
If this flag is set all alert rule expressions will be restricted to only match series with {<multitenant-label>=<tenant>}
-rules-dir string
Directory to write rules files. Default is '.' (default ".")
Alertmanager
Command line Arguments
-alertmanager-conf string
Path to alertmanager configuration file. Default is ./alertmanager.yml (default "./alertmanager.yml")
-alertmanagerURL string
URL of the alertmanager instance that is being used. Default is alertmanager:9093 (default "alertmanager:9093")
-multitenant-label string
LabelName to use for enabling multitenancy through route matching. Leave empty for single tenant use cases.
-port string
Port to listen for requests. Default is 9101 (default "9101")
HTTP API Documentation
Swagger documentation for the APIs can be found at prometheus/docs/swagger-v1.yml
and alertmanager/docs/swagger-v1.yml
Operation
The general way of using these services is by letting them take control of your Prometheus and Alertmanager configuration files. As such, they should be run on the same pod (if using kubernetes) as those services. Once set up, it is best to not edit these files manually as you may put it in a bad state that configmanager is not able to understand. Note that prometheus.yml is not directly modified by these services, so that is safe so long as you have a section like below:
rule_files:
- '/etc/prometheus/alert_rules/*_rules.yml'
Where at least one of the elements in the array is pointed to the same directory that configmanager is writing the rules files (controlled by command line arguments).
Building Docker Containers
Use the following commands to build the containers:
docker build -t prometheus-configurer -f Dockerfile.prometheus .
docker build -t alertmanager-configurer -f Dockerfile.alertmanager .
Deploying on Minikube
On your local machine start Minikube. Your exact command may be different due to different VM providers.
minikube start --mount --mount-string "<path-to>/prometheus-configmanager:/prometheus-configmanager"
SSH into the minikube vm with
minikube ssh
From minikube build all the docker containers:
$ cd /prometheus-configmanager
$ docker build -f Dockerfile.alertmanager -t alertmanager-configurer .
$ docker build -f Dockerfile.promtheus -t prom-configurer .
$ cd ui
$ docker build -t alerts-ui .
Back on your host machine:
$ cd <path-to-prometheus-configmanager-repo>
$ helm init
$ helm install --name prometheus-configmanager .
You can then check the status of the deployment with
$ helm status prometheus-configmanager
And you should see something like this:
LAST DEPLOYED: Mon Jun 29 14:02:43 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Deployment
NAME AGE
alertmanager-configurer 34m
alerts-ui 34m
prometheus-configurer 34m
==> v1/Pod(related)
NAME AGE
alertmanager-configurer-5b57b9b5d5-hns7d 34m
alerts-ui-85566df78c-7wcl2 34m
prometheus-configurer-575567dd95-x4mnj 34m
==> v1/Service
NAME AGE
alertmanager-configurer 34m
alerts-ui 34m
prometheus-configurer 34m
Third-Party Code Disclaimer
Prometheus-Configmanager contains dependencies which are not maintained by the maintainers of this project. Please read the disclaimer at THIRD_PARTY_CODE_DISCLAIMER.md.
License
Prometheus-Configmanager is MIT License licensed, as found in the LICENSE file.