gitlab-metrics-exporter

README

GitLab Metrics Exporter

NOTE: This project is under development and considered experimental. It is not in use at GitLab currently.

---

GitLab Metrics Exporter (GME) is meant to subsume the functionality of gitlab-exporter and in-app Ruby Prometheus exporters distributed with and running in gitlab-rails.

It has the following goals:

• Provide a single, unified mechanism to export application-specific metrics to Prometheus.
• Ability to run as a standalone system or as a sidecar process to gitlab-rails.
• Ability to collect and serve a variety of metrics using a flexible approach to probing.
• Ability to collect and serve metrics in a memory- and CPU-efficient way.
• Ability to exploit concurrency when running multiple probes.

Design

GME follows a simple design based on two primary concepts:

• Probes. A probe is responsible for sampling data. It is up to the probe how that is accomplished; it could query a connected data store, send HTTP requests, perform IPC or read from the file system. A probe returns metric data as a series of MetricGroups that can be consumed by a renderer. Probes can run in parallel. Metrics within a MetricGroup need not be ordered. However, all samples belonging to a particular metric must appear in a single MetricGroup.
• Renderers. A renderer takes a series of MetricGroups emitted from one or more probes and serializes it, so it can be ingested by Prometheus.

The exporter exposes a single endpoint, /metrics. When a client requests this endpoint, all probes will execute in parallel and their results are combined, serialized, and rendered to the client. This can be understood as a function: render(merge(probe1(), ..., probeN())).

Which probes should run in response to a request is specified via configuration switches.

Data model

For reasons of efficiency and simplicity, GME does not use the data model from the official Go Prometheus client. GME's data model is flat instead of hierarchical and only uses two concepts: Samples and MetricGroups. A Sample is a named and labeled 64 bit float value. A MetricGroup organizes lists of samples by metric type.

There is currently no MetricFamily in GME; however, a MetricGroup can be understood as a series of one or more metric families. If multiple metric families are emitted by a probe as one MetricGroup, then they will share a HELP line when rendered. If this is undesirable, one must emit a single MetricGroup per MetricFamily.

The differences between the two data models are summarized below.

GME type	Description	Prometheus type
Metric group	A list of samples of the same metric type	-
Sample	A labeled metric mapped to a numeric value	Metric + value for counters and gauges, Metric + Samples for histograms and summaries
Label	A metric dimension as a name/value pair	LabelPair
(metric family)	A single Metric for counters and gauges, several Metrics for histograms and summaries	MetricFamily

Probes

The following probes are currently implemented:

• self. This probe samples Go runtime, procfs, and HTTP server metrics.
• mmap. This probe reads db files written by prometheus-client-mmap. It is the format used to emit application metrics from gitlab-rails.

Renderers

The following renderers are currently implemented:

• text. This renderer implements Prometheus' default Text Exposition Format.

Configuration

You can see a list of configuration options by running gitlab-metrics-exporter -h.

Settings are passed as command line flags or through the environment.

See the following examples for how to run the server.

Run with defaults

This binds the server to all network interfaces using the default port and runs the self probe:

$ gitlab-metrics-exporter

Run with custom host or port

You can change the default host and port as follows:

$ gitlab-metrics-exporter --host 127.0.0.1 --port 1234

Or, alternatively, via the GME_SERVER_HOST and GME_SERVER_PORT environment variables.

Run with custom probes

To start the server on the default port and run the self and mmap probes:

$ gitlab-metrics-exporter --probe self --probe mmap --mmapdir /path/to/metrics

Probes can be set via the environment as well:

$ GME_PROBES=self,mmap gitlab-metrics-exporter --mmapdir /path/to/metrics

As can other flags:

$ GME_MMAP_METRICS_DIR=/path/to/metrics gitlab-metrics-exporter --probe self --probe mmap 

Configure logging

You can specify a log file (including stderr and stdout), log format, and log level:

$ gitlab-metrics-exporter --log-file /path/to/log --log-format json --log-level info

Or via environment variables:

$ GME_LOG_FILE=/path/to/log GME_LOG_FORMAT=json GME_LOG_LEVEL=info gitlab-metrics-exporter

Configure mutual TLS

You can enable mutual certificate verification between client and server as follows:

$ gitlab-metrics-exporter --cert-file /path/to/cert.pem --cert-key /path/to/key.pem

Or via environment variables:

$ GME_CERT_FILE=/path/to/cert.pem GME_CERT_KEY=/path/to/key.pem gitlab-metrics-exporter

Note:

• Certificate and key files must be encoded in PEM format.
• The cert-file must be a bundle file i.e. contain the server certifcate, the root CA and all intermediaries.
• Certificate validation is mutual i.e. all clients must present their own certificate.

Contributions

See CONTRIBUTING.