fluent-bit-out-prometheus-metrics

README

Fluent Bit Output Plugin for Log Derived Metrics

Purpose

To add missing Log Derived Metrics functionality to Fluent Bit. When combined with Grafana Loki, Prometheus, Cortex, and Grafana UI, you get a fully functional Data Observability Platform.

Design

The Output plugin leverages the prometheus pushgateway API by pushing the tracked metrics to pushgateway backed with persistent store. This allows for a fire-and-forget architecture to prevent fluent bit from blocking. Prometheus treats pushgateway as a metric scrape target.

While the plugin pairs nicely with Grafana Loki's Output plugin, it can be used standalone or coupled with other output plugins.

---

Configuration Parameters

Key	Description	Required	Default	Valid Options	Notes
id	Plugin instance id	Yes			Must be unique per [OUTPUT] section in a single fluent-bit.conf
job	Prometheus job label	Yes
url	HTTP Url for destination push gateway	Yes			Ex. http://127.0.0.1:9091
push_gateway_retries	Number of retry attempts to connect to push gateway	No	3
metric_type	Prometheus metric type	Yes	none	Counter, Gauge, Summary, Histogram
metric_name	Metric name sent to Prometheus	Yes
metric_help	Help string associated with metric	Yes			Enclose in double quotes
metric_constant_labels	Static JSON formatted key/value pairs to index metric	No			Although not required, {"instance":"1"} is recommended. Ex. {"instance":"1", "source":"fluent-bit"}
metric_variable_labels	Comma separated list of fluent bit fields to index metric. This is the key to log derived metrics. The value of these keys will vary depending on log line content.	No			Will be appended to any metric_constant_labels is configured. Ex. origin, status_code, method

Metric Specific Configurations

In addition to keys noted above.

Counter

See Prometheus Counter for details. No additional parameters are required Counter at this time.

Summary

See Prometheus Summary for details.

Key	Description	Required for Specific Metric Type	Default	Valid Options	Notes
metric_summary_observe_key	Single fluent bit field to observe for Summary metric type.	Yes

Histogram

See Prometheus Histogram for details.

Key	Description	Required for Specific Metric Type	Default	Valid Options	Notes
metric_histogram_bucket_type	Histogram bucket distribution	Yes		Linear, Exponential
metric_histogram_observe_key	Single fluent bit field to observe for Histogram metric type.	Yes

Histogram with Linear Bucket Type

Creates 'count' buckets, each 'width' wide, where the lowest bucket has an upper bound of 'start'. The final +Inf bucket is not counted and is not included. See Prometheus SDK reference - LinearBuckets() for details.

Key	Description	Required for Specific Metric Type	Default	Valid Options	Notes
metric_histogram_linear_buckets_count	Count of buckets	Yes		> 0	Ex. 5
metric_histogram_linear_buckets_width	Width of bucket	Yes			Ex. 5
metric_histogram_linear_buckets_start	Lowest bucket has an upper bound of Start	Yes			Ex. 20

Histogram with Exponential Bucket Type

Creates 'count' buckets, where the lowest bucket has an upper bound of 'start' and each following bucket's upper bound is 'factor' times the previous bucket's upper bound. The final +Inf bucket is not counted and not included. See Prometheus SDK reference - ExponentialBuckets() for details.

Key	Description	Required for Specific Metric Type	Default	Valid Options	Notes
metric_histogram_exponential_buckets_count	Count of buckets	Yes		> 0	Ex. 5
metric_histogram_exponential_buckets_factor	Each additional bucket upper bound is Factor times the previous bucket's upper bound	Yes			Ex. 1.5
metric_histogram_exponential_buckets_start	Lowest bucket has an upper bound of Start	Yes			Ex. 20

Gauge

See Prometheus Gauge for details.

Key	Description	Required for Specific Metric Type	Default	Valid Options	Notes
metric_gauge_method	Method selected to change the value of the Gauge	Yes		Set, Add, Sub, Inc, Dec	Set, Add, and Sub require a key as input. Inc and Dec only increment or decrement the gauge by 1.
metric_gauge_set_key	Single fluent bit field as input to Set method.	Yes with Set
metric_gauge_add_key	Single fluent bit field as input to Add method.	Yes with Add
metric_gauge_sub_key	Single fluent bit field as input to Sub method.	Yes with Sub

Example Configurations

The example folder contains a set of configurations showing each type of metric currently supported by the plugin plus Grafana Loki logs. These were used to create the dashboard pictured above.

• Check out https://github.com/neiman-marcus/fluent-bit-data-observability-platform for a full environment leveraging the example configuration.

---

Using the demo environment

Requirements:

• Either Docker Desktop, or docker and docker-compose.
• Make
• Curl

Quick start

make all

Build instructions

make build

Start Prometheus Push Gateway and Fluent bit

make start

Show current metrics

Execute repeatedly to see the metrics generated using the plugin.

make show-metrics

Show current logs

Execute repeatedly to see the latest container logs

make logs

Stop both containers

Shutdown demo environment.

make stop

Notes

The build-plugin configuration in the Makefile is referenced by the docker build environment.

Projects referenced above:

https://fluentbit.io/
https://grafana.com/oss/grafana/
https://grafana.com/oss/loki/
https://grafana.com/oss/prometheus/
https://grafana.com/oss/cortex/

Reference Talk: FluentCon 2021: Fluent Bit - Swiss Army Tool of Observability Data Ingestion

Author

• Michael Marshall - Project creation and development.

Conduct / Contributing / License

• Refer to our contribution guidelines to contribute to this project. See CONTRIBUTING.md.
• All contributions must follow our code of conduct. See CONDUCT.md.
• This project is licensed under the Apache 2.0 license. See LICENSE.

Acknowledgments

Thank you to the teams who create and maintain the individual projects listed above.