plugin

README

Fluent Bit Plugin

The Fluent Bit output plugin for ClickHouse can be used to write the collected logs from Fluent Bit into ClickHouse.

Configuration

An example configuration file can be found in the fluent-bit-cm.yaml ConfigMap. The following options are available:

Option	Description	Default
Metrics_Server_Address	The address, where the metrics server should listen on.	:2021
Address	The address, where ClickHouse is listining on, e.g. clickhouse-clickhouse.clickhouse.svc.cluster.local:9000.
Database	The name of the database for the logs.	logs
Username	The username, to authenticate to ClickHouse.
Password	The password, to authenticate to ClickHouse.
Dial_Timeout	ClickHouse dial timeout.	10s
Conn_Max_Lifetime	ClickHouse maximum connection lifetime.	1h
Max_Idle_Conns	ClickHouse maximum number of idle connections.	1
Max_Open_Conns	ClickHouse maximum number of open connections.	1
Async_Insert	Use async inserts to write logs into ClickHouse.	false
Wait_For_Async_Insert	Wait for the async insert operation.	false
Batch_Size	The size for how many log lines should be buffered, before they are written to ClickHouse.	10000
Flush_Interval	The maximum amount of time to wait, before logs are written to ClickHouse.	60s
Force_Number_Fields	A list of fields which should be parsed as number.	60s
Force_Underscores	Replace all . with _ in keys.	false
Log_Format	The log format for the Fluent Bit ClickHouse plugin. Must be console or json.	console
Log_Level	The log level for the Fluent Bit ClickHouse plugin. Must be debug, info, warn, error, fatal or panic.	info

Development

We are using kind for local development. To create a new Kubernetes cluster using kind you can run the cluster/cluster.sh script, which will create such a cluster with a Docker registry:

./cluster/cluster.sh

Once the cluster is running we can build and push the Docker image for Fluent Bit:

docker build -f cmd/plugin/Dockerfile -t localhost:5000/klogs:latest-plugin .
docker push localhost:5000/klogs:latest-plugin

# To run the Docker image locally, the following command can be used:
docker run -it --rm localhost:5000/klogs:latest-plugin

In the next step we have to create our ClickHouse cluster via the ClickHouse Operator. To do that we can deploy all the files from the cluster/clickhouse-operator and cluster/clickhouse folder:

k apply -f cluster/clickhouse-operator
k apply -f cluster/clickhouse

Once ClickHouse is running we have to connect to one ClickHouse instance to create our SQL schema. The schema can be found in the schema.sql file, just execute each SQL command one by one on the ClickHouse instance:

k exec -n clickhouse -it chi-clickhouse-sharded-0-0-0 -c clickhouse -- clickhouse-client

Now we can deploy Fluent Bit to ingest all logs into ClickHouse:

k apply -f cluster/fluent-bit/plugin
k logs -n fluent-bit -l app=fluent-bit -f

To check if the logs are arriving in ClickHouse you can use the following SQL commands:

SELECT count(*) FROM logs.logs;
SELECT * FROM logs.logs LIMIT 10;

SELECT count(*) FROM logs.logs_local;
SELECT * FROM logs.logs_local LIMIT 10;

To clean up all the created resources run the following commands:

kind delete cluster --name fluent-bit-clickhouse
docker stop kind-registry
docker rm kind-registry

Documentation

Overview

The fluent-bit-clickhouse Fluent Bit plugin can be used to write the logs collected by Fluent Bit to ClickHouse. It is heavily inspired by https://github.com/devcui/clickhouse-fluent-bit with some adjustments to the configuration and the table structure in ClickHouse, so that it is also possible to search through als fields of the logs. The way we are saving all the json fields of a log line is taken from the following gist: https://gist.github.com/alexey-milovidov/d6ffc9e0bc0bc72dd7bca90e76e3b83b.