README
FluentBit Subprocess Extension
This extension is experimental and may receive breaking changes or be removed at any time.
The fluentbit
extension facilitates running a FluentBit subprocess of the
collector. You are responsible for providing a configuration to FluentBit via the config
config option. This will be provided to FluentBit subprocess, along with a few other
config options to enhance the integration with the collector.
There are 2 typical ways to use Fluent Bit and Collector together: side-by-side and chained.
Side-by-side
With this approach Collector is responsible for traces and metrics and
Fluent Bit is responsible for logs. fluentbit
extension is used to start/stop and
provide configuration to Fluent Bit:
Chained
In this approach we use fluentbit
extension in conjunction with the fluentforward
receiver such that the FluentBit process will be configured to send to the
TCP socket opened by the fluentforward
receiver:
fluentbit
extension does not actually listen for the logs from FluentBit,
it just starts a FluentBit subprocess that will generally send to a fluentforward
receiver, which must be configured separately.
The downside of the chained approach is that log data is serialized and deserialized when going from Fluent Bit to Collector, which has performance implications. The benefit is that all logs are passed through the Collector which allows log data to be processed inside the Collector uniformly with traces and metrics (e.g. the same attributes added to all 3 signals if needed) and also allows to export logs in the formats supported by the Collector, which Fluent Bit does not necessarily support.
As of now, this extension is only targeted for Linux environments. It does not work on Windows or MacOS.
Note: if you are only collecting logs and not traces or metrics it is likely simpler to use Fluent Bit alone without Collector.
Example Config
extensions:
health_check:
fluentbit:
executable_path: /usr/src/fluent-bit/build/bin/fluent-bit
tcp_endpoint: 127.0.0.1:8006
config: |
[SERVICE]
parsers_file /usr/src/fluent-bit/conf/parsers.conf
[INPUT]
name tail
path /var/log/mylog
parser apache
receivers:
fluentforward:
endpoint: 0.0.0.0:8006
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 1s
static_configs:
- targets: ['127.0.0.1:8888']
# This will connect to the Fluent Bit subprocess's built-in HTTP
# monitoring server to grab Promtheus metrics.
- job_name: 'fluentbit'
scrape_interval: 1s
metrics_path: '/api/v1/metrics/prometheus'
static_configs:
- targets: ['127.0.0.1:2020']
service:
pipelines:
logs:
receivers: [fluentforward]
processors: []
exporters: [mylogsexporter]
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [mymetricsexporter]
extensions: [health_check, zpages, fluentbit, pprof]
Documentation
Index ¶
Constants ¶
const ( Starting procState = "starting" Running procState = "running" ShuttingDown procState = "shutting-down" Stopped procState = "stopped" Restarting procState = "restarting" Errored procState = "errored" )
Variables ¶
Functions ¶
func NewFactory ¶
func NewFactory() component.ExtensionFactory
NewFactory creates a factory for FluentBit extension.
Types ¶
type Config ¶
type Config struct { configmodels.ExtensionSettings `mapstructure:",squash"` // The TCP `host:port` to which the subprocess should send log entries. // This is required unless you are overridding `args` and providing the // output configuration yourself either in `args` or `config`. TCPEndpoint string `mapstructure:"tcp_endpoint"` // The path to the executable for FluentBit. Ideally should be an absolute // path since the CWD of the collector is not guaranteed to be stable. ExecutablePath string `mapstructure:"executable_path"` // Exec arguments to the FluentBit process. If you provide this, none of // the standard args will be set, and only these provided args will be // passed to FluentBit. The standard args will set the flush interval to 1 // second, configure the forward output with the given `tcp_endpoint` // option, enable the HTTP monitoring server in FluentBit, and set the // config file to stdin. The only required arg is `--config=/dev/stdin`, // since this extension passes the provided config to FluentBit via stdin. // If you set args manually, you will be responsible for setting the // forward output to the right port for the fluentforward receiver. See // `process.go#constructArgs` of this extension source to see the current // default args. Args []string `mapstructure:"args"` // A configuration for FluentBit. This is the text content of the config // itself, not a path to a config file. Config string `mapstructure:"config"` }
Config has the configuration for the fluentbit extension.