fluentbitextension

README

FluentBit Subprocess Extension

This extension is experimental and may receive breaking changes or be removed at any time.

The fluentbit extension facilitates running a FluentBit subprocess of the collector. This is meant to be used in conjunction with the fluentforward receiver such that the FluentBit subprocess will be configured to send to the TCP socket opened by the fluentforward receiver. This extension does not actually listen for the logs from FluentBit, it just starts a FluentBit subprocess that will generally send to a fluentforward receiver, which must be configured separately.

You are responsible for providing a configuration to FluentBit via the config config option. This will be provided to the subprocess, along with a few other config options to enhance the integration with the collector.

As of now, this extension is only targeted for Linux environments. It does not work on Windows or MacOS.

Example Config

extensions:
  health_check:
  fluentbit:
    executable_path: /usr/src/fluent-bit/build/bin/fluent-bit
    tcp_endpoint: 127.0.0.1:8006
    config: |
      [SERVICE]
          parsers_file /usr/src/fluent-bit/conf/parsers.conf
      [INPUT]
          name tail
          path /var/log/mylog
          parser apache
receivers:
  fluentforward:
    endpoint: 0.0.0.0:8006
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 1s
          static_configs:
            - targets: ['127.0.0.1:8888']
        # This will connect to the Fluent Bit subprocess's built-in HTTP
        # monitoring server to grab Promtheus metrics.
        - job_name: 'fluentbit'
          scrape_interval: 1s
          metrics_path: '/api/v1/metrics/prometheus'
          static_configs:
            - targets: ['127.0.0.1:2020']
service:
  pipelines:
    logs:
      receivers: [fluentforward]
      processors: []
      exporters: [mylogsexporter]
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [mymetricsexporter]
  extensions: [health_check, zpages, fluentbit, pprof]

Documentation

Index

• Constants
• func NewFactory() component.ExtensionFactory
• type Config

Constants

const (
	Starting     procState = "starting"
	Running      procState = "running"
	ShuttingDown procState = "shutting-down"
	Stopped      procState = "stopped"
	Restarting   procState = "restarting"
	Errored      procState = "errored"
)

Functions

func NewFactory

func NewFactory() component.ExtensionFactory

NewFactory creates a factory for FluentBit extension.

Types

type Config

type Config struct {
	configmodels.ExtensionSettings `mapstructure:",squash"`

	// The TCP `host:port` to which the subprocess should send log entries.
	// This is required unless you are overridding `args` and providing the
	// output configuration yourself either in `args` or `config`.
	TCPEndpoint string `mapstructure:"tcp_endpoint"`

	// The path to the executable for FluentBit. Ideally should be an absolute
	// path since the CWD of the collector is not guaranteed to be stable.
	ExecutablePath string `mapstructure:"executable_path"`

	// Exec arguments to the FluentBit process.  If you provide this, none of
	// the standard args will be set, and only these provided args will be
	// passed to FluentBit.  The standard args will set the flush interval to 1
	// second, configure the forward output with the given `tcp_endpoint`
	// option, enable the HTTP monitoring server in FluentBit, and set the
	// config file to stdin. The only required arg is `--config=/dev/stdin`,
	// since this extension passes the provided config to FluentBit via stdin.
	// If you set args manually, you will be responsible for setting the
	// forward output to the right port for the fluentforward receiver. See
	// `process.go#constructArgs` of this extension source to see the current
	// default args.
	Args []string `mapstructure:"args"`

	// A configuration for FluentBit.  This is the text content of the config
	// itself, not a path to a config file.
	Config string `mapstructure:"config"`
}

Config has the configuration for the fluentbit extension.