kafkametricsreceiver

README

Kafka Metrics Receiver

Status
Stability	beta: metrics
Distributions	contrib
Issues
Code Owners	@dmitryax

Kafka metrics receiver collects kafka metrics (brokers, topics, partitions, consumer groups) from kafka server, converting into otlp.

Prerequisites

This receiver supports Kafka versions:

• 2.X
• 3.X

Getting Started

Required settings (no defaults):

• scrapers: any combination of the following scrapers can be enabled.
  • topics
  • consumers
  • brokers

Metrics collected by the associated scraper are listed in metadata.yaml

Optional Settings (with defaults):

• cluster_alias: Alias name of the cluster. Adds kafka.cluster.alias resource attribute.
• protocol_version (default = 2.1.0): Kafka protocol version
• brokers (default = localhost:9092): the list of brokers to read from.
• resolve_canonical_bootstrap_servers_only (default = false): whether to resolve then reverse-lookup broker IPs during startup.
• topic_match (default = ^[^_].*$): regex pattern of topics to filter on metrics collection. The default filter excludes internal topics (starting with _).
• group_match (default = .*): regex pattern of consumer groups to filter on for metrics.
• client_id (default = otel-collector): consumer client id
• collection_interval (default = 1m): frequency of metric collection/scraping.
• initial_delay (default = 1s): defines how long this receiver waits before starting.
• tls: see TLS Configuration Settings for the full set of available options.
• auth (default none)
  • plain_text (Deprecated in v0.123.0: use sasl with mechanism set to PLAIN instead.)
    • username: The username to use.
    • password: The password to use
  • sasl
    • username: The username to use.
    • password: The password to use.
    • mechanism: The sasl mechanism to use (SCRAM-SHA-256, SCRAM-SHA-512, AWS_MSK_IAM, AWS_MSK_IAM_OAUTHBEARER or PLAIN)
    • aws_msk.region: AWS Region in case of AWS_MSK_IAM or AWS_MSK_IAM_OAUTHBEARER mechanism
    • aws_msk.broker_addr: MSK Broker address in case of AWS_MSK_IAM mechanism
  • tls ((Deprecated in v0.124.0: configure tls at the top level): this is an alias for tls at the top level.
  • kerberos
    • service_name: Kerberos service name
    • realm: Kerberos realm
    • use_keytab: Use of keytab instead of password, if this is true, keytab file will be used instead of password
    • username: The Kerberos username used for authenticate with KDC
    • password: The Kerberos password used for authenticate with KDC
    • config_file: Path to Kerberos configuration. i.e /etc/krb5.conf
    • keytab_file: Path to keytab file. i.e /etc/security/kafka.keytab
    • disable_fast_negotiation: Disable PA-FX-FAST negotiation (Pre-Authentication Framework - Fast). Some common Kerberos implementations do not support PA-FX-FAST negotiation. This is set to false by default.
• metadata
  • full (default = true): Whether to maintain a full set of metadata. When disabled, the client does not make the initial request to broker at the startup.
  • retry
    • max (default = 3): The number of retries to get metadata
    • backoff (default = 250ms): How long to wait between metadata retries

Examples:

1. Basic configuration with all scrapers:

receivers:
  kafkametrics:
    scrapers:
      - brokers
      - topics
      - consumers

1. Configuration with more optional settings:

For this example:

• A non-default broker is specified
• cluster alias is set to "kafka-prod"
• collection interval is 5 secs.
• Kafka protocol version is 3.0.0
• mTLS is configured

receivers:
  kafkametrics:
    cluster_alias: kafka-prod
    brokers: ["10.10.10.10:9092"]
    protocol_version: 3.0.0
    scrapers:
      - brokers
      - topics
      - consumers
    tls:
      ca_file: ca.pem
      cert_file: cert.pem
      key_file: key.pem
    collection_interval: 5s

Documentation

Index

• func NewFactory() receiver.Factory
• type Config
  • func (c *Config) Unmarshal(conf *confmap.Conf) error

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory creates kafkametrics receiver factory.

Types

type Config

type Config struct {
	scraperhelper.ControllerConfig `mapstructure:",squash"`
	configkafka.ClientConfig       `mapstructure:",squash"`

	// Alias name of the kafka cluster
	ClusterAlias string `mapstructure:"cluster_alias"`

	// TopicMatch topics to collect metrics on
	TopicMatch string `mapstructure:"topic_match"`

	// GroupMatch consumer groups to collect on
	GroupMatch string `mapstructure:"group_match"`

	// Cluster metadata refresh frequency
	// Configures the refresh frequency to update cached cluster metadata
	// Defaults to 10 minutes from Sarama library
	//
	// If Metadata.RefreshInterval is set, this will be ignored.
	//
	// Deprecated [v0.122.0]: use Metadata.RefreshInterval instead.
	RefreshFrequency time.Duration `mapstructure:"refresh_frequency"`

	// Scrapers defines which metric data points to be captured from kafka
	Scrapers []string `mapstructure:"scrapers"`

	// MetricsBuilderConfig allows customizing scraped metrics/attributes representation.
	metadata.MetricsBuilderConfig `mapstructure:",squash"`
}

Config represents user settings for kafkametrics receiver

func (*Config) Unmarshal

func (c *Config) Unmarshal(conf *confmap.Conf) error