k8slogreceiver

README

Kubernetes Log Receiver

The Kubernetes Log Receiver tails and parses logs in k8s environment.

Status
Stability	unmaintained: logs
Distributions	[]
Issues
Code coverage
Code Owners	| Seeking more code owners!
Emeritus	@h0cheung, @TylerHelmuth

There is only one mode of discovery for now, which is specified by the discovery.mode configuration option:

• daemonset-stdout: (default) Deployed as a DaemonSet, the receiver will read logs from the stdout of pods in the same node.

Two modes of discovery are planned to be supported in the future:

• daemonset-file: Deployed as a DaemonSet, the receiver will read logs from files inside pods in the same node.
• sidecar: Deployed as a sidecar container, the receiver will read logs from files.

Configuration

The following settings are common to all discovery modes:

Field	Default	Description
discovery.mode	daemonset-stdout	The mode of discovery. Only daemonset-stdout is supported now. daemonset-file and sidecar are coming soon.
extract		The rules to extract metadata from pods and containers. TODO default values.
TODO: add fields for reading files similar to filelogreceiver

When discovery.mode is not sidecar, there are additional configuration options:

Field	Default	Description
discovery.k8s_api.auth_type	serviceAccount	The authentication type of k8s api. Options are serviceAccount or kubeConfig.
discovery.host_root	/host-root	The directory which the root of host is mounted on.
discovery.runtime_apis		The runtime apis used to get log file paths. docker and cri-containerd are supported now. By default, it will try to automatically detect the cri-containerd.
discovery.node_from_env	KUBE_NODE_NAME	The environment variable name of node name.
discovery.filter	[]	The filter used to filter pods and containers. By default, all pods and containers will be collected.

Operators

Each operator performs a simple responsibility, such as parsing a timestamp or JSON. Chain together operators to process logs into a desired format.

• Every operator has a type.
• Every operator can be given a unique id. If you use the same type of operator more than once in a pipeline, you must specify an id. Otherwise, the id defaults to the value of type.
• Operators will output to the next operator in the pipeline. The last operator in the pipeline will emit from the receiver. Optionally, the output parameter can be used to specify the id of another operator to which logs will be passed directly.
• Only parsers and general purpose operators should be used.

Filters

When discovery.mode is not sidecar, the discovery.filter field can be used to filter pods and containers. The filter is a list of rules. Each rule is a map with the following fields:

Field	Description
annotations	MapFilters that filters pods by annotations.
labels	MapFilters that filters pods by labels.
env	MapFilters that filters containers by environment variables.
containers	ValueFilters that filters containers by name.
namespaces	ValueFilters that filters pods by namespace.
pods	ValueFilters that filters pods by name.

MapFilter

A MapFilter can be used to filter pods by maps, such as annotations or labels. It has the following fields:

Field	Description
op	The operation to perform. Options are: - "equals": (default) the value must be equal to the specified value. - "not-equals": the value must not be equal to the specified value. - "exists": the value must exist. - "not-exists": the value must not exist. - "matches": the value must match the specified regular expression. - "not-matches": the value must not match the specified regular expression.
key	The key of the map.
value	The value to match. Only used for "equals", "not-equals", "matches", and "not-matches" operations.

ValueFilter

A ValueFilter can be used to filter pods by string values, such as container names or namespaces. It has the following fields:

Field	Description
op	The operation to perform. Options are: - "equals": (default) the value must be equal to the specified value. - "not-equals": the value must not be equal to the specified value. - "matches": the value must match the specified regular expression. - "not-matches": the value must not match the specified regular expression.
value	The value to match.

Extract

The extract field can be used to extract fields from the log file path. It has the following fields:

Field	Description
metadata	A string slice of metadata to extract from the pods and containers.
env	A FieldExtractConfig that extracts fields from environment variables of containers.
annotations	A FieldExtractConfig that extracts fields from annotations of pods.
labels	A FieldExtractConfig that extracts fields from labels of pods.

FieldExtractConfig

A FieldExtractConfig can be used to extract fields from maps, such as annotations or labels. It has the following fields:

Field	Description
tag_name	Required. The name of the extracted attributes.
key	The key of the map (annotation, label or etc).Exactly one of key or key_regex must be specified.
key_regex	The regular expression of the key. Exactly one of key or key_regex must be specified.
regex	Optional. The regular expression to extract a submatch from the value.

Supported encodings

Key	Description
nop	No encoding validation. Treats the file as a stream of raw bytes
utf-8	UTF-8 encoding
utf-16le	UTF-16 encoding with little-endian byte order
utf-16be	UTF-16 encoding with big-endian byte order
ascii	ASCII encoding
big5	The Big5 Chinese character encoding

Other less common encodings are supported on a best-effort basis. See https://www.iana.org/assignments/character-sets/character-sets.xhtml for other encodings available.

Additional Terminology and Features

• An entry is the base representation of log data as it moves through a pipeline. All operators either create, modify, or consume entries.
• A field is used to reference values in an entry.
• A common expression syntax is used in several operators. For example, expressions can be used to filter or route entries.

Parsers with Embedded Operations

Many parsers operators can be configured to embed certain followup operations such as timestamp and severity parsing. For more information, see complex parsers.

Example - Collect logs from stdout of all containers

Receiver Configuration

receivers:
  k8slog:
    discovery:
      mode: daemonset-stdout
    operators:
      - type: recombine
        combine_field: body
        is_first_entry: body matches "^\\d{4}-\\d{2}-\\d{2}"
        max_log_size: 128kb
        source_identifier: attributes["k8s.pod.uid"]

Documentation

Index

• Constants
• func NewFactory() receiver.Factory
• type Config
  • func (c Config) Validate() error
• type ExtractConfig
• type FieldExtractConfig
• type FilterConfig
• type MapFilterConfig
• type RuntimeAPIConfig
  • func (c *RuntimeAPIConfig) Unmarshal(component *confmap.Conf) error
• type SourceConfig
  • func (c SourceConfig) Validate() error
• type ValueFilterConfig

Constants

const (
	DefaultMode        = ModeDaemonSetStdout
	DefaultHostRoot    = "/host_root"
	DefaultNodeFromEnv = "KUBE_NODE_NAME"
)

const (
	ModeDaemonSetStdout = "daemonset-stdout"
)

Functions

func NewFactory

func NewFactory() receiver.Factory

Types

type Config

type Config struct {
	Discovery SourceConfig  `mapstructure:"discovery"`
	Extract   ExtractConfig `mapstructure:"extract"`
}

Config is the configuration of a k8slog receiver

func (Config) Validate

func (c Config) Validate() error

type ExtractConfig

type ExtractConfig struct {
	// Metadata represents the list of metadata fields to extract from pod.
	// TODO: supported metadata fields and default values.
	Metadata []string `mapstructure:"metadata"`

	// Annotations represents the rules to extract from pod annotations.
	Annotations []FieldExtractConfig `mapstructure:"annotations"`

	// Labels represents the rules to extract from pod labels.
	Labels []FieldExtractConfig `mapstructure:"labels"`

	// Env represents the rules to extract from container environment variables.
	Env []FieldExtractConfig `mapstructure:"env"`
}

ExtractConfig allows specifying how to extract resource attributes from pod.

type FieldExtractConfig

type FieldExtractConfig struct {
	// TagName represents the name of the resource attribute that will be added to logs, metrics or spans.
	// When not specified, a default tag name will be used of the format:
	//   - k8s.pod.annotations.<annotation key>
	//   - k8s.pod.labels.<label key>
	//   - k8s.pod.env.<env key>
	//   - otel.env.<env key>
	// For example, if tag_name is not specified and the key is git_sha,
	// then the attribute name will be `k8s.pod.annotations.git_sha`.
	// When key_regex is present, tag_name supports back reference to both named capturing and positioned capturing.
	// For example, if your pod spec contains the following labels,
	//
	// app.kubernetes.io/component: mysql
	// app.kubernetes.io/version: 5.7.21
	//
	// and you'd like to add tags for all labels with prefix app.kubernetes.io/ and also trim the prefix,
	// then you can specify the following extraction rules:
	//
	// extract:
	//   labels:
	//     - tag_name: $$1
	//       key_regex: kubernetes.io/(.*)
	//
	// this will add the `component` and `version` tags to the spans or metrics.
	TagName string `mapstructure:"tag_name"`

	// Key represents the key (annotation, label or etc.) name. It uses exact match.
	Key string `mapstructure:"key"`
	// KeyRegex is a regular expression used to extract a Key that matches the regex.
	// Out of Key or KeyRegex, only one option is expected to be configured at a time.
	KeyRegex string `mapstructure:"key_regex"`

	// Regex is an optional field used to extract a sub-string from a complex field value.
	// The supplied regular expression must contain one named parameter with the string "value"
	// as the name. For example, if your pod spec contains the following annotation,
	//
	// kubernetes.io/change-cause: 2019-08-28T18:34:33Z APP_NAME=my-app GIT_SHA=58a1e39 CI_BUILD=4120
	//
	// and you'd like to extract the GIT_SHA and the CI_BUILD values as tags, then you must
	// specify the following two extraction rules:
	//
	// extract:
	//   annotations:
	//     - tag_name: git.sha
	//       key: kubernetes.io/change-cause
	//       regex: GIT_SHA=(?P<value>\w+)
	//     - tag_name: ci.build
	//       key: kubernetes.io/change-cause
	//       regex: JENKINS=(?P<value>[\w]+)
	//
	// this will add the `git.sha` and `ci.build` resource attributes.
	Regex string `mapstructure:"regex"`

	// From represents the source of the labels/annotations.
	// Env is always from container environment variables.
	// Supported values:
	// - pod (default): extract from pod labels/annotations.
	// May be supported in the future:
	// - namespace
	From string `mapstructure:"from"`
}

FieldExtractConfig allows specifying an extraction rule to extract a resource attribute from pod (or namespace) annotations (or labels). This is a copy of the config from the k8sattributes processor.

type FilterConfig

type FilterConfig struct {
	// Annotations represents the rules to filter containers based on pod annotations.
	Annotations []MapFilterConfig `mapstructure:"annotations"`

	// Labels represents the rules to filter containers based on pod labels.
	Labels []MapFilterConfig `mapstructure:"labels"`

	// Env represents the rules to filter containers based on pod environment variables.
	Env []MapFilterConfig `mapstructure:"env"`

	// Namespaces represents the rules to filter containers based on pod namespaces.
	Namespaces []ValueFilterConfig `mapstructure:"namespaces"`

	// Containers represents the rules to filter containers based on container names.
	Containers []ValueFilterConfig `mapstructure:"containers"`

	// Pods represents the rules to filter containers based on pod names.
	Pods []ValueFilterConfig `mapstructure:"pods"`
}

FilterConfig allows specifying how to filter containers to collect logs from. By default, all containers are collected from.

type MapFilterConfig

type MapFilterConfig struct {
	// Op represents how to compare the values.
	// Valid values are:
	// - "equals": (default) the value must be equal to the specified value.
	// - "not-equals": the value must not be equal to the specified value.
	// - "exists": the value must exist.
	// - "not-exists": the value must not exist.
	// - "matches": the value must match the specified regular expression.
	// - "not-matches": the value must not match the specified regular expression.
	Op string `mapstructure:"op"`

	// Key represents the key to compare against.
	Key string `mapstructure:"key"`

	// Value represents the value to compare against.
	// If Op is "exists" or "not-exists", this field is ignored.
	// If any of the values match, this rule is considered to match.
	Value string `mapstructure:"value"`
}

MapFilterConfig allows specifying a filter rule to filter containers based on key value pairs, such as pod annotations, labels or environment variables. Only if all the keys match, this rule is considered to match.

type RuntimeAPIConfig

type RuntimeAPIConfig struct {
	// contains filtered or unexported fields
}

func (*RuntimeAPIConfig) Unmarshal

func (c *RuntimeAPIConfig) Unmarshal(component *confmap.Conf) error

Unmarshal is the trick to unmarshal the config with different types.

type SourceConfig

type SourceConfig struct {
	// Mode represents the mode of the k8slog receiver.
	// Valid values are:
	// - "daemonset-stdout": (default) otel is deployed as a daemonset and collects logs from stdout of containers.
	//
	// Will be supported in the future:
	// - "daemonset-file": otel is deployed as a daemonset and collects logs from files inside containers.
	// - "sidecar": otel is deployed as a sidecar and collects logs from files.
	Mode string `mapstructure:"mode"`

	// NodeFromEnv represents the environment variable which contains the node name.
	NodeFromEnv string `mapstructure:"node_from_env"`

	// HostRoot represents the path which is used to mount the host's root filesystem.
	HostRoot string `mapstructure:"host_root"`

	// K8sAPI represents the configuration for the k8s API.
	K8sAPI k8sconfig.APIConfig `mapstructure:"k8s_api"`

	// RuntimeAPIs represents the configuration for the runtime APIs.
	RuntimeAPIs []RuntimeAPIConfig `mapstructure:"runtime_apis"`

	Filter []FilterConfig `mapstructure:"filter"`
}

SourceConfig allows specifying how to discover containers to collect logs from.

func (SourceConfig) Validate

func (c SourceConfig) Validate() error

type ValueFilterConfig

type ValueFilterConfig struct {
	// Op represents how to compare the value.
	// Valid values are:
	// - "equals": (default) the value must be equal to the specified value.
	// - "not-equals": the value must not be equal to the specified value.
	// - "matches": the value must match the specified regular expression.
	// - "not-matches": the value must not match the specified regular expression.
	Op string `mapstructure:"op"`

	// Value represents the value to compare against.
	Value string `mapstructure:"value"`
}

ValueFilterConfig allows specifying a filter rule to filter containers based on string values, such as pod names, namespaces, container names or pod UIDs. If any of the values match, this rule is considered to match.