kubeletstatsreceiver

README

Kubelet Stats Receiver

The Kubelet Stats Receiver pulls node, pod, container, and volume metrics from the API server on a kubelet and sends it down the metric pipeline for further processing.

Status
Stability	beta: metrics
Distributions	contrib, k8s
Issues
Code coverage
Code Owners	@dmitryax, @TylerHelmuth, @ChrsMark

Metrics

Details about the metrics produced by this receiver can be found in metadata.yaml with further documentation in documentation.md

Compatibility

Kubernetes Versions

This receiver is tested against the Kubernetes versions specified in the e2e-tests.yml workflow. These tested versions represent the officially supported Kubernetes versions for this component.

Configuration

A kubelet runs on a kubernetes node and has an API server to which this receiver connects. To configure this receiver, you have to tell it how to connect and authenticate to the API server and how often to collect data and send it to the next consumer.

Kubelet Stats Receiver supports both secure Kubelet endpoint exposed at port 10250 by default and read-only Kubelet endpoint exposed at port 10255. If auth_type set to none, the read-only endpoint will be used. The secure endpoint will be used if auth_type set to any of the following values:

• tls tells the receiver to use TLS for auth and requires that the fields ca_file, key_file, and cert_file also be set.
• serviceAccount tells this receiver to use the default service account token to authenticate to the kubelet API along with the default certificate which is signed by the cluster's root CA cert:
  • /var/run/secrets/kubernetes.io/serviceaccount/token
  • /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
• kubeConfig tells this receiver to use the kubeconfig file (KUBECONFIG env variable or ~/.kube/config) to authenticate and use API server proxy to access the kubelet API.
• initial_delay (default = 1s): defines how long this receiver waits before starting.

TLS Example

receivers:
  kubelet_stats:
    collection_interval: 20s
    initial_delay: 1s
    auth_type: "tls"
    ca_file: "/path/to/ca.crt"
    key_file: "/path/to/apiserver.key"
    cert_file: "/path/to/apiserver.crt"
    endpoint: "https://192.168.64.1:10250"
    insecure_skip_verify: true
exporters:
  file:
    path: "fileexporter.txt"
service:
  pipelines:
    metrics:
      receivers: [kubelet_stats]
      exporters: [file]

Service Account Authentication Example

Although it's possible to use kubernetes' hostNetwork feature to talk to the kubelet api from a pod, the preferred approach is to use the downward API.

Make sure the pod spec sets the node name as follows:

env:
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

Then the otel config can reference the K8S_NODE_NAME environment variable:

receivers:
  kubelet_stats:
    collection_interval: 20s
    auth_type: "serviceAccount"
    endpoint: "https://${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true
exporters:
  file:
    path: "fileexporter.txt"
service:
  pipelines:
    metrics:
      receivers: [kubelet_stats]
      exporters: [file]

Note: a missing or empty endpoint will cause the hostname on which the collector is running to be used as the endpoint. If the hostNetwork flag is set, and the collector is running in a pod, this hostname will resolve to the node's network namespace.

Custom CA

The service account client, by default, uses the CA certificate located at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt to validate the kubelet certificate. If the kubelet server uses a certificate issued by a different CA, specify the custom CA certificate path using the ca_file option.

AKS Custom CA example

This use case applies to AKS cluster, where the kubelet certificate is issued by /etc/kubernetes/certs/kubeletserver.crt

receivers:
  kubelet_stats:
    collection_interval: 20s
    auth_type: "serviceAccount"
    endpoint: "https://${env:K8S_NODE_NAME}:10250"
    ca_file: "/etc/kubernetes/certs/kubeletserver.crt"

Read Only Endpoint Example

The following config can be used to collect Kubelet metrics from read-only endpoint:

receivers:
  kubelet_stats:
    collection_interval: 20s
    auth_type: "none"
    endpoint: "http://${env:K8S_NODE_NAME}:10255"
exporters:
  file:
    path: "fileexporter.txt"
service:
  pipelines:
    metrics:
      receivers: [kubelet_stats]
      exporters: [file]

Kubeconfig example

The following config can be used to collect Kubelet metrics from read-only endpoint, proxied by the API server:

receivers:
  kubelet_stats:
    collection_interval: 20s
    auth_type: "kubeConfig"
    context: "my-context"
    insecure_skip_verify: true
    endpoint: "${env:K8S_NODE_NAME}"
exporters:
  file:
    path: "fileexporter.txt"
service:
  pipelines:
    metrics:
      receivers: [kubelet_stats]
      exporters: [file]

Note that using auth_type kubeConfig, the endpoint should only be the node name as the communication to the kubelet is proxied by the API server configured in the kubeConfig. insecure_skip_verify still applies by overriding the kubeConfig settings. If no context is specified, the current context or the default context is used.

Extra metadata labels

By default, all produced metrics get resource labels based on what kubelet /stats/summary endpoint provides. For some use cases it might be not enough. So it's possible to leverage other endpoints to fetch additional metadata entities and set them as extra labels on metric resource. Currently supported metadata include the following:

• container.id - to augment metrics with Container ID label obtained from container statuses exposed via /pods.
• k8s.volume.type - to collect volume type from the Pod spec exposed via /pods and have it as a label on volume metrics. If there's more information available from the endpoint than just volume type, those are synced as well depending on the available fields and the type of volume. For example, aws.volume.id would be synced from awsElasticBlockStore and gce.pd.name is synced for gcePersistentDisk.

[!WARNING] gce.pd.name, aws.volume.id, glusterfs.path, and glusterfs.endpoints.name resource attributes are all deprecated and will be removed soon.

If you want to have container.id label added to your metrics, use extra_metadata_labels field to enable it, for example:

receivers:
  kubelet_stats:
    collection_interval: 10s
    auth_type: "serviceAccount"
    endpoint: "${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true
    extra_metadata_labels:
      - container.id

If extra_metadata_labels is not set, no additional API calls is done to fetch extra metadata.

Collecting Additional Volume Metadata

When dealing with Persistent Volume Claims, it is possible to optionally sync metdadata from the underlying storage resource rather than just the volume claim. This is achieved by talking to the Kubernetes API. Below is an example, configuration to achieve this.

receivers:
  kubelet_stats:
    collection_interval: 10s
    auth_type: "serviceAccount"
    endpoint: "${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true
    extra_metadata_labels:
      - k8s.volume.type
    k8s_api_config:
      auth_type: serviceAccount

If k8s_api_config is set, the receiver will attempt to collect metadata from underlying storage resources for Persistent Volume Claims. For example, if a Pod is using a PVC backed by an EBS instance on AWS, the receiver would set the k8s.volume.type label to be awsElasticBlockStore rather than persistentVolumeClaim.

[!WARNING] awsElasticBlockStore resource attribute aws.volume.id is deprecated and will be removed soon

Important: When using k8s_api_config, the service account must have get permissions for persistentvolumeclaims and persistentvolumes resources. See Role-based access control for the required RBAC configuration.

Metric Groups

A list of metric groups from which metrics should be collected. By default, metrics from containers, pods and nodes will be collected. If metric_groups is set, only metrics from the listed groups will be collected. Valid groups are container, pod, node and volume. For example, if you're looking to collect only node and pod metrics from the receiver use the following configuration.

receivers:
  kubelet_stats:
    collection_interval: 10s
    auth_type: "serviceAccount"
    endpoint: "${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true
    metric_groups:
      - node
      - pod

Network metrics from all interfaces for Node and Pod

By default, k8s.[node|pod].network.* metrics are collected only for the default network interface (e.g. eth0). To enable network IO/error metrics collection from all available interfaces on Node/Pod level - you can use collect_all_network_interfaces configuration parameters. Please be aware that enabling this options will increase the amount of produced network metrics and increase network metrics cardinality, because of interface attribute. For example, if you would like to have network IO/error metrics from all network interfaces for both Pod and Node level you can use the following configuration.

receivers:
  kubelet_stats:
    collection_interval: 10s
    auth_type: "serviceAccount"
    endpoint: "${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true
    collect_all_network_interfaces:
      pod: true
      node: true

Collect k8s.{container,pod}.{cpu,memory}.node.utilization as ratio of total node's capacity

In order to calculate the k8s.container.cpu.node.utilization, k8s.pod.cpu.node.utilization, k8s.container.memory.node.utilization and k8s.pod.memory.node.utilization metrics, the information of the node's capacity must be retrieved from the k8s API. In this, the k8s_api_config needs to be set. In addition, the node name must be identified properly. The K8S_NODE_NAME env var can be set using the downward API inside the collector pod spec as follows:

env:
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

Then set node value to ${env:K8S_NODE_NAME} in the receiver's configuration:

receivers:
    kubelet_stats:
      collection_interval: 10s
      auth_type: 'serviceAccount'
      endpoint: '${env:K8S_NODE_NAME}:10250'
      node: '${env:K8S_NODE_NAME}'
      k8s_api_config:
        auth_type: serviceAccount
      metrics:
        k8s.container.cpu.node.utilization:
          enabled: true
        k8s.pod.cpu.node.utilization:
          enabled: true
        k8s.container.memory.node.utilization:
          enabled: true
        k8s.pod.memory.node.utilization:
          enabled: true

Optional parameters

The following parameters can also be specified:

• collection_interval (default = 10s): The interval at which to collect data.
• insecure_skip_verify (default = false): Whether or not to skip certificate verification.

The full list of settings exposed for this receiver are documented in config.go with detailed sample configurations in testdata/config.yaml.

Role-based access control

The Kubelet Stats Receiver needs get permissions on the nodes/stats resources. Additionally, when using extra_metadata_labels or any of the {request|limit}_utilization metrics the receiver also needs get permissions for nodes/pods resources.

When using k8s_api_config to collect detailed volume metadata from PersistentVolumeClaims (as described in Collecting Additional Volume Metadata), the receiver also needs get permissions for persistentvolumeclaims and persistentvolumes resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  - apiGroups: [""]
    resources: ["nodes/stats"]
    verbs: ["get"]

  # Only needed if you are using extra_metadata_labels or
  # are collecting the request/limit utilization metrics
  - apiGroups: [""]
    resources: ["nodes/pods"]
    verbs: ["get"]

  # Only needed if you are using k8s_api_config to collect
  # detailed volume metadata from PersistentVolumeClaims
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get"]

Documentation

Index

• func NewFactory() receiver.Factory
• type Config
  • func (cfg *Config) Unmarshal(componentParser *confmap.Conf) error
  • func (cfg *Config) Validate() error
• type NetworkInterfacesEnablerConfig

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory creates a factory for kubeletstats receiver.

Types

type Config

type Config struct {
	scraperhelper.ControllerConfig `mapstructure:",squash"`

	kube.ClientConfig       `mapstructure:",squash"`
	confignet.TCPAddrConfig `mapstructure:",squash"`

	// ExtraMetadataLabels contains list of extra metadata that should be taken from /pods endpoint
	// and put as extra labels on metrics resource.
	// No additional metadata is fetched by default, so there are no extra calls to /pods endpoint.
	// Supported values include container.id and k8s.volume.type.
	ExtraMetadataLabels []kubelet.MetadataLabel `mapstructure:"extra_metadata_labels"`

	// MetricGroupsToCollect provides a list of metrics groups to collect metrics from.
	// "container", "pod", "node" and "volume" are the only valid groups.
	MetricGroupsToCollect []kubelet.MetricGroup `mapstructure:"metric_groups"`

	// Configuration of the Kubernetes API client.
	K8sAPIConfig *k8sconfig.APIConfig `mapstructure:"k8s_api_config"`

	// NodeName is the node name to limit the discovery of nodes.
	// For example, node name can be set using the downward API inside the collector
	// pod spec as follows:
	//
	// env:
	//   - name: K8S_NODE_NAME
	//     valueFrom:
	//       fieldRef:
	//         fieldPath: spec.nodeName
	//
	// Then set this value to ${env:K8S_NODE_NAME} in the configuration.
	NodeName string `mapstructure:"node"`

	// MetricsBuilderConfig allows customizing scraped metrics/attributes representation.
	metadata.MetricsBuilderConfig `mapstructure:",squash"`

	// NetworkCollectAllInterfaces allows to enable collecting metrics from all network interfaces instead of default one
	// Can be set separately for Pod and Node network metrics
	NetworkCollectAllInterfaces NetworkInterfacesEnablerConfig `mapstructure:"collect_all_network_interfaces"`
}

func (*Config) Unmarshal

func (cfg *Config) Unmarshal(componentParser *confmap.Conf) error

func (*Config) Validate

func (cfg *Config) Validate() error

type NetworkInterfacesEnablerConfig

type NetworkInterfacesEnablerConfig struct {
	PodMetrics  bool `mapstructure:"pod"`
	NodeMetrics bool `mapstructure:"node"`
}