kubernetes

README

Kubernetes Input Plugin

The Kubernetes plugin talks to the Kubelet API and gathers metrics about the running pods and containers for a single host. It is assumed that this plugin is running as part of a daemonset within a kubernetes installation. This means that telegraf is running on every node within the cluster. Therefore, you should configure this plugin to talk to its locally running kubelet.

Kubernetes is a fast moving project, with a new minor release every 3 months. As such, this plugin aims to maintain support only for versions that are supported by the major cloud providers, namely, 4 release over 2 years.

Host IP

To find the ip address of the host you are running on you can issue a command like the following:

curl -s $API_URL/api/v1/namespaces/$POD_NAMESPACE/pods/$HOSTNAME \
  --header "Authorization: Bearer $TOKEN" \
  --insecure | jq -r '.status.hostIP'

This example uses the downward API to pass in the $POD_NAMESPACE and $HOSTNAME is the hostname of the pod which is set by the kubernetes API. See the Kubernetes docs for a full example of generating a bearer token to explore the Kubernetes API.

Series Cardinality Warning

This plugin may produce a high number of series which, when not controlled for, will cause high load on your database. Use the following techniques to avoid cardinality issues:

• Use metric filtering options to exclude unneeded measurements and tags.
• Write to a database with an appropriate retention policy.
• Consider using the Time Series Index.
• Monitor your databases series cardinality.
• Consult the InfluxDB documentation for the most up-to-date techniques.

Global configuration options

In addition to the plugin-specific configuration settings, plugins support additional global and plugin configuration settings. These settings are used to modify metrics, tags, and field or create aliases and configure ordering, etc. See the CONFIGURATION.md for more details.

Configuration

# Read metrics from the kubernetes kubelet api
[[inputs.kubernetes]]
  ## URL for the kubelet, if empty read metrics from all nodes in the cluster
  url = "http://127.0.0.1:10255"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /var/run/secrets/kubernetes.io/serviceaccount/token
  ##
  ## To re-read the token at each interval, please use a file with the
  ## bearer_token option. If given a string, Telegraf will always use that
  ## token.
  # bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ## OR
  # bearer_token_string = "abc_123"

  ## Kubernetes Node Metric Name
  ## The default Kubernetes node metric name (i.e. kubernetes_node) is the same
  ## for the kubernetes and kube_inventory plugins. To avoid conflicts, set this
  ## option to a different value.
  # node_metric_name = "kubernetes_node"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

DaemonSet

For recommendations on running Telegraf as a DaemonSet see Monitoring Kubernetes Architecture or view the Helm charts:

• Telegraf
• InfluxDB
• Chronograf
• Kapacitor

Metrics

• kubernetes_node

  • tags:
    • node_name
  • fields:
    • cpu_usage_nanocores
    • cpu_usage_core_nanoseconds
    • memory_available_bytes
    • memory_usage_bytes
    • memory_working_set_bytes
    • memory_rss_bytes
    • memory_page_faults
    • memory_major_page_faults
    • network_rx_bytes
    • network_rx_errors
    • network_tx_bytes
    • network_tx_errors
    • fs_available_bytes
    • fs_capacity_bytes
    • fs_used_bytes
    • runtime_image_fs_available_bytes
    • runtime_image_fs_capacity_bytes
    • runtime_image_fs_used_bytes

• kubernetes_pod_container

  • tags:
    • container_name
    • namespace
    • node_name
    • pod_name
  • fields:
    • cpu_usage_nanocores
    • cpu_usage_core_nanoseconds
    • memory_usage_bytes
    • memory_working_set_bytes
    • memory_rss_bytes
    • memory_page_faults
    • memory_major_page_faults
    • rootfs_available_bytes
    • rootfs_capacity_bytes
    • rootfs_used_bytes
    • logsfs_available_bytes
    • logsfs_capacity_bytes
    • logsfs_used_bytes

• kubernetes_pod_volume

  • tags:
    • volume_name
    • namespace
    • node_name
    • pod_name
  • fields:
    • available_bytes
    • capacity_bytes
    • used_bytes

• kubernetes_pod_network

  • tags:
    • namespace
    • node_name
    • pod_name
  • fields:
    • rx_bytes
    • rx_errors
    • tx_bytes
    • tx_errors

Example Output

kubernetes_node
kubernetes_pod_container,container_name=deis-controller,namespace=deis,node_name=ip-10-0-0-0.ec2.internal,pod_name=deis-controller-3058870187-xazsr cpu_usage_core_nanoseconds=2432835i,cpu_usage_nanocores=0i,logsfs_available_bytes=121128271872i,logsfs_capacity_bytes=153567944704i,logsfs_used_bytes=20787200i,memory_major_page_faults=0i,memory_page_faults=175i,memory_rss_bytes=0i,memory_usage_bytes=0i,memory_working_set_bytes=0i,rootfs_available_bytes=121128271872i,rootfs_capacity_bytes=153567944704i,rootfs_used_bytes=1110016i 1476477530000000000
kubernetes_pod_network,namespace=deis,node_name=ip-10-0-0-0.ec2.internal,pod_name=deis-controller-3058870187-xazsr rx_bytes=120671099i,rx_errors=0i,tx_bytes=102451983i,tx_errors=0i 1476477530000000000
kubernetes_pod_volume,volume_name=default-token-f7wts,namespace=default,node_name=ip-172-17-0-1.internal,pod_name=storage-7 available_bytes=8415240192i,capacity_bytes=8415252480i,used_bytes=12288i 1546910783000000000
kubernetes_system_container

Documentation

Index

• type CPUMetrics
• type Container
• type ContainerMetrics
• type FileSystemMetrics
• type Item
• type Kubernetes
  • func (k *Kubernetes) Gather(acc telegraf.Accumulator) error
  • func (k *Kubernetes) Init() error
  • func (k *Kubernetes) LoadJSON(url string, v interface{}) error
  • func (*Kubernetes) SampleConfig() string
• type MemoryMetrics
• type Metadata
• type NetworkMetrics
• type NodeMetrics
• type PodMetrics
• type PodReference
• type Pods
• type RuntimeMetrics
• type Spec
• type SummaryMetrics
• type VolumeMetrics

Types

type CPUMetrics

type CPUMetrics struct {
	Time                 time.Time `json:"time"`
	UsageNanoCores       int64     `json:"usageNanoCores"`
	UsageCoreNanoSeconds int64     `json:"usageCoreNanoSeconds"`
}

CPUMetrics represents the cpu usage data of a pod or node

type Container

type Container struct {
	Name  string `json:"name"`
	Image string `json:"image"`
}

type ContainerMetrics

type ContainerMetrics struct {
	Name      string            `json:"name"`
	StartTime time.Time         `json:"startTime"`
	CPU       CPUMetrics        `json:"cpu"`
	Memory    MemoryMetrics     `json:"memory"`
	RootFS    FileSystemMetrics `json:"rootfs"`
	LogsFS    FileSystemMetrics `json:"logs"`
}

ContainerMetrics represents the metric data collect about a container from the kubelet

type FileSystemMetrics

type FileSystemMetrics struct {
	AvailableBytes int64 `json:"availableBytes"`
	CapacityBytes  int64 `json:"capacityBytes"`
	UsedBytes      int64 `json:"usedBytes"`
}

FileSystemMetrics represents disk usage metrics for a pod or node

type Item

type Item struct {
	Metadata Metadata `json:"metadata"`
	Spec     Spec     `json:"spec"`
}

type Kubernetes

type Kubernetes struct {
	URL               string          `toml:"url"`
	BearerToken       string          `toml:"bearer_token"`
	BearerTokenString string          `toml:"bearer_token_string" deprecated:"1.24.0;1.35.0;use 'BearerToken' with a file instead"`
	NodeMetricName    string          `toml:"node_metric_name"`
	LabelInclude      []string        `toml:"label_include"`
	LabelExclude      []string        `toml:"label_exclude"`
	ResponseTimeout   config.Duration `toml:"response_timeout"`
	Log               telegraf.Logger `toml:"-"`

	tls.ClientConfig
	// contains filtered or unexported fields
}

Kubernetes represents the config object for the plugin

func (*Kubernetes) Gather

func (k *Kubernetes) Gather(acc telegraf.Accumulator) error

Gather collects kubernetes metrics from a given URL

func (*Kubernetes) Init

func (k *Kubernetes) Init() error

func (*Kubernetes) LoadJSON

func (k *Kubernetes) LoadJSON(url string, v interface{}) error

func (*Kubernetes) SampleConfig

func (*Kubernetes) SampleConfig() string

type MemoryMetrics

type MemoryMetrics struct {
	Time            time.Time `json:"time"`
	AvailableBytes  int64     `json:"availableBytes"`
	UsageBytes      int64     `json:"usageBytes"`
	WorkingSetBytes int64     `json:"workingSetBytes"`
	RSSBytes        int64     `json:"rssBytes"`
	PageFaults      int64     `json:"pageFaults"`
	MajorPageFaults int64     `json:"majorPageFaults"`
}

MemoryMetrics represents the memory metrics for a pod or node

type Metadata

type Metadata struct {
	Name      string            `json:"name"`
	Namespace string            `json:"namespace"`
	Labels    map[string]string `json:"labels"`
}

type NetworkMetrics

type NetworkMetrics struct {
	Time     time.Time `json:"time"`
	RXBytes  int64     `json:"rxBytes"`
	RXErrors int64     `json:"rxErrors"`
	TXBytes  int64     `json:"txBytes"`
	TXErrors int64     `json:"txErrors"`
}

NetworkMetrics represents network usage data for a pod or node

type NodeMetrics

type NodeMetrics struct {
	NodeName         string             `json:"nodeName"`
	SystemContainers []ContainerMetrics `json:"systemContainers"`
	StartTime        time.Time          `json:"startTime"`
	CPU              CPUMetrics         `json:"cpu"`
	Memory           MemoryMetrics      `json:"memory"`
	Network          NetworkMetrics     `json:"network"`
	FileSystem       FileSystemMetrics  `json:"fs"`
	Runtime          RuntimeMetrics     `json:"runtime"`
}

NodeMetrics represents detailed information about a node

type PodMetrics

type PodMetrics struct {
	PodRef     PodReference       `json:"podRef"`
	StartTime  *time.Time         `json:"startTime"`
	Containers []ContainerMetrics `json:"containers"`
	Network    NetworkMetrics     `json:"network"`
	Volumes    []VolumeMetrics    `json:"volume"`
}

PodMetrics contains metric data on a given pod

type PodReference

type PodReference struct {
	Name      string `json:"name"`
	Namespace string `json:"namespace"`
}

PodReference is how a pod is identified

type Pods

type Pods struct {
	Kind       string `json:"kind"`
	APIVersion string `json:"apiVersion"`
	Items      []Item `json:"items"`
}

type RuntimeMetrics

type RuntimeMetrics struct {
	ImageFileSystem FileSystemMetrics `json:"imageFs"`
}

RuntimeMetrics contains metric data on the runtime of the system

type Spec

type Spec struct {
	Containers []Container `json:"containers"`
}

type SummaryMetrics

type SummaryMetrics struct {
	Node NodeMetrics  `json:"node"`
	Pods []PodMetrics `json:"pods"`
}

SummaryMetrics represents all the summary data about a particular node retrieved from a kubelet

type VolumeMetrics

type VolumeMetrics struct {
	Name           string `json:"name"`
	AvailableBytes int64  `json:"availableBytes"`
	CapacityBytes  int64  `json:"capacityBytes"`
	UsedBytes      int64  `json:"usedBytes"`
}

VolumeMetrics represents the disk usage data for a given volume