prometheus

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2022 License: MIT Imports: 40 Imported by: 0

README

Prometheus Input Plugin

The prometheus input plugin gathers metrics from HTTP servers exposing metrics in Prometheus format.

Configuration
# Read metrics from one or many prometheus clients
[[inputs.prometheus]]
  ## instance_id is required on input plugins
  instance_id = "prometheus"
  ## An array of urls to scrape metrics from.
  urls = ["http://localhost:9100/metrics"]
    
  ## Url tag name (tag containing scrapped url. optional, default is "url")
  # url_tag = "url"
  
  ## Whether the timestamp of the scraped metrics will be ignored.
  ## If set to true, the gather time will be used.
  # ignore_timestamp = false
  
  ## An array of Kubernetes services to scrape metrics from.
  # kubernetes_services = ["http://my-service-dns.my-namespace:9100/metrics"]
  
  ## Kubernetes config file to create client from.
  # kube_config = "/path/to/kubernetes.config"
  
  ## Scrape Kubernetes pods for the following prometheus annotations:
  ## - prometheus.io/scrape: Enable scraping for this pod
  ## - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
  ##     set this to 'https' & most likely set the tls config.
  ## - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
  ## - prometheus.io/port: If port is not 9102 use this annotation
  # monitor_kubernetes_pods = true
  
  ## Get the list of pods to scrape with either the scope of
  ## - cluster: the kubernetes watch api (default, no need to specify)
  ## - node: the local cadvisor api; for scalability. Note that the config node_ip or the environment variable NODE_IP must be set to the host IP.
  # pod_scrape_scope = "cluster"
  
  ## Only for node scrape scope: node IP of the node that circonusis running on.
  ## Either this config or the environment variable NODE_IP must be set.
  # node_ip = "10.180.1.1"
 
  ## Only for node scrape scope: interval in seconds for how often to get updated pod list for scraping.
  ## Default is 60 seconds.
  # pod_scrape_interval = 60
  
  ## Restricts Kubernetes monitoring to a single namespace
  ##   ex: monitor_kubernetes_pods_namespace = "default"
  # monitor_kubernetes_pods_namespace = ""
  ## The name of the label for the pod that is being scraped.
  ## Default is 'namespace' but this can conflict with metrics that have the label 'namespace'
  # pod_namespace_label_name = "namespace"
  # label selector to target pods which have the label
  # kubernetes_label_selector = "env=dev,app=nginx"
  # field selector to target pods
  # eg. To scrape pods on a specific node
  # kubernetes_field_selector = "spec.nodeName=$HOSTNAME"

  # cache refresh interval to set the interval for re-sync of pods list. 
  # Default is 60 minutes.
  # cache_refresh_interval = 60

  ## Scrape Services available in Consul Catalog
  # [inputs.prometheus.consul]
  #   enabled = true
  #   agent = "http://localhost:8500"
  #   query_interval = "5m"

  #   [[inputs.prometheus.consul.query]]
  #     name = "a service name"
  #     tag = "a service tag"
  #     url = 'http://{{if ne .ServiceAddress ""}}{{.ServiceAddress}}{{else}}{{.Address}}{{end}}:{{.ServicePort}}/{{with .ServiceMeta.metrics_path}}{{.}}{{else}}metrics{{end}}'
  #     [inputs.prometheus.consul.query.tags]
  #       host = "{{.Node}}"
  
  ## Use bearer token for authorization. ('bearer_token' takes priority)
  # bearer_token = "/path/to/bearer/token"
  ## OR
  # bearer_token_string = "abc_123"
  
  ## HTTP Basic Authentication username and password. ('bearer_token' and
  ## 'bearer_token_string' take priority)
  # username = ""
  # password = ""
  
  ## Specify timeout duration for slower prometheus clients (default is 3s)
  # response_timeout = "3s"
  
  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

urls can contain a unix socket as well. If a different path is required (default is /metrics for both http[s] and unix) for a unix socket, add path as a query parameter as follows: unix:///var/run/prometheus.sock?path=/custom/metrics

Kubernetes Service Discovery

URLs listed in the kubernetes_services parameter will be expanded by looking up all A records assigned to the hostname as described in Kubernetes DNS service discovery.

This method can be used to locate all Kubernetes headless services.

Kubernetes scraping

Enabling this option will allow the plugin to scrape for prometheus annotation on Kubernetes pods. Currently, you can run this plugin in your kubernetes cluster, or we use the kubeconfig file to determine where to monitor. Currently the following annotation are supported:

  • prometheus.io/scrape Enable scraping for this pod.
  • prometheus.io/scheme If the metrics endpoint is secured then you will need to set this to https & most likely set the tls config. (default 'http')
  • prometheus.io/path Override the path for the metrics endpoint on the service. (default '/metrics')
  • prometheus.io/port Used to override the port. (default 9102)

Using the monitor_kubernetes_pods_namespace option allows you to limit which pods you are scraping.

The setting pod_namespace_label_name allows you to change the label name for the namespace of the pod you are scraping. The default is namespace, but this will overwrite a label with the name namespace from a metric scraped.

Using pod_scrape_scope = "node" allows more scalable scraping for pods which will scrape pods only in the node that circonus-unified-agent is running. It will fetch the pod list locally from the node's kubelet. This will require running circonus-unified-agent in every node of the cluster. Note that either node_ip must be specified in the config or the environment variable NODE_IP must be set to the host IP. The latter can be done in the yaml of the pod running circonus-unified-agent:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

If using node level scrape scope, pod_scrape_interval specifies how often (in seconds) the pod list for scraping should updated. If not specified, the default is 60 seconds.

The pod running circonus-unified-agent will need to have the proper rbac configuration in order to be allowed to call the k8s api to discover and watch pods in the cluster. A typical configuration will create a service account, a cluster role with the appropriate rules and a cluster role binding to tie the cluster role to the service account. Example of configuration for cluster level discovery:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: circonus-k8s-role-{{.Release.Name}}
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
---
# Rolebinding for namespace to cluster-admin
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: circonus-k8s-role-{{.Release.Name}}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: circonus-k8s-role-{{.Release.Name}}
subjects:
- kind: ServiceAccount
  name: circonus-k8s-{{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: circonus-k8s-{{ .Release.Name }}
Consul Service Discovery

Enabling this option and configuring consul agent url will allow the plugin to query consul catalog for available services. Using query_interval the plugin will periodically query the consul catalog for services with name and tag and refresh the list of scraped urls. It can use the information from the catalog to build the scraped url and additional tags from a template.

Multiple consul queries can be configured, each for different service. The following example fields can be used in url or tag templates:

  • Node
  • Address
  • NodeMeta
  • ServicePort
  • ServiceAddress
  • ServiceTags
  • ServiceMeta

For full list of available fields and their type see struct CatalogService in https://github.com/hashicorp/consul/blob/master/api/catalog.go

Bearer Token

If set, the file specified by the bearer_token parameter will be read on each interval and its contents will be appended to the Bearer string in the Authorization header.

Usage for Caddy HTTP server

If you want to monitor Caddy, you need to use Caddy with its Prometheus plugin:

  • Download Caddy+Prometheus plugin here
  • Add the prometheus directive in your CaddyFile
  • Restart Caddy
  • Configure agent to fetch metrics on it:
[[inputs.prometheus]]
  ## instance_id is a required value on input plugins
  instance_id = "caddy"
  ## An array of urls to scrape metrics from.
  urls = ["http://localhost:9180/metrics"]

This is the default URL where Caddy Prometheus plugin will send data. For more details, please read the Caddy Prometheus documentation.

Metrics

Measurement names are based on the Metric Family and tags are created for each label. The value is added to a field named based on the metric type.

All metrics receive the url tag indicating the related URL specified in the agent configuration. If using Kubernetes service discovery the address tag is also added indicating the discovered ip address.

Example Output

Source

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.4545e-05
go_gc_duration_seconds{quantile="0.25"} 7.6999e-05
go_gc_duration_seconds{quantile="0.5"} 0.000277935
go_gc_duration_seconds{quantile="0.75"} 0.000706591
go_gc_duration_seconds{quantile="1"} 0.000706591
go_gc_duration_seconds_sum 0.00113607
go_gc_duration_seconds_count 4
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 15
# HELP cpu_usage_user circonus-unified-agent collected metric
# TYPE cpu_usage_user gauge
cpu_usage_user{cpu="cpu0"} 1.4112903225816156
cpu_usage_user{cpu="cpu1"} 0.702106318955865
cpu_usage_user{cpu="cpu2"} 2.0161290322588776
cpu_usage_user{cpu="cpu3"} 1.5045135406226022

Output

prometheus,quantile=1,url=http://example.org:9273/metrics go_gc_duration_seconds=0.005574303 1556075100000000000
prometheus,quantile=0.75,url=http://example.org:9273/metrics go_gc_duration_seconds=0.0001046 1556075100000000000
prometheus,quantile=0.5,url=http://example.org:9273/metrics go_gc_duration_seconds=0.0000719 1556075100000000000
prometheus,quantile=0.25,url=http://example.org:9273/metrics go_gc_duration_seconds=0.0000579 1556075100000000000
prometheus,quantile=0,url=http://example.org:9273/metrics go_gc_duration_seconds=0.0000349 1556075100000000000
prometheus,url=http://example.org:9273/metrics go_gc_duration_seconds_count=324,go_gc_duration_seconds_sum=0.091340353 1556075100000000000
prometheus,url=http://example.org:9273/metrics go_goroutines=15 1556075100000000000
prometheus,cpu=cpu0,url=http://example.org:9273/metrics cpu_usage_user=1.513622603430151 1505776751000000000
prometheus,cpu=cpu1,url=http://example.org:9273/metrics cpu_usage_user=5.829145728641773 1505776751000000000
prometheus,cpu=cpu2,url=http://example.org:9273/metrics cpu_usage_user=2.119071644805144 1505776751000000000
prometheus,cpu=cpu3,url=http://example.org:9273/metrics cpu_usage_user=1.5228426395944945 1505776751000000000

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrProtocolError = errors.New("prometheus protocol error")

Functions

func Parse 

func Parse(buf []byte, header http.Header) ([]cua.Metric, error)

Parse returns a slice of Metrics from a text representation of a metrics

func ParseV2 

func ParseV2(buf []byte, header http.Header) ([]cua.Metric, error)

Parse returns a slice of Metrics from a text representation of a metrics

Types

type ConsulConfig added in v0.0.47 

type ConsulConfig struct {
	Agent         string          `toml:"agent"`
	Queries       []*ConsulQuery  `toml:"query"`
	QueryInterval config.Duration `toml:"query_interval"`
	Enabled       bool            `toml:"enabled"`
}

Address of the Consul agent. The address must contain a hostname or an IP address and optionally a port (format: "host:port").

type ConsulQuery added in v0.0.47 

type ConsulQuery struct {
	ServiceExtraTags map[string]string `toml:"tags"` // Extra tags to add to metrics found in Consul

	ServiceName string `toml:"name"` // A name of the searched services (not ID)
	ServiceTag  string `toml:"tag"`  // A tag of the searched services
	ServiceDc   string `toml:"dc"`   // A DC of the searched services
	ServiceURL  string `toml:"url"`  // A template URL of the Prometheus gathering interface. The hostname part of the URL will be replaced by discovered address and port.
	// contains filtered or unexported fields
}

One Consul service discovery query

type Prometheus 

type Prometheus struct {
	// An array of urls to scrape metrics from.
	URLs []string `toml:"urls"`

	// An array of Kubernetes services to scrape metrics from.
	KubernetesServices []string

	// Location of kubernetes config file
	KubeConfig string

	// Label Selector/s for Kubernetes
	KubernetesLabelSelector string `toml:"kubernetes_label_selector"`

	// Field Selector/s for Kubernetes
	KubernetesFieldSelector string `toml:"kubernetes_field_selector"`

	// Consul SD configuration
	ConsulConfig ConsulConfig `toml:"consul"`

	// Bearer Token authorization file path
	BearerToken       string `toml:"bearer_token"`
	BearerTokenString string `toml:"bearer_token_string"`

	// Basic authentication credentials
	Username string `toml:"username"`
	Password string `toml:"password"`

	ResponseTimeout internal.Duration `toml:"response_timeout"`

	MetricVersion int `toml:"metric_version"`

	URLTag string `toml:"url_tag"`

	tls.ClientConfig

	Log cua.Logger

	// Should we scrape Kubernetes services for prometheus annotations
	MonitorPods           bool   `toml:"monitor_kubernetes_pods"`
	PodScrapeScope        string `toml:"pod_scrape_scope"`
	NodeIP                string `toml:"node_ip"`
	PodScrapeInterval     int    `toml:"pod_scrape_interval"`
	PodNamespace          string `toml:"monitor_kubernetes_pods_namespace"`
	PodNamespaceLabelName string `toml:"pod_namespace_label_name"`

	// Only for monitor_kubernetes_pods=true
	CacheRefreshInterval int `toml:"cache_refresh_interval"`
	// contains filtered or unexported fields
}

func (*Prometheus) AddressToURL 

func (p *Prometheus) AddressToURL(u *url.URL, address string) *url.URL

func (*Prometheus) Description 

func (p *Prometheus) Description() string

func (*Prometheus) Gather 

func (p *Prometheus) Gather(ctx context.Context, acc cua.Accumulator) error

Reads stats from all configured servers accumulates stats. Returns one of the errors encountered while gather stats (if any).

func (*Prometheus) GetAllURLs 

func (p *Prometheus) GetAllURLs() (map[string]URLAndAddress, error)

func (*Prometheus) Init 

func (p *Prometheus) Init() error

func (*Prometheus) SampleConfig 

func (p *Prometheus) SampleConfig() string

func (*Prometheus) Start 

func (p *Prometheus) Start(_, a cua.Accumulator) error

Start will start the Kubernetes scraping if enabled in the configuration

func (*Prometheus) Stop 

func (p *Prometheus) Stop()

type URLAndAddress 

type URLAndAddress struct {
	OriginalURL *url.URL
	URL         *url.URL
	Tags        map[string]string
	Address     string
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.