tagger

README

package tagger

The Tagger is the central source of truth for client-side entity tagging. It runs Collectors that detect entities and collect their tags. Tags are then stored in memory (by the TagStore) and can be queried by the tagger.Tag() method. Calling once tagger.Init() after the config package is ready is needed to enable collection.

The package methods use a common defaultTagger object, but we can create a custom Tagger object for testing.

The package will implement an IPC mechanism (a server and a client) to allow other agents to query the DefaultTagger and avoid duplicating the information in their process. Switch between local and client mode will be done via a build flag.

The tagger is also available to python checks via the tagger module exporting the get_tags() function. This function accepts the same arguments as the Go Tag() function, and returns an empty list on errors.

Collector

A Collector connects to a single information source and pushes types.TagInfo structs to a channel, towards the Tagger. It can either run in streaming mode, pull or fetchonly mode, depending of what's most efficient for the data source:

Streamer

The DockerCollector runs in stream mode as it collects events from the docker daemon and reacts to them, sending updates incrementally.

Puller

The KubernetesCollector will run in pull mode as it needs to query and filter a full entity list every time. It will only push updates to the store though, by keeping an internal state of the latest revision.

FetchOnly

The ECSCollector does not push updates to the Store by itself, but is only triggered on cache misses. As tasks don't change after creation, there's no need for periodic pulling. It is designed to run alongside DockerCollector, that will trigger deletions in the store.

TagStore

The TagStore reads types.TagInfo structs and stores them in a in-memory cache. Cache invalidation is triggered by the collectors (or source) by either:

• sending new tags for the same Entity, all the tags from this Source will be removed and replaced by the new tags
• sending a types.TagInfo with DeleteEntity set, all the tags collected for this entity by the specified source (but not others) will be deleted when prune() is called.

TagCardinality

types.TagInfo accepts and store tags that have different cardinality. TagCardinality can be:

• LowCardinality: in the host count order of magnitude
• OrchestratorCardinality: tags that change value for each pod or task
• HighCardinality: typically tags that change value for each web request, user agent, container, etc.

Entity IDs

Tagger entities are identified by a string-typed ID, with one of the following forms:

Service	Tagger Entity
workloadmeta.KindContainer	container_id://<sha>
workloadmeta.KindContainerImageMetadata	container_image_metadata://<sha>
workloadmeta.KindGardenContainer	container_id://<sha>
workloadmeta.KindKubernetesPod	kubernetes_pod_uid://<uid>
workloadmeta.KindECSTask	ecs_task://<task-id>
CloudFoundry LRP	<processGuid>/<svcName>/<instanceGuid> or <appGuid>/<svcName>
Container runtime or orchestrator	(none)
Kubernetes Endpoint	kube_endpoint_uid://<namespace>/<name>/<ip>
Kubernetes Service	kube_service://<namespace>/<name>
SNMP Config	config hash

Tagger

The Tagger handles the glue between Collectors and TagStore and the cache miss logic. If the tags from the TagStore are missing some sources, they will be manually queried in a block way, and the cache will be updated.

For convenience, the package creates a defaultTagger object that is used when calling the tagger.Tag() method.

               +-----------+
               | Collector |
               +---+-------+
                   |
                   |
+--------+      +--+-------+       +-------------+
|  User  <------+  Tagger  +-------> IPC handler |
|packages|      +--+-----^-+       +-------------+
+--------+         |     |
                   |     |
                +--v-----+-+
                | TagStore |
                +----------+

Documentation

Overview

Package tagger provides the tagger component for the Datadog Agent

Index

• Variables
• func AccumulateTagsFor(entity string, cardinality types.TagCardinality, tb tagset.TagsAccumulator) error
• func AgentTags(cardinality types.TagCardinality) ([]string, error)
• func EnrichTags(tb tagset.TagsAccumulator, originInfo taggertypes.OriginInfo)
• func GetEntity(entityID string) (*types.Entity, error)
• func GetEntityHash(entity string, cardinality types.TagCardinality) string
• func GlobalTags(cardinality types.TagCardinality) ([]string, error)
• func List() types.TaggerListResponse
• func ResetCaptureTagger()
• func SetGlobalTaggerClient(t Component)
• func SetNewCaptureTagger(newCaptureTagger Component)
• func StandardTags(entity string) ([]string, error)
• func Tag(entity string, cardinality types.TagCardinality) ([]string, error)
• func UnlockGlobalTaggerClient()
• type AgentTypeForTagger
• type Component
  • func GetTaggerInstance() Component
• type Params
  • func NewCLCRunnerRemoteTaggerParams() Params
  • func NewFakeTaggerParams() Params
  • func NewNodeRemoteTaggerParams() Params
  • func NewNodeRemoteTaggerParamsWithFallback() Params
  • func NewTaggerParams() Params
  • func NewTaggerParamsForCoreAgent(_ config.Component) Params

Variables

var (

	// ChecksCardinality defines the cardinality of tags we should send for check metrics
	// this can still be overridden when calling get_tags in python checks.
	ChecksCardinality types.TagCardinality

	// DogstatsdCardinality defines the cardinality of tags we should send for metrics from
	// dogstatsd.
	DogstatsdCardinality types.TagCardinality
)

Functions

func AccumulateTagsFor

func AccumulateTagsFor(entity string, cardinality types.TagCardinality, tb tagset.TagsAccumulator) error

AccumulateTagsFor is an interface function that queries taggerclient singleton

func AgentTags

func AgentTags(cardinality types.TagCardinality) ([]string, error)

AgentTags is an interface function that queries taggerclient singleton

func EnrichTags

func EnrichTags(tb tagset.TagsAccumulator, originInfo taggertypes.OriginInfo)

EnrichTags is an interface function that queries taggerclient singleton

func GetEntity

func GetEntity(entityID string) (*types.Entity, error)

GetEntity returns the hash for the provided entity id.

func GetEntityHash

func GetEntityHash(entity string, cardinality types.TagCardinality) string

GetEntityHash is an interface function that queries taggerclient singleton

func GlobalTags

func GlobalTags(cardinality types.TagCardinality) ([]string, error)

GlobalTags is an interface function that queries taggerclient singleton

func List

func List() types.TaggerListResponse

List the content of the defaulTagger

func ResetCaptureTagger

func ResetCaptureTagger()

ResetCaptureTagger will reset capture tagger

func SetGlobalTaggerClient

func SetGlobalTaggerClient(t Component)

SetGlobalTaggerClient sets the global taggerClient instance

func SetNewCaptureTagger

func SetNewCaptureTagger(newCaptureTagger Component)

SetNewCaptureTagger will set capture tagger in global tagger instance by using provided capture tagger

func StandardTags

func StandardTags(entity string) ([]string, error)

StandardTags is an interface function that queries taggerclient singleton

func Tag

func Tag(entity string, cardinality types.TagCardinality) ([]string, error)

Tag is an interface function that queries taggerclient singleton

func UnlockGlobalTaggerClient

func UnlockGlobalTaggerClient()

UnlockGlobalTaggerClient releases the initOnce lock on the global tagger. For testing only.

Types

type AgentTypeForTagger

type AgentTypeForTagger uint8

AgentTypeForTagger represents agent types that tagger is used for

const (
	LocalTaggerAgent AgentTypeForTagger = 1 << iota
	NodeRemoteTaggerAgent
	CLCRunnerRemoteTaggerAgent
	FakeTagger
)

Define agent type for tagger

type Component

type Component interface {
	Start(ctx context.Context) error
	Stop() error
	Tag(entity string, cardinality types.TagCardinality) ([]string, error)
	AccumulateTagsFor(entity string, cardinality types.TagCardinality, tb tagset.TagsAccumulator) error
	Standard(entity string) ([]string, error)
	List() types.TaggerListResponse
	GetEntity(entityID string) (*types.Entity, error)
	Subscribe(cardinality types.TagCardinality) chan []types.EntityEvent
	Unsubscribe(ch chan []types.EntityEvent)
	GetEntityHash(entity string, cardinality types.TagCardinality) string
	AgentTags(cardinality types.TagCardinality) ([]string, error)
	GlobalTags(cardinality types.TagCardinality) ([]string, error)
	SetNewCaptureTagger(newCaptureTagger Component)
	ResetCaptureTagger()
	EnrichTags(tb tagset.TagsAccumulator, originInfo taggertypes.OriginInfo)
}

Component is the component type.

func GetTaggerInstance

func GetTaggerInstance() Component

GetTaggerInstance returns the global Tagger instance

type Params

type Params struct {
	AgentTypeForTagger                 AgentTypeForTagger
	FallBackToLocalIfRemoteTaggerFails bool
}

Params provides the kind of agent we're instantiating workloadmeta for

func NewCLCRunnerRemoteTaggerParams

func NewCLCRunnerRemoteTaggerParams() Params

NewCLCRunnerRemoteTaggerParams creates a Params struct with the CLCRunnerRemoteTagger type

func NewFakeTaggerParams

func NewFakeTaggerParams() Params

NewFakeTaggerParams creates a Params struct with the FakeTagger type and for testing purposes

func NewNodeRemoteTaggerParams

func NewNodeRemoteTaggerParams() Params

NewNodeRemoteTaggerParams creates a Params struct with the NodeRemoteTagger type

func NewNodeRemoteTaggerParamsWithFallback

func NewNodeRemoteTaggerParamsWithFallback() Params

NewNodeRemoteTaggerParamsWithFallback creates a Params struct with the NodeRemoteTagger type and fallback to local tagger if remote tagger fails

func NewTaggerParams

func NewTaggerParams() Params

NewTaggerParams creates a Params struct with the default LocalTagger type

func NewTaggerParamsForCoreAgent

func NewTaggerParamsForCoreAgent(_ config.Component) Params

NewTaggerParamsForCoreAgent is a constructor function for creating core agent tagger params