sumologicprocessor

README

Sumo Logic Processor

Status
Stability	beta: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@rnishtala-sumo, @chan-tim-sumo
Emeritus	@aboguszewski-sumo, @kasia-kujawa, @mat-rumian, @sumo-drosiek

The Sumo Logic processor (config name: sumologic) modifies the metadata on logs, metrics and traces sent to Sumo Logic so that the Sumo Logic apps can make full use of the ingested data.

Configuration

processors:
  sumologic:
    # Defines whether the `cloud.namespace` resource attribute should be added.
    # default = true
    add_cloud_namespace: {true,false}

    # Defines whether attributes should be translated
    # from OpenTelemetry to Sumo Logic conventions;
    # see "Attribute translation" documentation chapter from this document.
    # default = true
    translate_attributes: {true,false}

    # Specifies whether telegraf metric names should be translated to match
    # Sumo Logic conventions expected in Sumo Logic host related apps (for example
    # `procstat_num_threads` => `Proc_Threads` or `cpu_usage_irq` => `CPU_Irq`).
    # See `translate_telegraf_metrics_processor.go` for full list of translations.
    # default = true
    translate_telegraf_attributes: {true, false}

    # Specifies whether docker stats metric names should be translated to match
    # Sumo Logic conventions expected in Sumo Logic host related apps (for example
    # `container.cpu.usage.kernelmode` => `cpu_usage.usage_in_kernelmode` or `container.memory.usage.max` => `max_usage`).
    # It also translates some resource attribute names (for example `container.id` => `container.FullID`).
    # See `translate_docker_metrics_processor.go` for full list of translations.
    # default = false
    translate_docker_metrics: {true, false}

    # Specifies if attributes should be nested, basing on their keys.
    # See "Nesting attributes" documentation chapter from this document.
    nest_attributes:
      # Defines whether attributes should be nested.
      # default = false
      enabled: {true, false}

      # Defines the string used to separate key names in attributes that are to be nested.
      # default = "."
      separator: <separator>

      # Defines a list of allowed prefixes to be nested.
      # For example, if "kubernetes." is in this list, then all "kubernetes.*" attributes will be nested.
      # default = []
      include: [<prefix>]

      # Defines a list of prefixes not allowed to be nested.
      # For example, if "k8s." is in this list, then all "k8s.*" attributes will not be nested.
      # default = []
      exclude: [<prefix>]

      # If enabled, then maps that would have only one value will be squashed.
      # For example,{"k8s": {"pods": {"a": "A", "b": "B"}}}
      # will be squashed to {"k8s.pods": {"a": "A", "b": "B"}}
      # default = false
      squash_single_values: {true, false}

    # Specifies if attributes matching given pattern should be mapped to a common key.
    # See "Aggregating attributes" documentation chapter from this document.
    # default = []
    aggregate_attributes:
      - attribute: <attribute>
        prefixes: [<prefix>]
      - ...

    # Defines whether attribute should be added to record attributes.
    # Currently available attributes are `severity_number`, `severity_text`,
    # `trace_id`, `span_id`
    # all are disabled by default, enabled: false
    field_attributes:
      severity_number:
        enabled: true
        name: "loglevel"

Features

Adding cloud.namespace resource attribute

Some of the apps in Sumo Logic require the cloud.namespace attribute to be set to better understand the data coming from AWS EC2, AWS ECS and AWS Elactic Beanstalk. This attribute is similar to the standard OpenTelemetry attribute cloud.provider. In the future, the Sumo Logic apps might switch to the standard cloud.provider attribute. Before this happens, the following mapping defines the relationship between cloud.provider and cloud.namespace values:

cloud.platform	cloud.namespace
aws_ec2	aws/ec2
aws_ecs	ecs
aws_elastic_beanstalk	ElasticBeanstalk

When this processor's add_cloud_namespace setting is set to true, the processor looks for the above mentioned cloud.platform resource attribute values and if found, adds the corresponding cloud.namespace resource attribute.

If the cloud.platform resource attribute is not found or has a value that is not in the table, nothing is added.

Attribute translation

Attribute translation changes some of the attribute keys from OpenTelemetry convention to Sumo Logic convention. For example, OpenTelemetry convention for the attribute containing Kubernetes pod name is k8s.pod.name, but Sumo Logic expects it to be in attribute named pod.

If attribute with target name eg. pod already exists, translation is not being done for corresponding attribute (k8s.pod.name in this example).

This feature is turned on by default. To turn it off, set the translate_attributes configuration option to false. Note that this may cause some of Sumo Logic apps, built-in dashboards to not work correctly.

Note: the attributes are not translated for traces.

Below is a list of all attribute keys that are being translated.

Open Telemetry convention	Sumo Logic convention
cloud.account.id	AccountId
cloud.availability_zone	AvailabilityZone
cloud.platform	aws_service
cloud.region	Region
host.id	InstanceId
host.name	host
host.type	InstanceType
k8s.cluster.name	Cluster
k8s.container.name	container
k8s.daemonset.name	daemonset
k8s.deployment.name	deployment
k8s.namespace.name	namespace
k8s.node.name	node
k8s.service.name	service
k8s.pod.hostname	host
k8s.pod.name	pod
k8s.pod.uid	pod_id
k8s.replicaset.name	replicaset
k8s.statefulset.name	statefulset
service.name	service
log.file.path_resolved	_sourceName

Nesting attributes

Nesting attributes allows to change the structure of attributes (both resource level and record level attributes) basing on their keys by nesting attributes with common paths into maps. Common path is defined as a common prefix that consists of strings separated by a given separator string (by default a dot - .) and end with that separator, for example: if the separator is ., then xyz is a common prefix path for xyz.abc.qwe and xyz.foo, but sumo is not a prefix path for neither sumologic.foo.baz nor sumologic.bar.baz.

Example

The following set of attributes:

{
  "kubernetes.container_name": "xyz",
  "kubernetes.host.name": "the host",
  "kubernetes.host.address": "127.0.0.1",
  "kubernetes.namespace_name": "sumologic",
  "another_attr": "42"
}

Should be translated into such set:

{
  "kubernetes": {
    "container_name": "xyz",
    "namespace_name": "sumologic",
    "host": {
      "name": "the host",
      "address": "127.0.0.1"
    }
  },
 "another_attr": "42"
}

Allowlist and denylist

Properties include and exclude in the config allow to define list of prefixes that are allowed or not allowed to be nested.

For example, with the following config:

nest_attributes:
  enabled: true
  include: ["kubernetes.host."]
  exclude: ["kubernetes.host.naming"]

and following input:

{
"kubernetes.container_name": "xyz",
"kubernetes.host.name": "the host",
"kubernetes.host.naming_convention": "random",
"kubernetes.host.address": "127.0.0.1",
"kubernetes.namespace_name": "sumologic",
}

The output is:

{
"kubernetes.container_name": "xyz",
"kubernetes": {
  "host": {
    "name": "the host",
    "address": "127.0.0.1"
    }
  },
"kubernetes.host.naming_convention": "random",
"kubernetes.namespace_name": "sumologic",
}

Known issues

This feature has undefined behavior when in input there are various keys that will be mapped to the same nested structure. For example, given the following attributes:

{
  "a.b.c.": "d",
  "a": {
    "b": {
      "c": "e"
    }
  }
}

It is not possible to predict, what will be the result. It will have the following structure:

{
  "a": {
    "b": {
      "c": ...
    }
  }
}

However, it is not possible to know beforehand if the value under key c will be equal to d or e. Refer to GitHube issue for details and discussion.

Aggregating attributes

Aggregating attributes allows to map attributes with keys with given prefixes to a common key. For example, it is possible to map all attributes with keys with prefixes pod_ to a key pods. The names in resulting maps are the original key names with trimmed prefix.

For given input, mapping keys with prefixes pod_ to a key pods:

{
  "pod_a": "x",
  "pod_b": "y",
  "pod_c": "z"
}

The result is:

{
  "pods": {
    "a": "x",
    "b": "y",
    "c": "z"
  }
}

Severity Attribute

Some fields that log entries consist of are not displayed as fields in Sumo Logic out of the box. It's possible to convert specific attributes to fields. List of currently supported attributes is below:

severity_number severity_text span_id trace_id

In order to report one of them as field, following configuration is needed (below example for severity_number):

  field_attributes:
      severity_number:
        enabled: true
        name: "loglevel"

In this case severity_number from log record will be visible in sumo backend as a field named "loglevel".

Documentation

Index

• Constants
• func NewFactory() processor.Factory
• type Config
  • func (cfg *Config) Validate() error
• type NestingProcessor
  • func (*NestingProcessor) ConfigPropertyName() string
• type NestingProcessorConfig

Constants

const (
	SeverityNumberAttributeName = "loglevel"
	SeverityTextAttributeName   = "severitytext"
	SpanIDAttributeName         = "spanid"
	TraceIDAttributeName        = "traceid"
)

Functions

func NewFactory

func NewFactory() processor.Factory

NewFactory returns a new factory for the processor.

Types

type Config

type Config struct {
	AddCloudNamespace           bool                      `mapstructure:"add_cloud_namespace"`
	TranslateAttributes         bool                      `mapstructure:"translate_attributes"`
	TranslateTelegrafAttributes bool                      `mapstructure:"translate_telegraf_attributes"`
	NestAttributes              *NestingProcessorConfig   `mapstructure:"nest_attributes"`
	AggregateAttributes         []aggregationPair         `mapstructure:"aggregate_attributes"`
	LogFieldsAttributes         *logFieldAttributesConfig `mapstructure:"field_attributes"`
	TranslateDockerMetrics      bool                      `mapstructure:"translate_docker_metrics"`
}

func (*Config) Validate

func (cfg *Config) Validate() error

Validate config

type NestingProcessor

type NestingProcessor struct {
	// contains filtered or unexported fields
}

func (*NestingProcessor) ConfigPropertyName

func (*NestingProcessor) ConfigPropertyName() string

type NestingProcessorConfig

type NestingProcessorConfig struct {
	Separator          string   `mapstructure:"separator"`
	Enabled            bool     `mapstructure:"enabled"`
	Include            []string `mapstructure:"include"`
	Exclude            []string `mapstructure:"exclude"`
	SquashSingleValues bool     `mapstructure:"squash_single_values"`
}