azuredataexplorerexporter

README

Azure Data Explorer Exporter

Status
Stability	beta
Supported pipeline types	logs, metrics, traces
Distributions	contrib

This exporter sends metrics, logs and trace data to Azure Data Explorer.

Configuration

The following settings are required:

• cluster_uri (no default): The cluster name of the provisioned ADX cluster to ingest the data.
• application_id (no default): The client id to connect to the cluster and ingest data.
• application_key (no default): The cluster secret corresponding to the client id.
• tenant_id (no default): The tenant id where the application_id is referenced from.

The following settings can be optionally configured and have default values:

Note that the database tables are expected to be created upfront before the exporter is in operation , the definition of these are in the section Database and Table definition scripts

• db_name (default = "oteldb"): The ADX database where the tables are present to ingest the data.

• metrics_table_name (default = OTELMetrics): The target table in the database db_name that stores exported metric data.

• logs_table_name (default = OTELLogs): The target table in the database db_name that stores exported logs data.

• traces_table_name (default = OTELTraces): The target table in the database db_name that stores exported traces data.

  Optionally the following table mappings can be specified if the data needs to be mapped to a different table on ADX. This uses json table mapping that can be used to map attributes to target tables

• metrics_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.metrics_table_name

• logs_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.logs_table_name

• traces_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.traces_table_name

• ingestion_type (possible values=queued / managed, default = queued): ADX ingest can happen in managed streaming or queued modes.

Note: Streaming ingestion has to be enabled on ADX [configure the ADX cluster] in case of streaming option. Refer the query below to check if streaming is enabled

.show database <DB-Name> policy streamingingestion

An example configuration is provided as follows:

exporters:
  azuredataexplorer:
    # Kusto cluster uri
    cluster_uri: "https://CLUSTER.kusto.windows.net"
    # Client Id
    application_id: "f80da32c-108c-415c-a19e-643f461a677a"
    # The client secret for the client
    application_key: "xx-xx-xx-xx"
    # The tenant
    tenant_id: "21ff9e36-fbaa-43c8-98ba-00431ea10bc3"
    # Database for the logs
    db_name: "oteldb"
    # Metric table name
    metrics_table_name: "OTELMetrics"
    # Log table name
    logs_table_name: "OTELLogs"
    # Traces table
    traces_table_name: "OTELTraces"
    # Metric table mapping name
    metrics_table_json_mapping: "otelmetrics_mapping"
    # Log table mapping name
    logs_table_json_mapping: "otellogs_mapping"
    # Traces mapping table
    traces_table_json_mapping: "oteltraces_mapping"
    # Type of ingestion managed or queued
    ingestion_type : "managed"

Attribute mapping

This exporter maps OpenTelemetry trace, metric and log attributes to specific structures in the ADX tables. These can then be extended by usage of update policies in ADX if needed

Traces

ADX Table column	Description / OpenTelemetry attribute
TraceId	A valid trace identifier is a 16-byte array with at least one non-zero byte.
SpanId	A valid span identifier is an 8-byte array with at least one non-zero byte.
ParentId	A parent spanId, for the current span
SpanName	The span name
SpanStatus	Status of the Span.
SpanKind	SpanKind describes the relationship between the Span, its parents, and its children in a Trace
StartTime	A start timestamp
EndTime	An end timestamp
TraceAttributes	Custom metric attributes set from the application. Also contains the instrumentation scope name and version
ResourceAttributes	The resource attributes JSON map as specified in open telemetry resource semantics
Events	A list of timestamped Events
Links	A list of Links to other Spans

Metrics

ADX Table column	Description / OpenTelemetry attribute
Timestamp	The timestamp of the datapoint
MetricName	The name of the datapoint
MetricType	The type / datapoint type of the metric (e.g. Sum, Histogram, Gauge etc.)
MetricUnit	Unit of measure of the metric
MetricDescription	Description about the metric
MetricValue	The metric value measured for the datapoint
MetricAttributes	Custom metric attributes set from the application. Also contains the instrumentation scope name and version
Host	The host.name extracted from Host resource semantics. If empty , the hostname of the exporter is used
ResourceAttributes	The resource attributes JSON map as specified in open telemetry resource semantics

Logs

ADX Table column	Description / OpenTelemetry attribute
Timestamp	The timestamp of the datapoint
ObservedTimestamp	Time when the event was observed.
TraceId	Request trace id.
SpanId	Request span id.
SeverityText	The severity text (also known as log level)
SeverityNumber	Numerical value of the severity.
Body	The body of the log record.
LogsAttributes	Custom metric attributes set from the application. Also contains the instrumentation scope name and version
ResourceAttributes	The resource attributes JSON map as specified in open telemetry resource semantics

Database and Table definition scripts

The following tables need to be created in the database specified in the configuration.

.create-merge table <Logs-Table-Name> (Timestamp:datetime, ObservedTimestamp:datetime, TraceID:string, SpanID:string, SeverityText:string, SeverityNumber:int, Body:string, ResourceAttributes:dynamic, LogsAttributes:dynamic) 

.create-merge table <Metrics-Table-Name> (Timestamp:datetime, MetricName:string, MetricType:string, MetricUnit:string, MetricDescription:string, MetricValue:real, Host:string, ResourceAttributes:dynamic,MetricAttributes:dynamic) 

.create-merge table <Traces-Table-Name> (TraceID:string, SpanID:string, ParentID:string, SpanName:string, SpanStatus:string, SpanKind:string, StartTime:datetime, EndTime:datetime, ResourceAttributes:dynamic, TraceAttributes:dynamic, Events:dynamic, Links:dynamic) 

//Enable streaming ingestion( for managed streaming) for the created tables using
.alter table <Table-Name> policy streamingingestion enable

Optional configurations/Enhancements suggestions

• Using update policies , the collected data can further be processed as per application need. The following is an example where histogram metrics are exported to a histo specific table (buckets and aggregates)

.create table HistoBucketData (Timestamp: datetime, MetricName: string , MetricType: string , Value: double, LE: double, Host: string , ResourceAttributes: dynamic, MetricAttributes: dynamic )

.create function 
with ( docstring = "Histo bucket processing function", folder = "UpdatePolicyFunctions") ExtractHistoColumns()
{
    OTELMetrics
    | where MetricType == 'Histogram' and MetricName has "_bucket"
    | extend f=parse_json(MetricAttributes)
    | extend le=todouble(f.le)
    | extend M_name=replace_string(MetricName, '_bucket','')
    | project Timestamp, MetricName=M_name, MetricType, MetricValue, LE=le, Host, ResourceAttributes, MetricAttributes
}

.alter table HistoBucketData policy update 
@'[{ "IsEnabled": true, "Source": "OTELMetrics","Query": "ExtractHistoColumns()", "IsTransactional": false, "PropagateIngestionProperties": false}]'

//Below code creates a table which only contains count and sum values of Histogram metric type and attaches an update policy to it

 .create table HistoData (Timestamp: datetime, MetricName: string , MetricType: string , Count: double, Sum: double, Host: string , ResourceAttributes: dynamic, MetricAttributes: dynamic)

 .create function 
with ( docstring = "Histo sum count processing function", folder = "UpdatePolicyFunctions") ExtractHistoCountColumns()
{
   OTELMetrics
    | where MetricType =='Histogram'
    | where MetricName has "_count"
    | extend Count=MetricValue
    | extend M_name=replace_string(MetricName, '_bucket','')
    | join kind=inner (OTELMetrics
    | where MetricType =='Histogram'
    | where MetricName has "_sum"
    | project Sum = MetricValue , Timestamp)
 on Timestamp | project Timestamp, MetricName=M_name, MetricType, Count, Sum, Host, ResourceAttributes, MetricAttributes
}

.alter table HistoData policy update 
@'[{ "IsEnabled": true, "Source": "RawMetricsData","Query": "ExtractHistoCountColumns()", "IsTransactional": false, "PropagateIngestionProperties": false}]'

Documentation

Index

• func NewFactory() exporter.Factory
• type AdxLog
• type AdxMetric
• type AdxTrace
• type Config
  • func (adxCfg *Config) Validate() error
• type Event
• type Link

Functions

func NewFactory

func NewFactory() exporter.Factory

Creates a factory for the ADX Exporter

Types

type AdxLog

type AdxLog struct {
	Timestamp          string                 // The timestamp of the occurrence. Formatted into string as RFC3339Nano
	ObservedTimestamp  string                 // The timestamp of logs observed in opentelemetry collector.  Formatted into string as RFC3339Nano
	TraceID            string                 // TraceId associated to the log
	SpanID             string                 // SpanId associated to the log
	SeverityText       string                 // The severity level of the log
	SeverityNumber     int32                  // The severity number associated to the log
	Body               string                 // The body/Text of the log
	ResourceAttributes map[string]interface{} // JSON Resource attributes that can then be parsed.
	LogsAttributes     map[string]interface{} // JSON attributes that can then be parsed.
}

type AdxMetric

type AdxMetric struct {
	Timestamp string // The timestamp of the occurrence. A metric is measured at a point of time. Formatted into string as RFC3339Nano
	// Including name, the Metric object is defined by the following properties:
	MetricName        string                 // Name of the metric field
	MetricType        string                 // The data point type (e.g. Sum, Gauge, Histogram ExponentialHistogram, Summary)
	MetricUnit        string                 // The metric stream’s unit
	MetricDescription string                 // The metric stream’s description
	MetricValue       float64                // the value of the metric
	MetricAttributes  map[string]interface{} // JSON attributes that can then be parsed. Extrinsic properties
	// Additional properties
	Host               string                 // The hostname for analysis of the metric. Extracted from https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/host/
	ResourceAttributes map[string]interface{} // The originating Resource attributes. Refer https://opentelemetry.io/docs/reference/specification/resource/sdk/
}

This is derived from the specification https://opentelemetry.io/docs/reference/specification/metrics/datamodel/

type AdxTrace

type AdxTrace struct {
	TraceID            string                 // TraceID associated to the Trace
	SpanID             string                 // SpanID associated to the Trace
	ParentID           string                 // ParentID associated to the Trace
	SpanName           string                 // The SpanName of the Trace
	SpanStatus         string                 // The SpanStatus associated to the Trace
	SpanKind           string                 // The SpanKind of the Trace
	StartTime          string                 // The start time of the occurrence. Formatted into string as RFC3339Nano
	EndTime            string                 // The end time of the occurrence. Formatted into string as RFC3339Nano
	ResourceAttributes map[string]interface{} // JSON Resource attributes that can then be parsed.
	TraceAttributes    map[string]interface{} // JSON attributes that can then be parsed.
	Events             []*Event               // Array containing the events in a span
	Links              []*Link                // Array containing the link in a span
}

type Config

type Config struct {
	ClusterURI         string              `mapstructure:"cluster_uri"`
	ApplicationID      string              `mapstructure:"application_id"`
	ApplicationKey     configopaque.String `mapstructure:"application_key"`
	TenantID           string              `mapstructure:"tenant_id"`
	Database           string              `mapstructure:"db_name"`
	MetricTable        string              `mapstructure:"metrics_table_name"`
	LogTable           string              `mapstructure:"logs_table_name"`
	TraceTable         string              `mapstructure:"traces_table_name"`
	MetricTableMapping string              `mapstructure:"metrics_table_json_mapping"`
	LogTableMapping    string              `mapstructure:"logs_table_json_mapping"`
	TraceTableMapping  string              `mapstructure:"traces_table_json_mapping"`
	IngestionType      string              `mapstructure:"ingestion_type"`
}

Config defines configuration for Azure Data Explorer Exporter

func (*Config) Validate

func (adxCfg *Config) Validate() error

Validate checks if the exporter configuration is valid

type Event

type Event struct {
	EventName       string
	Timestamp       string
	EventAttributes map[string]interface{}
}

type Link

type Link struct {
	TraceID            string
	SpanID             string
	TraceState         string
	SpanLinkAttributes map[string]interface{}
}