azureblobexporter

README

Azure Blob Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code coverage
Code Owners	@hgaol, @MovieStoreGuy

Configuration

The following settings are required:

• url: Must be specified if auth type is not connection_string. If auth type is connection_string, it's optional or will be override by the auth.connection_string. Azure storage account endpoint. This setting might be replaced with endpoint for future. e.g. https://.blob.core.windows.net/
• auth (no default): Authentication method for exporter to ingest data.
  • type (no default): Authentication type for exporter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and workload_identity.
  • tenand_id: Tenand Id for the client, only needed when type is service_principal and workload_identity.
  • client_id: Client Id for the auth, only needed when type is service_principal, user_managed_identity and workload_identity.
  • client_secret: Secret for the client, only needed when type is service_principal.
  • connection_string: Connection string to the endpoint. Only needed for connection_string auth type. Once provided, it'll override the url parameter to the storage account.
  • federated_token_file: The path of the projected service account token file, only needed when type is workload_identity.

The following settings can be optionally configured and have default values:

• container: container for metrics, logs and traces. A container organizes a set of blobs, similar to a directory in a file system. More details can refer this.
  • metrics (default metrics): container to store metrics. default value is metrics.
  • logs (default logs): container to store logs. default value is logs.
  • traces (default traces): container to store traces. default value is traces.
• blob_name_format: the final blob name will be blob_name
  • template_enabled (default false): enables Go template parsing for blob name formats. If parsing fails, it will not throw an error but will log a warning and continue formatting the blob name using other rules.
  • metrics_format (default 2006/01/02/metrics_15_04_05.json): blob name format. The date format follows constants in Golang, refer here.
  • logs_format (default 2006/01/02/logs_15_04_05.json): blob name format.
  • traces_format (default 2006/01/02/traces_15_04_05.json): blob name format.
  • serial_num_range (default 10000): a range of random number to be appended after blob_name. e.g. blob_name_{serial_num}.
  • serial_num_before_extension (default false): places the serial number before the file extension if there is one. e.g blob_name_{serial_num}.json instead of blob_name.json_{serial_num}
• format (default json): json or proto. which present otel json or otel protobuf format, the file extension will be json or pb.
• encodings (default using encoding specified in format, which is json): if specified, uses the encoding extension to encode telemetry data. Overrides format.
  • logs (default nil): encoding component id.
  • metrics (default nil): encoding component id.
  • traces (default nil): encoding component id.
• append_blob: configures append blob behavior. When enabled, telemetry data is appended to a single blob instead of creating new blobs. This can be useful for aggregating data or reducing the number of blobs created.
  • enabled (default false): determines whether to use append blob mode.
  • separator (default \n): string to insert between appended data blocks.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false

Blob Name Templates

When template_enabled is true, you can use Go templates in metrics_format, logs_format, and traces_format to create dynamic blob names based on telemetry data. The root object for the template is the telemetry data itself (pmetric.Metrics, plog.Logs, or ptrace.Traces).

The following template functions are available:

Function	Description	Example
getResourceMetricAttr	Gets a resource attribute from metrics data.	{{ getResourceMetricAttr . 0 "service.name" }}
getResourceLogAttr	Gets a resource attribute from logs data.	{{ getResourceLogAttr . 0 "service.name" }}
getResourceSpanAttr	Gets a resource attribute from traces data.	{{ getResourceSpanAttr . 0 "service.name" }}
getScopeMetricAttr	Gets a scope attribute from metrics data.	{{ getScopeMetricAttr . 0 0 "scope.name" }}
getScopeLogAttr	Gets a scope attribute from logs data.	{{ getScopeLogAttr . 0 0 "scope.name" }}
getScopeSpanAttr	Gets a scope attribute from traces data.	{{ getScopeSpanAttr . 0 0 "scope.name" }}
getMetric	Gets a metric object. You can chain to access its fields.	{{ (getMetric . 0 0 0).Name }}
getLogRecord	Gets a log record object. You can chain to access its fields.	{{ (getLogRecord . 0 0 0).TraceID }}
getSpan	Gets a span object. You can chain to access its fields.	{{ (getSpan . 0 0 0).Name }}

An example configuration is provided as follows:

extensions:
  zpages:
    endpoint: localhost:55679
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"

exporter:
  azureblob/1:
    url: "https://<your-account>.blob.core.windows.net/"
    container:
      logs: "logs"
      metrics: "metrics"
      traces: "traces"
    blob_name_format:
      template_enabled: true
      metrics_format: `{{ getResourceMetricAttr . 0 "service.name" }}/2006/01/02/metrics.json`
      logs_format: `{{ getScopeLogAttr . 0 0 "scope.name" }}/2006/01/02/logs.json`
      traces_format: `{{ (getSpan . 0 0 0).Name }}/2006/01/02/traces.json`
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=<your-acount>;AccountKey=<account-key>;EndpointSuffix=core.windows.net"
    encodings:
      logs: text_encoding
    append_blob:
      enabled: true
      separator: "\n"

Append Blob

When append_blob is enabled:

• The exporter will create append blobs instead of block blobs
• New data will be appended to existing blobs rather than creating new ones
• The configured separator will be inserted between data blocks
• If the blob doesn't exist, it will be created automatically

Documentation

Overview

Package azureblobexporter stores OpenTelemetry data as an Azure Storage Blob exporter.

Index

• func NewFactory() exporter.Factory
• type AppendBlob
• type AuthType
• type Authentication
• type BlobNameFormat
• type Config
  • func (c *Config) Validate() error
• type Encodings
• type TelemetryConfig

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for Azure Blob exporter.

Types

type AppendBlob

type AppendBlob struct {
	Enabled   bool   `mapstructure:"enabled"`
	Separator string `mapstructure:"separator"`
}

type AuthType

type AuthType string

const (
	ConnectionString      AuthType = "connection_string"
	SystemManagedIdentity AuthType = "system_managed_identity"
	UserManagedIdentity   AuthType = "user_managed_identity"
	ServicePrincipal      AuthType = "service_principal"
	WorkloadIdentity      AuthType = "workload_identity"
)

type Authentication

type Authentication struct {
	// Type is the authentication type. supported values are connection_string, service_principal, system_managed_identity and user_managed_identity
	Type AuthType `mapstructure:"type"`

	// TenantID is the tenand id for the AAD App. It's only needed when type is service principal.
	TenantID string `mapstructure:"tenant_id"`

	// ClientID is the AAD Application client id. It's needed when type is service principal or user managed identity
	ClientID string `mapstructure:"client_id"`

	ClientSecret string `mapstructure:"client_secret"`

	// ConnectionString to the endpoint.
	ConnectionString string `mapstructure:"connection_string"`

	// FederatedTokenFile is the path to the file containing the federated token. It's needed when type is workload_identity.
	FederatedTokenFile string `mapstructure:"federated_token_file"`
}

type BlobNameFormat

type BlobNameFormat struct {
	MetricsFormat            string            `mapstructure:"metrics_format"`
	LogsFormat               string            `mapstructure:"logs_format"`
	TracesFormat             string            `mapstructure:"traces_format"`
	SerialNumRange           int64             `mapstructure:"serial_num_range"`
	SerialNumBeforeExtension bool              `mapstructure:"serial_num_before_extension"`
	TemplateEnabled          bool              `mapstructure:"template_enabled"`
	Params                   map[string]string `mapstructure:"params"`
}

type Config

type Config struct {
	// URL is the endpoint to the azure storage account. This is only required until there is an azure auth extension in the future.
	URL string `mapstructure:"url"`

	// A container organizes a set of blobs, similar to a directory in a file system.
	Container TelemetryConfig `mapstructure:"container"`
	Auth      Authentication  `mapstructure:"auth"`

	// BlobNameFormat is the format of the blob name. It controls the uploaded blob name, e.g. "2006/01/02/metrics_15_04_05.json"
	BlobNameFormat BlobNameFormat `mapstructure:"blob_name_format"`

	// FormatType is the format of encoded telemetry data. Supported values are json and proto.
	FormatType string `mapstructure:"format"`

	// AppendBlob configures append blob behavior
	AppendBlob AppendBlob `mapstructure:"append_blob"`

	// Encoding extension to apply for logs/metrics/traces. If present, overrides the marshaler configuration option and format.
	Encodings Encodings `mapstructure:"encodings"`

	configretry.BackOffConfig `mapstructure:"retry_on_failure"`
}

Config contains the main configuration options for the azure storage blob exporter

func (*Config) Validate

func (c *Config) Validate() error

type Encodings

type Encodings struct {
	Logs    *component.ID `mapstructure:"logs"`
	Metrics *component.ID `mapstructure:"metrics"`
	Traces  *component.ID `mapstructure:"traces"`
}

type TelemetryConfig

type TelemetryConfig struct {
	Logs    string `mapstructure:"logs"`
	Metrics string `mapstructure:"metrics"`
	Traces  string `mapstructure:"traces"`
}