azureblobreceiver

README

Azure Blob Receiver

Status
Stability	alpha: logs, traces
Distributions	contrib
Issues
Code coverage
Code Owners	@eedorenko, @mx-psi, @dyl10s

This receiver reads logs and trace data from Azure Blob Storage.

Each blob is expected to contain a single payload. By default the payload is decoded as OTLP/JSON, but the encoding is configurable per signal via logs.encoding and traces.encoding (see below). In addition to the built-in otlp_json and otlp_proto encodings, you may reference an encoding extension by its component ID to decode other formats.

Modes of Operation

The receiver supports two modes of operation:

1. Event Hub mode (recommended): Uses Azure Event Hub to receive notifications when new blobs are created. This is more efficient as it processes blobs immediately upon creation.

2. Polling mode: Periodically polls the blob containers for new blobs. Use this mode when Event Hub is not available or not desired. Blobs are processed and deleted every 10 seconds.

The mode is automatically selected based on whether event_hub.endpoint is configured.

Configuration

The following settings can be optionally configured:

• event_hub:
  • endpoint: (no default): Azure Event Hub endpoint triggering on the Blob Create event. If not specified, the receiver uses polling mode.
• auth (default = connection_string): Specifies the used authentication method. Supported values are connection_string, service_principal, default.
• cloud (default = "AzureCloud"): Defines which Azure Cloud to use when using the service_principal authentication method. Value is either AzureCloud or AzureUSGovernment.
• logs:
  • container_name: (default = "logs"): Name of the blob container with the logs
  • encoding: (default = "otlp_json"): Encoding of log blob payloads. Either one of the built-in values otlp_json or otlp_proto, or the ID of an encoding extension that implements plog.Unmarshaler.
• traces:
  • container_name: (default = "traces"): Name of the blob container with the traces
  • encoding: (default = "otlp_json"): Encoding of trace blob payloads. Either one of the built-in values otlp_json or otlp_proto, or the ID of an encoding extension that implements ptrace.Unmarshaler.

Authenticating using a connection string requires configuration of the following additional setting:

• connection_string: Azure Blob Storage connection key, which can be found in the Azure Blob Storage resource on the Azure Portal.

Authenticating using service principal requires configuration of the following additional settings:

• service_principal:
  • tenant_id
  • client_id
  • client_secret
• storage_account_url: Azure Storage Account url

The service principal method also requires the Storage Blob Data Contributor role on the logs and traces containers.

Example configurations

Using Event Hub mode with connection string authentication:

receivers:
  azure_blob:
    connection_string: DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=+idLkHYcL0MUWIKYHm2j4Q==;EndpointSuffix=core.windows.net
    event_hub:
      endpoint: Endpoint=sb://oteldata.servicebus.windows.net/;SharedAccessKeyName=otelhubbpollicy;SharedAccessKey=mPJVubIK5dJ6mLfZo1ucsdkLysLSQ6N7kddvsIcmoEs=;EntityPath=otellhub

Using Event Hub mode with service principal authentication:

receivers:
  azure_blob:
    auth: service_principal
    service_principal:
      tenant_id: "${tenant_id}"
      client_id: "${client_id}"
      client_secret: "${env:CLIENT_SECRET}"
    storage_account_url: https://accountName.blob.core.windows.net
    event_hub:
      endpoint: Endpoint=sb://oteldata.servicebus.windows.net/;SharedAccessKeyName=otelhubbpollicy;SharedAccessKey=mPJVubIK5dJ6mLfZo1ucsdkLysLSQ6N7kddvsIcmoEs=;EntityPath=otellhub

Using polling mode (no Event Hub):

receivers:
  azure_blob:
    connection_string: DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=+idLkHYcL0MUWIKYHm2j4Q==;EndpointSuffix=core.windows.net

Using an encoding extension to decode log blobs:

extensions:
  text_encoding:

receivers:
  azure_blob:
    connection_string: DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=+idLkHYcL0MUWIKYHm2j4Q==;EndpointSuffix=core.windows.net
    logs:
      encoding: text_encoding

Behavior

In Event Hub mode, the receiver subscribes to blob events published by Azure Blob Storage and handled by Azure Event Hub. When it receives a Blob Create event, it reads the logs or traces from the corresponding blob and deletes it after processing.

In polling mode, the receiver periodically lists all blobs in the configured containers, processes each blob, and deletes it after processing.

Documentation

Index

• Constants
• func NewFactory() receiver.Factory
• type AuthType
  • func (e *AuthType) UnmarshalText(text []byte) error
• type CloudType
  • func (e *CloudType) UnmarshalText(text []byte) error
• type Config
  • func (c Config) Validate() (err error)
• type EventHubConfig
• type LogsConfig
• type ServicePrincipalConfig
• type TracesConfig

Constants

const (
	AzureCloudType           = "AzureCloud"
	AzureGovernmentCloudType = "AzureUSGovernment"
)

const (
	// EncodingOTLPJSON denotes OTLP/JSON-encoded blob payloads.
	EncodingOTLPJSON = "otlp_json"
	// EncodingOTLPProto denotes OTLP/Protobuf-encoded blob payloads.
	EncodingOTLPProto = "otlp_proto"
)

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory returns a factory for Azure Blob receiver.

Types

type AuthType

type AuthType string

const (
	ServicePrincipalAuth AuthType = "service_principal"
	ConnectionStringAuth AuthType = "connection_string"
	DefaultAuth          AuthType = "default"
)

func (*AuthType) UnmarshalText

func (e *AuthType) UnmarshalText(text []byte) error

type CloudType

type CloudType string

func (*CloudType) UnmarshalText

func (e *CloudType) UnmarshalText(text []byte) error

type Config

type Config struct {
	// Type of authentication to use
	Authentication AuthType `mapstructure:"auth"`
	// Azure Blob Storage connection key,
	// which can be found in the Azure Blob Storage resource on the Azure Portal. (no default)
	ConnectionString string `mapstructure:"connection_string"`
	// Storage Account URL, used with Service Principal authentication
	StorageAccountURL string `mapstructure:"storage_account_url"`
	// Configuration for the Service Principal credentials
	ServicePrincipal ServicePrincipalConfig `mapstructure:"service_principal"`
	// Azure Cloud to authenticate against, used with Service Principal authentication
	Cloud CloudType `mapstructure:"cloud"`
	// Configurations of Azure Event Hub triggering on the `Blob Create` event
	EventHub EventHubConfig `mapstructure:"event_hub"`
	// Logs related configurations
	Logs LogsConfig `mapstructure:"logs"`
	// Traces related configurations
	Traces TracesConfig `mapstructure:"traces"`
	// contains filtered or unexported fields
}

func (Config) Validate

func (c Config) Validate() (err error)

Validate validates the configuration by checking for missing or invalid fields

type EventHubConfig

type EventHubConfig struct {
	// Azure Event Hub endpoint triggering on the `Blob Create` event
	EndPoint string `mapstructure:"endpoint"`
	// contains filtered or unexported fields
}

type LogsConfig

type LogsConfig struct {
	// Name of the blob container with the logs (default = "logs")
	ContainerName string `mapstructure:"container_name"`
	// Encoding of log blobs in this container (default = "otlp_json").
	// Either one of the built-in values "otlp_json" or "otlp_proto", or the
	// ID of an encoding extension that implements plog.Unmarshaler.
	Encoding string `mapstructure:"encoding"`
	// contains filtered or unexported fields
}

type ServicePrincipalConfig

type ServicePrincipalConfig struct {
	// Tenant ID, used with Service Principal authentication
	TenantID string `mapstructure:"tenant_id"`
	// Client ID, used with Service Principal authentication
	ClientID string `mapstructure:"client_id"`
	// Client secret, used with Service Principal authentication
	ClientSecret configopaque.String `mapstructure:"client_secret"`
	// contains filtered or unexported fields
}

type TracesConfig

type TracesConfig struct {
	// Name of the blob container with the traces (default = "traces")
	ContainerName string `mapstructure:"container_name"`
	// Encoding of trace blobs in this container (default = "otlp_json").
	// Either one of the built-in values "otlp_json" or "otlp_proto", or the
	// ID of an encoding extension that implements ptrace.Unmarshaler.
	Encoding string `mapstructure:"encoding"`
	// contains filtered or unexported fields
}