awss3exporter

README

AWS S3 Exporter

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code coverage
Code Owners	@atoulme, @pdelewski, @Erog38

Schema supported

This exporter targets to support proto/json format.

Exporter Configuration

The following exporter configuration parameters are supported.

Name	Description	Default
region	AWS region.	"us-east-1"
s3_bucket	S3 bucket
s3_base_prefix	root prefix for the S3 key applied to all files.
s3_prefix	prefix for the S3 key that can be overridden dynamically by resource_attrs_to_s3 parameter.
s3_partition_format	filepath formatting for the partition; See strftime for format specification.	"year=%Y/month=%m/day=%d/hour=%H/minute=%M"
s3_partition_timezone	timezone used to format partition	Local
role_arn	the Role ARN to be assumed
file_prefix	file prefix defined by user
marshaler	marshaler used to produce output data	otlp_json
encoding	Encoding extension to use to marshal data. Overrides the marshaler configuration option if set.
encoding_file_extension	file format extension suffix when using the encoding configuration option. May be left empty for no suffix to be appended.
endpoint	(REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from region and s3_bucket
storage_class	S3 storageclass	STANDARD
acl	S3 Object Canned ACL	none (does not set by default)
s3_force_path_style	set this to true to force the request to use path-style addressing	false
disable_ssl	set this to true to disable SSL when sending requests	false
compression	should the file be compressed	none
sending_queue	exporters common queuing	disabled
timeout	exporters common timeout	5s
resource_attrs_to_s3	determines the mapping of S3 configuration values to resource attribute values for uploading operations.
retry_mode	The retryer implementation, the supported values are "standard", "adaptive" and "nop". "nop" will set the retryer as aws.NopRetryer, which effectively disable the retry.	standard
retry_max_attempts	The max number of attempts for retrying a request if the retry_mode is set. Setting max attempts to 0 will allow the SDK to retry all retryable errors until the request succeeds, or a non-retryable error is returned.	3
retry_max_backoff	the max backoff delay that can occur before retrying a request if retry_mode is set	20s
unique_key_func_name	Name of the function to use for generating a unique portion of the key name, defaults to a random integer. Only supported value is uuidv7.
retry_on_failure	see Retry on Failure for the full set of available options.

Marshaler

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

• otlp_json (default): the OpenTelemetry Protocol format, represented as json.

• otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.

• sumo_ic: the Sumo Logic Installed Collector Archive format.

  • _sourceCategory, _sourceHost, and _sourceName is needed

    resource/add_source_category:
      attributes:
      - action: insert
        key: _sourceCategory
        value: "value"
      - action: insert
        key: _sourceHost
        value: "value"
      - action: insert
        key: _sourceName
        value: "value"
    

  This format is supported only for logs.

• body: export the log body as string. This format is supported only for logs.

Encoding

Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.

See https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding.

Compression

• none (default): No compression will be applied
• gzip: Files will be compressed with gzip.
• zstd: Files will be compressed with zstd.

resource_attrs_to_s3

• s3_bucket: Defines which resource attribute's value should be used as the S3 bucket. When this option is set, it dynamically overrides s3uploader/s3_bucket. If the specified resource attribute exists in the data,
  its value will be used as the bucket; otherwise, s3uploader/s3_bucket will serve as the fallback.
• s3_prefix: Defines which resource attribute's value should be used as the S3 prefix. When this option is set, it dynamically overrides s3uploader/s3_prefix. If the specified resource attribute exists in the data,
  its value will be used as the prefix; otherwise, s3uploader/s3_prefix will serve as the fallback.

Example Configurations

Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'

    # Optional (disabled by default)
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 100

    # Optional (5s by default)
    timeout: 20s      

Logs and traces will be stored inside 'databucket' in the following path format.

metric/year=YYYY/month=MM/day=DD/hour=HH/minute=mm

Partition Formatting

By setting the s3_partition_format option, users can specify the file path for their logs. See the strftime reference for more formatting options.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'

In this case, logs and traces would be stored in the following path format.

metric/YYYY/MM/DD/HH/mm

Optionally along with s3_partition_format you can provide s3_partition_timezone as name from IANA Time Zone database to change default local timezone to custom, for example UTC or Europe/London.

Base Path Configuration

The s3_base_prefix option allows you to specify a root path inside the bucket that is not overridden by resource_attrs_to_s3. If provided, s3_prefix will be appended to this base path.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_base_prefix: 'environment/prod'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'

In this case, logs and traces would be stored in the following path format.

environment/prod/metric/YYYY/MM/DD/HH/mm

Data routing based on resource attributes

When resource_attrs_to_s3/s3_bucket or resource_attrs_to_s3/s3_prefix is configured, the S3 bucket and/or prefix are dynamically derived from specified resource attributes in your data. If the attribute values are unavailable, the bucket and prefix will fall back to the values defined in s3uploader/s3_bucket and s3uploader/s3_prefix respectively.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'
    resource_attrs_to_s3:
      s3_bucket: "com.awss3.bucket"
      s3_prefix: "com.awss3.prefix"

In this case, metrics, logs and traces would be stored in the following path format examples:

bucket1/prefix1/YYYY/MM/DD/HH/mm
bucket2/foo-prefix/YYYY/MM/DD/HH/mm
bucket3/prefix-bar/YYYY/MM/DD/HH/mm
databucket/metric/YYYY/MM/DD/HH/mm
...

Base Path with Resource Attributes

When using both s3_base_prefix and resource_attrs_to_s3/s3_prefix, the s3_base_prefix is always used while s3_prefix can be dynamically overridden by resource attributes.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_base_prefix: 'environment/prod'
      s3_prefix: 'default-metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'
    resource_attrs_to_s3:
      s3_prefix: "com.awss3.prefix"

In this configuration:

• Base Prefix: environment/prod (always included)
• Prefix: Dynamically set from resource attribute com.awss3.prefix if available, otherwise falls back to default-metric

Path format examples:

# When resource attribute com.awss3.prefix = "service-a/metrics"
environment/prod/service-a/metrics/YYYY/MM/DD/HH/mm

# When resource attribute com.awss3.prefix = "service-b/logs"  
environment/prod/service-b/logs/YYYY/MM/DD/HH/mm

# When resource attribute is unavailable (fallback)
environment/prod/default-metric/YYYY/MM/DD/HH/mm

This allows you to maintain consistent organizational structure (via base path) while dynamically routing different data types or services to specific subdirectories.

Retry

Standard is the default retryer implementation used by service clients. See the retry package documentation for details on what errors are considered as retryable by the standard retryer implementation.

See also the aws-sdk-go reference for more information.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      retry_mode: "standard"
      retry_max_attempts: 5
      retry_max_backoff: "30s"

AWS Credential Configuration

This exporter follows default credential resolution for the aws-sdk-go.

Follow the guidelines for the credential configuration.

OpenTelemetry Collector Helm Chart for Kubernetes

For example, when using OpenTelemetry Collector Helm Chart you could use extraEnvs in the values.yaml.

extraEnvs:
- name: AWS_ACCESS_KEY_ID
  value: "< YOUR AWS ACCESS KEY >"
- name: AWS_SECRET_ACCESS_KEY
  value: "< YOUR AWS SECRET ACCESS KEY >"

Documentation

Overview

Package awss3exporter stores OpenTelemetry data as an AWS S3 exporter.

Index

• Constants
• Variables
• func NewFactory() exporter.Factory
• type Config
  • func (c *Config) Validate() error
• type MarshalerType
• type ResourceAttrsToS3
• type S3UploaderConfig

Constants

const (
	DefaultRetryMode        = "standard"
	DefaultRetryMaxAttempts = 3
	DefaultRetryMaxBackoff  = 20 * time.Second
)

const (
	SourceCategoryKey = "_sourceCategory"
	SourceHostKey     = "_sourceHost"
	SourceNameKey     = "_sourceName"
)

Variables

var ErrUnknownMarshaler = errors.New("unknown marshaler")

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for S3 exporter.

Types

type Config

type Config struct {
	configretry.BackOffConfig `mapstructure:"retry_on_failure"`

	QueueSettings   configoptional.Optional[exporterhelper.QueueBatchConfig] `mapstructure:"sending_queue"`
	TimeoutSettings exporterhelper.TimeoutConfig                             `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
	S3Uploader      S3UploaderConfig                                         `mapstructure:"s3uploader"`
	MarshalerName   MarshalerType                                            `mapstructure:"marshaler"`

	// Encoding to apply. If present, overrides the marshaler configuration option.
	Encoding              *component.ID     `mapstructure:"encoding"`
	EncodingFileExtension string            `mapstructure:"encoding_file_extension"`
	ResourceAttrsToS3     ResourceAttrsToS3 `mapstructure:"resource_attrs_to_s3"`
}

Config contains the main configuration options for the s3 exporter

func (*Config) Validate

func (c *Config) Validate() error

type MarshalerType

type MarshalerType string

const (
	OtlpProtobuf MarshalerType = "otlp_proto"
	OtlpJSON     MarshalerType = "otlp_json"
	SumoIC       MarshalerType = "sumo_ic"
	Body         MarshalerType = "body"
)

type ResourceAttrsToS3

type ResourceAttrsToS3 struct {
	// S3Bucket indicates the mapping of the bucket name used for uploading to a specific resource attribute value.
	S3Bucket string `mapstructure:"s3_bucket"`
	// S3Prefix indicates the mapping of the key (directory) prefix used for writing into the bucket to a specific resource attribute value.
	S3Prefix string `mapstructure:"s3_prefix"`
	// contains filtered or unexported fields
}

ResourceAttrsToS3 defines the mapping of S3 uploading configuration values to resource attribute values.

type S3UploaderConfig

type S3UploaderConfig struct {
	Region string `mapstructure:"region"`
	// S3Bucket is the bucket name to be uploaded to.
	S3Bucket string `mapstructure:"s3_bucket"`
	// S3BasePrefix is the root key (directory) prefix used to write the file.
	S3BasePrefix string `mapstructure:"s3_base_prefix"`
	// S3Prefix is the key (directory) prefix to write to inside the bucket. Appended to S3BasePrefix if provided.
	S3Prefix string `mapstructure:"s3_prefix"`
	// S3PartitionFormat is used to provide the rollup on how data is written. Uses [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) formatting.
	S3PartitionFormat string `mapstructure:"s3_partition_format"`
	// S3PartitionTimezone is used to provide timezone for partition time. Defaults to Local timezone.
	S3PartitionTimezone string `mapstructure:"s3_partition_timezone"`
	// FilePrefix is the filename prefix used for the file to avoid any potential collisions.
	FilePrefix string `mapstructure:"file_prefix"`
	// Endpoint is the URL used for communicated with S3.
	Endpoint string `mapstructure:"endpoint"`
	// RoleArn is the role policy to use when interacting with S3
	RoleArn string `mapstructure:"role_arn"`
	// S3ForcePathStyle sets the value for force path style.
	S3ForcePathStyle bool `mapstructure:"s3_force_path_style"`
	// DisableSLL forces communication to happen via HTTP instead of HTTPS.
	DisableSSL bool `mapstructure:"disable_ssl"`
	// ACL is the canned ACL to use when uploading objects.
	ACL string `mapstructure:"acl"`

	StorageClass string `mapstructure:"storage_class"`
	// Compression sets the algorithm used to process the payload
	// before uploading to S3.
	// Valid values are: `gzip`, `zstd`, or no value set.
	Compression configcompression.Type `mapstructure:"compression"`

	// RetryMode specifies the retry mode for S3 client, default is "standard".
	// Valid values are: "standard", "adaptive", or "nop".
	// "nop" will disable retry by setting the retryer to aws.NopRetryer.
	RetryMode string `mapstructure:"retry_mode"`
	// RetryMaxAttempts specifies the maximum number of attempts for S3 client.
	// Default is 3 (SDK default).
	RetryMaxAttempts int `mapstructure:"retry_max_attempts"`
	// RetryMaxBackoff specifies the maximum backoff delay for S3 client.
	// Default is 20 seconds (SDK default).
	RetryMaxBackoff time.Duration `mapstructure:"retry_max_backoff"`

	// UniqueKeyFuncName specifies a function to use for generating a unique string as part of the S3 key.
	// If unspecified, a default function will be used that generates a random string.
	// Valid values are: "uuidv7"
	UniqueKeyFuncName string `mapstructure:"unique_key_func_name"`
}

S3UploaderConfig contains aws s3 uploader related config to controls things like bucket, prefix, batching, connections, retries, etc.