awslogsencodingextension

package module

v0.150.0 Latest Latest Go to latest Published: Apr 13, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

README ¶

AWS Logs Encoding Extension

This extension unmarshalls logs encoded in formats produced by AWS services.

Status
Stability	alpha
Distributions	contrib
Issues
Code Owners	@axw, @constanca-m, @Kavindu-Dodan

This extension unmarshals logs encoded in formats produced by AWS services, including:

Amazon CloudWatch Logs Subscription Filters.
VPC flow log records sent to S3 in plain text.
- Parquet support still to be added.
S3 access log records.
AWS CloudTrail logs.
ELB access logs:
AWS Network Firewall logs.
(More to be added later.)

Example for Amazon CloudWatch Logs Subscription Filters:

extensions:
  awslogs_encoding/cloudwatch:
    format: cloudwatch

receivers:
  awsfirehose:
    endpoint: :1234
    encoding: awslogs_encoding/cloudwatch

Example for VPC flow logs:

extensions:
  awslogs_encoding/vpcflow:
    format: vpcflow
    vpcflow:
      # options [parquet, plain-text]. 
      # parquet option still needs to be implemented.
      file_format: plain-text 
      # Optional: format of the VPC flow log. Used when processing VPC flow logs arriving through CloudWatch Logs subscription filters. 
      # Ignored when decoding VPC flow logs sent to S3, which include the format as a file header.
      # Accepts a space delimited list of fields in the VPC flow log.
      # When unset, built-in default is used matching fields of Version 2 VPC flow logs format.
      format: version interface-id srcaddr dstaddr

Example for S3 access logs:

extensions:
  awslogs_encoding/s3access:
    format: s3access

Example for CloudTrail logs:

extensions:
  awslogs_encoding/cloudtrail:
    format: cloudtrail

Example for ELB access logs:

extensions:
  awslogs_encoding/elbaccess:
    format: elbaccess

Example for AWS Network Firewall logs:

extensions:
  awslogs_encoding/networkfirewall:
    format: networkfirewall

Log Format Identification

All logs processed by this extension are automatically tagged with an encoding.format attribute at the scope level to identify the source format. This allows you to easily filter and route logs based on their AWS service origin.

The pattern used is aws.<format_name>.

Examples:

VPC Flow Logs: encoding.format:"aws.vpcflow"
ELB Access Logs: encoding.format:"aws.elbaccess"

Format Values

The following format values are supported in the awslogsencodingextension to identify different AWS log types:

AWS Log Type	Format Value	Description
VPC Flow Logs	`vpcflow`	Virtual Private Cloud flow log records
ELB Access Logs	`elbaccess`	Elastic Load Balancer access logs (ALB, NLB, CLB)
S3 Access Logs	`s3access`	Amazon S3 server access logs
CloudTrail Logs	`cloudtrail`	AWS CloudTrail API call logs
WAF Logs	`waf`	AWS Web Application Firewall logs
CloudWatch Logs	`cloudwatch`	CloudWatch Logs Subscription Filter events
Network Firewall Logs	`networkfirewall`	AWS Network Firewall event logs (Alert/Flow, TLS)

Breaking Change Notice

Format values have been simplified in v0.137.0

The old format values are deprecated and will be unsupported in v0.138.0.

AWS Log Type	Old Format Value (Deprecated)	New Format Value
VPC Flow Logs	`vpc_flow_log`	`vpcflow`
ELB Access Logs	`elb_access_log`	`elbaccess`
S3 Access Logs	`s3_access_log`	`s3access`
CloudTrail Logs	`cloudtrail_log`	`cloudtrail`
WAF Logs	`waf_log`	`waf`
CloudWatch Logs	`cloudwatch_logs_subscription_filter`	`cloudwatch`

Migration Path

If you're using the old format values you should update the encoding extension configuration with the new format values.

Feature Gates

Following feature gates are available to modify the behavior of the AWS Logs encoding extension. Enabling a feature gate can be done using --feature-gates argument similar to below,

otelcol --config=config.yaml --feature-gates --feature-gates=<FEATURE_GATE_ID>

VPC Flow Log Start Field ISO-8601 Format

Feature Gate ID: extension.awslogsencoding.vpcflow.start.iso8601
Stage: Alpha
Description: When enabled, the aws.vpc.flow.start field will be formatted as an ISO-8601 string instead of a Unix timestamp integer in seconds since epoch.
Default: Disabled (legacy behavior)

Behavior

Feature Gate State	Field Type	Format	Example
Disabled (Default)	`int64`	Unix seconds since epoch	`1609459200`
Enabled	`string`	ISO-8601 with milliseconds	`"2021-01-01T00:00:00.000Z"`

CloudTrail logs UserIdentity prefix

Feature Gate ID: extension.awslogsencoding.cloudtrail.enable.user.identity.prefix
Stage: Alpha
Description: When enabled, CloudTrail log userIdentity attributes will use 'aws.user_identity' prefix. This helps to preserve the attribute origin.
Default: Disabled

Behavior

Original attribute name	Default OTel attribute name	OTel attribute name with feature gate enabled
`userIdentity.userId`	`user.id`	`user.id`
`userIdentity.userName`	`user.name`	`user.name`
`userIdentity.accountId`	`aws.user_identity.account_id`	`aws.user_identity.account_id`
`userIdentity.accessKeyId`	`aws.access_key.id`	`aws.user_identity.access_key.id`
`userIdentity.identityStoreArn`	`aws.identity_store.arn`	`aws.user_identity.identity_store.arn`
`userIdentity.invokedBy`	`aws.user_identity.invoked_by`	`aws.user_identity.invoked_by`
`userIdentity.principalId`	`aws.principal.id`	`aws.user_identity.principal.id`
`userIdentity.arn`	`aws.principal.arn`	`aws.user_identity.principal.arn`
`userIdentity.type`	`aws.principal.type`	`aws.user_identity.principal.type`

Streaming Support

The extension implements streaming support which allows processing of input data to be processed without loading entire logs into memory. The implementation follows encoding.LogsDecoderExtension contract and streamed unmarhaling is exposed through NewLogsDecoder.

Note that, unlike non-streaming unmarshaling, caller is expected to detect and perform decompression operations (e.g. un-gzip). This allows streaming implementation to work independently of compression algorithms and buffer sizes.

The table below summarizes streaming support details for each log type, along with the offset tracking mechanism,

Log Type	Sub Log Type/Source	Offset Tracking	Notes
CloudTrail	Generic records	Number of records processed	Number of records are used as CloudTrail logs arrives as a JSON. Streaming is done on internal `Records` array
CloudTrail	CloudWatch trigger	Number of bytes processed	If non-zero offset is given, then invocation returns EOF with an empty log. The offset carries the full record length
CloudTrail	Digest record	Number of bytes processed	If non-zero offset is given, then invocation returns EOF with an empty log. The offset carries the full record length
ELB Access Logs	ALB/NLB/CLB	Bytes processed
Network Firewall	Alert/Flow/TLS	Bytes processed
S3 Access Logs	-	Bytes processed
Subscription filter	-	Number of records processed	Supports processing multi-line inputs and offset tracks number of records that get processed
VPC Flow Logs	S3 plain text	Bytes processed
VPC Flow Logs	CloudWatch subscription filter	Bytes processed	If non-zero offset is given, then invocation returns EOF with an empty log. The offset carries the full record length
WAF Logs	-	Bytes processed

Produced Records per Format

VPC flow log record fields

VPC flow log record fields are mapped this way in the resulting OpenTelemetry log:

Flow log field	Attribute in OpenTelemetry log
`version`	`aws.vpc.flow.log.version`
`account-id`	`cloud.account.id`
`interface-id`	`network.interface.name`
`srcaddr`	`source.address`: if `pkt-srcaddr` not filled or the same `network.peer.address`: otherwise
`pkt-srcaddr`	`source.address` if filled
`dstaddr`	`destination.address`: if `pkt-dstaddr` not filled or the same `network.peer.address`: otherwise
`pkt-dstaddr`	`destination.address` if filled
`srcport`	`source.port`
`dstport`	`destination.port`
`protocol`	`network.protocol.name`
`packets`	`aws.vpc.flow.packets`
`bytes`	`aws.vpc.flow.bytes`
`start`	`aws.vpc.flow.start`
`end`	Log timestamp
`action`	`aws.vpc.flow.action`
`log-status`	`aws.vpc.flow.status`
`vpc-id`	`aws.vpc.id`
`subnet-id`	`aws.vpc.subnet.id`
`instance-id`	`host.id`
`tcp-flags`	`network.tcp.flags`
`type`	`network.type`
`region`	`cloud.region`
`az-id`	`aws.az.id`
`sublocation-type`	`aws.sublocation.type`
`sublocation-id`	`aws.sublocation.id`
`pkt-src-aws-service`	`aws.vpc.flow.source.service`
`pkt-dst-aws-service`	`aws.vpc.flow.destination.service`
`flow-direction`	`network.io.direction`
`traffic-path`	`aws.vpc.flow.traffic_path`
`ecs-cluster-arn`	`aws.ecs.cluster.arn`
`ecs-cluster-name`	`aws.ecs.cluster.name`
`ecs-container-instance-arn`	`aws.ecs.container.instance.arn`
`ecs-container-instance-id`	`aws.ecs.container.instance.id`
`ecs-container-id`	`aws.ecs.container.id`
`ecs-second-container-id`	`aws.ecs.second.container.id`
`ecs-service-name`	`aws.ecs.service.name`
`ecs-task-definition-arn`	`aws.ecs.task.definition.arn`
`ecs-task-arn`	`aws.ecs.task.arn`
`ecs-task-id`	`aws.ecs.task.id`
`reject-reason`	`aws.vpc.flow.reject_reason`

S3 access log record fields

S3 access log record fields are mapped this way in the resulting OpenTelemetry log:

AWS field	OpenTelemetry Field
Bucket owner	`aws.s3.owner`
Bucket	`aws.s3.bucket`
Time	Log timestamp
Remote IP	`source.address`
Requester	`user.id`
Request ID	`aws.request_id`
Operation	`rpc.method`
Key	`aws.s3.key`
Request URI	Split by space: 1. `http.request.method` 2. URL is parsed: 1. `url.path` 2. `url.scheme` 3. `url.query` 3. Protocol splits in character `/`: 1.`network.protocol.name` 2. `network.protocol.version`
HTTP status	`http.response.status_code`
Error code	`error.type`
Bytes sent	`http.response.body.size`
Object size	`aws.s3.object.size`
Total time	`duration`
Turn around time	`aws.s3.turn_around_time`
Referer	`http.request.header.referer`
User-Agent	`user_agent.original`
Version ID	`aws.s3.version_id`
Host ID	`aws.extended_request_id`
Signature version	`aws.signature.version`
Cipher suite	`tls.cipher`
Authentication Type	`aws.s3.auth_type`
Host header	`http.request.header.host`
TLS version	`tls.protocol.version`
Access point ARN	`aws.s3.access_point.arn`
aclRequired	`aws.s3.acl_required`

AWS WAF log record fields

AWS WAF log record fields are mapped this way in the resulting OpenTelemetry log:

Original log field	OpenTelemetry field
`webaclId`	`cloud.resource_id` Also splits the value to get: 1.`cloud.region` 2.`cloud.account.id`
`terminatingRuleId`	`aws.waf.terminating_rule.id`
`terminatingRuleType`	`aws.waf.terminating_rule.type`
`httpSourceName`	`aws.waf.source.name`
`httpSourceId`	`aws.waf.source.id`
`httpRequest.clientIp`	`client.address`
`httpRequest.headers`	Each header is saved under `http.request.header.<header_name>`
`httpRequest.uri`	`url.path`
`httpRequest.args`	`url.query`
`httpRequest.httpMethod`	`http.request.method`
`httpRequest.httpVersion`	Splits in: 1.`network.protocol.name` 2.`network.protocol.version`
`httpRequest.requestId`	`aws.request_id`
`httpRequest.fragment`	`url.fragment`
`httpRequest.scheme`	`url.scheme`
`httpRequest.country`	`geo.country.iso_code`
`httpRequest.host`	`server.address`
`responseCodeSent`	`http.response.status_code`
`ja3Fingerprint`	`tls.client.ja3`
`ja4Fingerprint`	`tls.client.ja4`
`formatVersion`	Currently not supported
`terminatingRuleMatchDetails`	Currently not supported
`ruleGroupList`	Currently not supported
`rateBasedRuleList`	Currently not supported
`nonTerminatingMatchingRules`	Currently not supported
`requestHeadersInserted`	Currently not supported
`labels`	Currently not supported
`captchaResponse`	Currently not supported
`cfDistributionTenantId`	Currently not supported
`challengeResponse`	Currently not supported
`oversizeFields`	Currently not supported

CloudTrail record fields

Processed CloudTrail records come in two formats,

CloudTrail event records
CloudTrail digest record

CloudTrail event records

CloudTrail event records get mapped with following attributes in the resulting OpenTelemetry log:

CloudTrail event field	Attribute in OpenTelemetry log	Note
`apiVersion`	`aws.cloudtrail.api_version`
`eventID`	`aws.cloudtrail.event_id`
`eventVersion`	`aws.cloudtrail.event_version`
`eventCategory`	`aws.event.category`
`errorCode`	`aws.error.code`
`managementEvent`	`aws.event.management`
`errorMessage`	`aws.error.message`
`readOnly`	`aws.event.read_only`
`insightDetails`	`aws.insight_details` (as a map, if available)
`requestParameters`	`aws.request.parameters` (map of all request parameters)
`requestID`	`aws.request_id`
`resources`	`aws.resources` (as an array, if available)
`responseElements`	`aws.response.elements` (map of all response elements)
`additionalEventData`	`aws.cloudtrail.additional_event_data` (map of all additional event data)
`sessionCredentialFromConsole`	`aws.session.console` (set to true if value is "true")
`sharedEventID`	`aws.shared_event_id`
`recipientAccountId`	`cloud.account.id`
`awsRegion`	`cloud.region`
`eventName`	`rpc.method`
`eventSource`	`rpc.service`
`eventType`	`rpc.system`
`tlsDetails.clientProvidedHostHeader`	`server.address`
`sourceIPAddress`	`source.address`
`tlsDetails.cipherSuite`	`tls.cipher`
`tlsDetails.tlsVersion`	`tls.protocol.version`
`userAgent`	`user_agent.original`
`userIdentity.userId`	`user.id`	See feature gate
`userIdentity.userName`	`user.name`	See feature gate
`userIdentity.accountId`	`aws.user_identity.account_id`	See feature gate
`userIdentity.accessKeyId`	`aws.access_key.id`	See feature gate
`userIdentity.identityStoreArn`	`aws.identity_store.arn`	See feature gate
`userIdentity.invokedBy`	`aws.user_identity.invoked_by`	See feature gate
`userIdentity.principalId`	`aws.principal.id`	See feature gate
`userIdentity.arn`	`aws.principal.arn`	See feature gate
`userIdentity.type`	`aws.principal.type`	See feature gate
`userIdentity.sessionContext.attributes.creationDate`	`aws.user_identity.session_context.attributes.creation_date`
`userIdentity.sessionContext.attributes.mfaAuthenticated`	`aws.user_identity.session_context.attributes.mfa_authenticated`
`userIdentity.sessionContext.sessionIssuer.type`	`aws.user_identity.session_context.issuer.type`
`userIdentity.sessionContext.sessionIssuer.principalId`	`aws.user_identity.session_context.issuer.principal_id`
`userIdentity.sessionContext.sessionIssuer.arn`	`aws.user_identity.session_context.issuer.arn`
`userIdentity.sessionContext.sessionIssuer.accountId`	`aws.user_identity.session_context.issuer.account_id`
`userIdentity.sessionContext.sessionIssuer.userName`	`aws.user_identity.session_context.issuer.user_name`

CloudTrail digest record

CloudTrail digest record get mapped with following attributes in the resulting OpenTelemetry log:

CloudTrail digest field	Attribute in OpenTelemetry log
awsAccountId	cloud.account.id
digestEndTime	aws.cloudtrail.digest.end_time
digestS3Bucket	aws.cloudtrail.digest.s3_bucket
digestS3Object	aws.cloudtrail.digest.s3_object
newestEventTime	aws.cloudtrail.digest.newest_event
oldestEventTime	aws.cloudtrail.digest.oldest_event
previousDigestS3Bucket	aws.cloudtrail.digest.previous_s3_bucket
previousDigestS3Object	aws.cloudtrail.digest.previous_s3_object
logFiles.s3Bucket	aws.cloudtrail.digest.log_files.s3_bucket
logFiles.s3Object	aws.cloudtrail.digest.log_files.s3_bucket
logFiles.newestEventTime	aws.cloudtrail.digest.log_files.newest_event_time
logFiles.oldestEventTime	aws.cloudtrail.digest.log_files.oldest_event_time

Following fields are not included in the derived OpenTelemetry log:

digestPublicKeyFingerprint
digestSignatureAlgorithm
previousDigestHashValue
previousDigestHashAlgorithm
previousDigestSignature
logFiles.hashValue
logFiles.hashAlgorithm

All request parameters and response elements are included directly as nested maps in the attributes, preserving their original structure.

ELB Access Log Fields

ELB access log record fields are mapped this way in the resulting OpenTelemetry log:

Application Load Balancer (ALB)

AWS Fields are according to documentation.

AWS Field	OpenTelemetry Field(s)
type	`network.protocol.name`
time	Log timestamp
elb	`cloud.resource_id`
client:port	`client.address`, `client.port`
received_bytes	`http.request.size`
sent_bytes	`http.response.size`
"request"	`url.full`, `http.request.method`, `network.protocol.version`
ssl_cipher	`tls.cipher`
ssl_protocol	`tls.protocol.version`
elb_status_code	`aws.elb.status.code`
user_agent	`user_agent.original`
domain_name	`url.domain`
target:port	`destination.address`, `destination.port`
request_processing_time	`aws.alb.request_processing_time`
target_processing_time	`aws.elb.target_processing_time`
response_processing_time	`aws.elb.response_processing_time`
target_status_code	`aws.elb.backend.status.code`
target_group_arn	`aws.elb.target_group_arn`
"trace_id"	`aws.elb.aws_trace_id`
"chosen_cert_arn"	`aws.elb.chosen_cert_arn`
matched_rule_priority	Currently not supported
request_creation_time	Currently not supported
"actions_executed"	`aws.elb.actions_executed`
"redirect_url"	`aws.elb.redirect_url`
"error_reason"	`aws.elb.error_reason`
"target:port_list"	Currently not supported
"target_status_code_list"	Currently not supported
"classification"	`aws.elb.classification`
"classification_reason"	`aws.elb.classification_reason`
conn_trace_id	`aws.elb.connection_trace_id`
transformed_host	`aws.elb.transformed_host`
transformed_uri	`aws.elb.transformed_uri`
request_transform_status	`aws.elb.request_transform_status`

Network Load Balancer (NLB)

AWS Fields are according to documentation.

AWS Field	OpenTelemetry Field(s)
type	`network.protocol.name`
version	`network.protocol.version`
time	Log timestamp
elb	`cloud.resource_id`
listener	`aws.elb.tls.listener.resource_id`
client:port	`client.address`, `client.port`
destination:port	`destination.address`, `destination.port`
received_bytes	`http.request.size`
sent_bytes	`http.response.size`
tls_cipher	`tls.cipher`
tls_protocol_version	`tls.protocol.version`
domain_name	`url.domain`
connection_time	`aws.elb.connection_time`
tls_handshake_time	`aws.elb.tls_handshake_time`
incoming_tls_alert	`aws.elb.incoming_tls_alert`
chosen_cert_arn	`aws.elb.chosen_cert_arn`
chosen_cert_serial	`aws.elb.chosen_cert_serial`
tls_named_group	`aws.elb.tls_named_group`
alpn_fe_protocol	`aws.elb.alpn_fe_protocol`
alpn_be_protocol	`aws.elb.alpn_be_protocol`
alpn_client_preference_list	`aws.elb.alpn_client_preference_list`
tls_connection_creation_time	`aws.elb.tls_connection_creation_time`

Classic Load Balancer (CLB)

AWS Fields are according to documentation

AWS Field	OpenTelemetry Field(s)
time	Log timestamp
elb	`cloud.resource_id`
client:port	`client.address`, `client.port`
elb_status_code	`aws.elb.status.code`
backend_status_code	`aws.elb.backend.status.code`
received_bytes	`http.request.size`
sent_bytes	`http.response.size`
"request"	`url.full`, `http.request.method`, `network.protocol.name`, `network.protocol.version`
ssl_cipher	`tls.cipher`
ssl_protocol	`tls.protocol.version`
user_agent	`user_agent.original`
backend:port	Currently not supported
request_processing_time	Currently not supported
backend_processing_time	Currently not supported
response_processing_time	Currently not supported

AWS Network Firewall log record fields

AWS Network Firewall produces three types of logs: Alert, Flow, and TLS logs. AWS Network Firewall logging content internally uses Suricata open source threat detection engine for its alert and flow (netflow) event types. See the following field references for more information:

The following fields are common across all log types:

Common log fields	Attribute in OpenTelemetry log
`firewall_name`	`aws.networkfirewall.name`
`availability_zone`	`cloud.availability_zone`
`event_timestamp`	Log timestamp
`event.event_type`	`aws.networkfirewall.event.type`
`event.src_ip`	`source.address`
`event.src_port`	`source.port`
`event.dest_ip`	`destination.address`
`event.dest_port`	`destination.port`
`event.proto`	`network.transport`
`event.flow_id`	`aws.networkfirewall.flow_id`

Flow or `netflow` type log fields

Flow log fields	Attribute in OpenTelemetry log
`event.netflow.pkts`	`aws.networkfirewall.netflow.packets`
`event.netflow.bytes`	`aws.networkfirewall.netflow.bytes`
`event.netflow.start`	`aws.networkfirewall.netflow.start`
`event.netflow.end`	`aws.networkfirewall.netflow.end`
`event.netflow.age`	`aws.networkfirewall.netflow.age`
`event.netflow.max_ttl`	`aws.networkfirewall.netflow.max_ttl`
`event.netflow.min_ttl`	`aws.networkfirewall.netflow.min_ttl`
`event.netflow.tx_cnt`	`aws.networkfirewall.netflow.transaction.count`

Alert log fields

Alert log fields	Attribute in OpenTelemetry log
`event.alert.action`	`aws.networkfirewall.alert.action`
`event.alert.signature`	`aws.networkfirewall.alert.signature`
`event.alert.signature_id`	`aws.networkfirewall.alert.signature_id`
`event.alert.rev`	`aws.networkfirewall.alert.rev`
`event.alert.category`	`aws.networkfirewall.alert.category`
`event.alert.severity`	`aws.networkfirewall.alert.severity`
`event.alert.gid`	`aws.networkfirewall.alert.gid`
`event.alert.metadata.affected_product`	`aws.networkfirewall.alert.metadata.affected_product`
`event.alert.metadata.attack_target`	`aws.networkfirewall.alert.metadata.attack_target`
`event.alert.metadata.deployment`	`aws.networkfirewall.alert.metadata.deployment`
`event.alert.metadata.former_category`	`aws.networkfirewall.alert.metadata.former_category`
`event.alert.metadata.malware_family`	`aws.networkfirewall.alert.metadata.malware_family`
`event.alert.metadata.performance_impact`	`aws.networkfirewall.alert.metadata.performance_impact`
`event.alert.metadata.signature_severity`	`aws.networkfirewall.alert.metadata.signature_severity`
`event.alert.metadata.created_at`	`aws.networkfirewall.alert.metadata.created_at`
`event.alert.metadata.updated_at`	`aws.networkfirewall.alert.metadata.updated_at`

TLS log fields

See TLS inspection page and Suricata fields for more details.

TLS log fields	Attribute in OpenTelemetry log
`event.sni`	`server.address`
`event.revocation_check.leaf_cert_fpr`	`aws.networkfirewall.tls.revocation_check.leaf_cert_fpr`
`event.revocation_check.action`	`aws.networkfirewall.tls.revocation_check.action`
`event.revocation_check.status`	`aws.networkfirewall.tls.revocation_check.status`
`event.tls_error.error_message`	`aws.networkfirewall.tls.error.message`
`event.tls.subject`	`tls.client.subject`
`event.tls.issuer`	`tls.client.issuer`
`event.tls.session_resumed`	`tls.resumed`

HTTP log fields

HTTP log fields	Attribute in OpenTelemetry log
`event.http.hostname`	`url.domain`
`event.http.url`	`url.path`
`event.http.http_user_agent`	`user_agent.original`
`event.http.http_content_type`	`http.request.header.content-type`
`event.http.cookie`	`http.request.header.cookie`

CloudWatch Logs Subscription Filter record fields

CloudWatch Logs Subscription Filter events are mapped to OpenTelemetry logs with the following resource attributes:

CloudWatch Logs field	Attribute in OpenTelemetry log
`owner`	`cloud.account.id`
`logGroup`	`aws.log.group.names` (array)
`logStream`	`aws.log.stream.names` (array)

Each log event's timestamp is converted to the OpenTelemetry log timestamp, and the message is set as the log body.

Extracted Fields for Centralized Logging

When using CloudWatch Logs centralization to consolidate logs from multiple AWS accounts and regions into a central account, you can enable emitSystemFields in your CloudWatch Logs subscription filter to include the original account ID and region in each log event.

To enable extracted fields, set the emitSystemFields parameter when creating or updating your CloudWatch Logs subscription filter:

When emitSystemFields is enabled, the following fields are extracted and mapped to OpenTelemetry semantic conventions:

Extracted field	Attribute in OpenTelemetry log
`@aws.account`	`cloud.account.id`
`@aws.region`	`cloud.region`

Note: When extracted fields are present, they take precedence over the owner field for cloud.account.id. Logs with different extracted field values (different account IDs or regions) are automatically grouped into separate ResourceLogs to ensure proper resource attribution in OpenTelemetry.

Documentation ¶

Overview ¶

Package awslogsencodingextension provides an encoding extension for unmarshalling logs produced by various AWS services.

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func NewFactory ¶

func NewFactory() extension.Factory

Types ¶

type Config ¶

type Config struct {
	// Format defines the AWS logs format.
	//
	// Current valid values are:
	// - cloudwatch
	// - vpcflow
	// - s3access
	// - waf
	// - cloudtrail
	// - elbaccess
	// - networkfirewall
	//
	Format string `mapstructure:"format"`

	VPCFlowLogConfig vpcflowlog.Config `mapstructure:"vpcflow"`
	// Deprecated: use VPCFlowLogConfig instead. It will be removed in v0.138.0
	VPCFlowLogConfigV1 vpcflowlog.Config `mapstructure:"vpc_flow_log"`
	// contains filtered or unexported fields
}

func (*Config) Validate ¶

func (cfg *Config) Validate() error

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
internal
constants
metadata
unmarshaler
unmarshaler/cloudtraillog
unmarshaler/elb-access-log
unmarshaler/network-firewall-log
unmarshaler/s3-access-log
unmarshaler/subscription-filter
unmarshaler/vpc-flow-log
unmarshaler/waf

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

README ¶

AWS Logs Encoding Extension

Log Format Identification

Format Values

Breaking Change Notice

Migration Path

Feature Gates

VPC Flow Log Start Field ISO-8601 Format

Behavior

CloudTrail logs UserIdentity prefix

Behavior

Streaming Support

Produced Records per Format

VPC flow log record fields

S3 access log record fields

AWS WAF log record fields

CloudTrail record fields

CloudTrail event records

CloudTrail digest record

ELB Access Log Fields

Application Load Balancer (ALB)

Network Load Balancer (NLB)

Classic Load Balancer (CLB)

AWS Network Firewall log record fields

Flow or netflow type log fields

Alert log fields

TLS log fields

HTTP log fields

CloudWatch Logs Subscription Filter record fields

Extracted Fields for Centralized Logging

Documentation ¶

Overview ¶

Index ¶

Constants ¶

Variables ¶

Functions ¶

func NewFactory ¶

Types ¶

type Config ¶

func (*Config) Validate ¶

Source Files ¶

Directories ¶

Flow or `netflow` type log fields