cloudflarereceiver

README

Cloudflare Receiver

The Cloudflare Receiver allows Cloudflare's LogPush Jobs to send logs over HTTPS from the Cloudflare logs aggregation system to an OpenTelemetry collector.

Status
Stability	alpha: logs
Distributions	contrib
Issues
Code coverage
Code Owners	@dehaansa | Seeking more code owners!

Getting Started

To successfully operate this receiver, you must follow these steps in order:

1. Have a Cloudflare site at the Enterprise plan level.
   • At the time the receiver was written, LogPush was available only for Enterprise sites.
2. Create a LogPush HTTP destination job following the directions provided by Cloudflare. When the job is created, it will attempt to validate the connection to the receiver.
   • If you've configured the receiver with a secret to validate requests, ensure you add the value to the destination_conf parameter of the LogPush job by adding its value as a query parameter under the header_X-CF-Secret parameter. For example, "destination_conf": "https://example.com?header_X-CF-Secret=abcd1234".
   • If you want the receiver to parse one of the fields as the log record's timestamp (EdgeStartTimestamp is the default), you must configure the timestamp format in your LogPush job to match your data:
     • Cloudflare Logpush default timestamp_format is unixnano, which the receiver parses automatically.
     • Use &timestamps=rfc3339 in logpull_options for RFC3339-formatted strings (with or without fractional seconds).
     • In output_options, set timestamp_format: "rfc3339" for RFC3339 strings.
     • In output_options, set timestamp_format: "unix" for Unix seconds.
     • In output_options, set timestamp_format: "unixnano" for Unix nanoseconds.
   • The receiver expects the uploaded logs to be in ndjson format with no template, prefix, suffix, or delimiter changes based on the options in output_options. The only settings supported by this receiver in output_options are:
     • field_names
     • timestamp_format
     • sample_rate
     • CVE-2021-44228
3. If the LogPush job creates successfully, the receiver is correctly configured and the LogPush job was able to send it a "test" message. If the job failed to create, the most likely issue is with the SSL configuration. Check both the LogPush API response and the receiver's logs for more details.

Optional

If the receiver will be handling TLS termination:

1. Receive a properly CA signed SSL certificate for use on the collector host.
2. Configure the receiver using the previously acquired SSL certificate, and then start the collector.

Configuration

• tls (Optional - Cloudflare requires TLS, and self-signed will not be sufficient)
  • cert_file
    • You may need to append your CA certificate to the server's certificate, if it is not a CA known to the LogPush API.
  • key_file
• endpoint
  • The endpoint on which the receiver will await requests from Cloudflare
• secret
  • If this value is set, the receiver expects to see it in any valid requests under the X-CF-Secret header
• timestamp_field (default: EdgeStartTimestamp)
  • This receiver was built with the Cloudflare http_requests dataset in mind, but should be able to support any Cloudflare dataset. If using another dataset, you will need to set the timestamp_field appropriately in order to have the log record be associated with the correct timestamp.
• timestamp_format (default: unixnano)
  • One of unix, unixnano, or rfc3339, matching how your LogPush job encodes the timestamp field.
• attributes
  • This parameter allows the receiver to be configured to set log record attributes based on fields found in the log message. The fields are not removed from the log message when set in this way. Only string, boolean, integer or float fields can be mapped using this parameter.
  • When the attributes configuration is empty, the receiver will automatically ingest all fields from the log messages as attributes, using the original field names as attribute names.
• separator (default: .)
  • The separator used to join nested fields in the log message when setting attributes. For example, if the log message contains a field "RequestHeaders": { "Content-Type": "application/json" }, and the separator is set to ., the attribute will be set as RequestHeaders.Content_Type. If the separator is set to _, it will be set as RequestHeaders_Content_Type.
• max_request_body_size: configures the maximum allowed body size in bytes for a single request. Default: 20971520 (20MiB)

Example:

receivers:
  cloudflare:
    logs:
      tls:
        key_file: some_key_file
        cert_file: some_cert_file
      endpoint: 0.0.0.0:12345
      secret: 1234567890abcdef1234567890abcdef
      timestamp_field: EdgeStartTimestamp
      timestamp_format: rfc3339
      attributes:
        ClientIP: http_request.client_ip
        ClientRequestURI: http_request.uri

Example with automatic attribute ingestion:

receivers:
  cloudflare:
    logs:
      tls:
        key_file: some_key_file
        cert_file: some_cert_file
      endpoint: 0.0.0.0:12345
      secret: 1234567890abcdef1234567890abcdef
      timestamp_field: EdgeStartTimestamp
      timestamp_format: rfc3339
      attributes:
        # Specifying no attributes ingests them all

Documentation

Index

• func NewFactory() receiver.Factory
• type Config
  • func (c *Config) Validate() error
• type LogsConfig

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory returns the component factory for the cloudflarereceiver

Types

type Config

type Config struct {
	Logs LogsConfig `mapstructure:"logs"`
	// contains filtered or unexported fields
}

Config holds all the parameters to start an HTTP server that can be sent logs from CloudFlare

func (*Config) Validate

func (c *Config) Validate() error

type LogsConfig

type LogsConfig struct {
	Secret          string                  `mapstructure:"secret"`
	Endpoint        string                  `mapstructure:"endpoint"`
	TLS             *configtls.ServerConfig `mapstructure:"tls"`
	Attributes      map[string]string       `mapstructure:"attributes"`
	TimestampField  string                  `mapstructure:"timestamp_field"`
	TimestampFormat string                  `mapstructure:"timestamp_format"`
	Separator       string                  `mapstructure:"separator"`
	// MaxRequestBodySize sets the maximum request body size in bytes. 0 means no limit.
	// Default: 20MB
	MaxRequestBodySize int64 `mapstructure:"max_request_body_size,omitempty"`
	// contains filtered or unexported fields
}