sllogformatprocessor

README

ScienceLogic Format Processor

Status
Stability	logs beta
Supported pipeline types	logs
Distributions	contrib

The ScienceLogic format processor accepts logs and places them into batches annotated with the attributes required for processing by other ScienceLogic components. Each batch is forwarded as a resource log entry with resource attributes that identify the log stream, i.e. from an instance of an application running on a single host. The configuration describes where to find the appropriate attributes and how to present them in the metadata attributes. To work with different receivers, you can define multiple profiles that match against incoming logs in the order configured. A match requires the following attributes:

• service_group: Domain of anomaly correlation
• host: Host or computer name
• logbasename: Application in lowercase, e.g. postgres
• severity: [OPTIONAL] Location to find severity value

Additional optional attributes can be configured as labels. These attributes can be derived from the following sources in the incoming stream:

• lit: A literal injected from the configuration
• rattr: Resource attribute, also called resource label
• attr: Log record attribute
• body: The message body as a string or map to elements within

The syntax for associating metadata looks like:

<destination>:
  exp:
    source: <source>[:<key path>]
  rename: <replacement name>
  validate: <regexp>

If rename is omitted, the key path is used as the attribute name if available followed by the resulting value. It is recommended to specify rename for service group, host, logbasename, and all literals.

Optional validate will cause the profile attribute not to match if the value does not matche specified golang regular expression.

In addition to sources an expression can be formed from the following operators with associated expressions A and B:

• rmprefix: Remove prefix B from A if matched
• rmsuffix: Remove suffix B from A if matched
• rmtail: Remove everything from A after and including the last match of B
• alphanum: Filter out all characters that are not letters or numbers from A
• unescape: Filter out ESC character
• lc: Transform A to lowercase
• regexp: Concatinate all captures from A using golang regexp B
• and: Concatinate all results from expressions
• or: Return the first expression result that is not empty

The syntax for operators looks like:

<destination>:
  exp:
    op: <operator>
    exps:
    - <expression A>
    - <expression B>
    - <expression C ...>
  rename: <replacement name>
  validate: <regexp>

The expressions under exps are either source or a single op with associated exps of its own.

Profiles have an additional configuration for the message format with the following values:

• event: Prefix the message with timestamp and severity
• message: Forward the message body as is
• container: Special handling for logs from docker containers

The attributes assigned by this processor for consumption by ScienceLogic commponents include the following resource attributes:

• sl_service_group: Domain of anomaly correlation
• sl_host: Host or computer name
• sl_logbasename: Application in lowercase, e.g. postgres
• sl_format: Format option from the matching profile
• sl_metadata: An encoding of all log stream metadata

And the following log record attribute:

• sl_msg: The log message body formatted for consumption by ScienceLogic

The following configuration options can be modified:

• send_batch_size (default = 8192): Number of spans, metric data points, or log records after which a batch will be sent regardless of the timeout.
• timeout (default = 200ms): Time duration after which a batch will be sent regardless of size.
• send_batch_max_size (default = 0): The upper limit of the batch size. 0 means no upper limit of the batch size. This property ensures that larger batches are split into smaller units. It must be greater than or equal to send_batch_size.

Examples:

processors:
  sllogformat:
    send_batch_size: 10000
    timeout: 10s
    profiles:
    - service_group: # windows event log
        exp:
          source: lit:default
        rename: ze_deployment_name
      host:
        exp:
          source: body:computer
        rename: host
        validate: "^myhost$"
      logbasename:
        exp:
          op: lc
          exps:
          - op: alphanum
            exps:
              - op: rmprefix
                exps:
                  - source: body:provider.name
                  - source: lit:Microsoft-Windows-
        rename: logbasename
      labels:
      - exp:
          source: body:channel
        rename: win_channel
      - exp:
          source: body:keywords
        rename: win_keywords
      message:
        exp:
          op: or
          exps:
            - source: body:message
            - source: body:event_data
            - source: body:keywords
      format: event
    - service_group: # docker logs
        exp:
          source: lit:default
        rename: ze_deployment_name
      host:
        exp:
          source: rattr:host.name
        rename: host
      logbasename:
        exp:
          source: attr:container_id
        rename: logbasename
      labels:
      - exp:
          source: rattr:os.type
      - exp:
          source: attr:log.file.path
        rename: zid_path
      message:
        exp:
          source: body
      format: container

Documentation

Index

• Constants
• func FilterASCII(in string) string
• func NewFactory() processor.Factory
• type Config
  • func (c *Config) MatchProfile(log *zap.Logger, rl plog.ResourceLogs, ils plog.ScopeLogs, lr plog.LogRecord) (*ConfigResult, *StreamTokenReq, error)
  • func (cfg *Config) Validate() error
• type ConfigAttribute
• type ConfigExpression
• type ConfigProfile
• type ConfigResult
• type ContainerLogEntry
• type Operator
• type Parser
  • func (p *Parser) EvalElem(attribute *ConfigAttribute) (string, string)
• type StreamTokenReq

Constants

const (
	CfgSourceRattr     string = "rattr"
	CfgSourceAttr      string = "attr"
	CfgSourceBody      string = "body"
	CfgSourceLit       string = "lit"
	CfgFormatMessage   string = "message"
	CfgFormatContainer string = "container"
	CfgFormatEvent     string = "event"
	CfgOpRmprefix      string = "rmprefix"
	CfgOpRmsuffix      string = "rmsuffix"
	CfgOpRmtail        string = "rmtail"
	CfgOpAlphaNum      string = "alphanum"
	CfgOpLc            string = "lc"
	CfgOpUnescape      string = "unescape"
	CfgOpReplace       string = "replace"
	CfgOpRegexp        string = "regexp"
	CfgOpAnd           string = "and"
	CfgOpOr            string = "or"
)

const CMaxNumExps = 10

Functions

func FilterASCII

func FilterASCII(in string) string

func NewFactory

func NewFactory() processor.Factory

NewFactory returns a new factory for the Batch processor.

Types

type Config

type Config struct {
	// Science Logic input profiles
	Profiles []ConfigProfile `mapstructure:"profiles"`

	// Timeout sets the time after which a batch will be sent regardless of size.
	Timeout time.Duration `mapstructure:"timeout"`

	// SendBatchSize is the size of a batch which after hit, will trigger it to be sent.
	SendBatchSize uint32 `mapstructure:"send_batch_size"`

	// SendBatchMaxSize is the maximum size of a batch. It must be larger than SendBatchSize.
	// Larger batches are split into smaller units.
	// Default value is 0, that means no maximum size.
	SendBatchMaxSize uint32 `mapstructure:"send_batch_max_size"`
}

Config defines configuration for batch processor.

func (*Config) MatchProfile

func (c *Config) MatchProfile(log *zap.Logger, rl plog.ResourceLogs, ils plog.ScopeLogs, lr plog.LogRecord) (*ConfigResult, *StreamTokenReq, error)

func (*Config) Validate

func (cfg *Config) Validate() error

Validate checks if the processor configuration is valid

type ConfigAttribute

type ConfigAttribute struct {
	Exp      *ConfigExpression `mapstructure:"exp"`
	Rename   string            `mapstructure:"rename"`
	Validate string            `mapstructure:"validate"`
}

type ConfigExpression

type ConfigExpression struct {
	Source string              `mapstructure:"source"`
	Op     string              `mapstructure:"op"`
	Exps   []*ConfigExpression `mapstructure:"exps"`
}

type ConfigProfile

type ConfigProfile struct {
	ServiceGroup *ConfigAttribute   `mapstructure:"service_group"`
	Host         *ConfigAttribute   `mapstructure:"host"`
	Logbasename  *ConfigAttribute   `mapstructure:"logbasename"`
	Severity     *ConfigAttribute   `mapstructure:"severity"`
	Labels       []*ConfigAttribute `mapstructure:"labels"`
	Message      *ConfigAttribute   `mapstructure:"message"`
	Format       string             `mapstructure:"format"`
}

type ConfigResult

type ConfigResult struct {
	ServiceGroup string   `mapstructure:"service_group"`
	Host         string   `mapstructure:"host"`
	Logbasename  string   `mapstructure:"logbasename"`
	Severity     string   `mapstructure:"severity"`
	Labels       []string `mapstructure:"labels"`
	Message      string   `mapstructure:"message"`
	Format       string   `mapstructure:"format"`
}

type ContainerLogEntry

type ContainerLogEntry struct {
	Log       string `json:"log"`
	Timestamp string `json:"timestamp"`
	Stream    string `json:"stream"`
}

A formatted container log entry

type Operator

type Operator func(string, string) string

type Parser

type Parser struct {
	Log   *zap.Logger
	Rattr pcommon.Map
	Attr  pcommon.Map
	Body  pcommon.Value
}

func (*Parser) EvalElem

func (p *Parser) EvalElem(attribute *ConfigAttribute) (string, string)

type StreamTokenReq

type StreamTokenReq struct {
	Stream             string            `json:"stream"`
	Logbasename        string            `json:"logbasename"`
	ContainerLog       bool              `json:"container_log"`
	LogType            string            `json:"log_type"`
	ForwardedLog       bool              `json:"forwarded_log"`
	Tz                 string            `json:"tz"`
	ZeLogCollectorVers string            `json:"Ze_log_collector_vers"`
	Ids                map[string]string `json:"ids"`
	Cfgs               map[string]string `json:"cfgs"`
	Tags               map[string]string `json:"tags"`
}