format

package
v0.5.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 17, 2019 License: Apache-2.0 Imports: 23 Imported by: 6

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Aggregate added in v0.5.0 

type Aggregate struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Aggregate formatter plugin

Aggregate is a formatter which can group up further formatter. The `ApplyTo` settings will be pass on and overwritten in the child formatter. This plugin could be useful to setup complex configs with metadata handling in more readable format.

Parameters

- ApplyTo: This value chooses the part of the message the formatting should be applied to. Use "" to target the message payload; other values specify the name of a metadata field to target. This value will also used for further child modulators! By default this parameter is set to "".

- Modulators: Defines a list of child modulators to be applied to a message when it arrives at this formatter. Please try to use only content based formatter and filter! If a modulator changes the stream of a message the message is NOT routed to this stream anymore.

Examples

This example show a useful case for format.Aggregate plugin: 

exampleConsumerA:
  Type: consumer.Console
  Streams: "foo"
  Modulators:
    - format.MetadataCopy:
        CopyToKeys: ["foo", "bar"]
    - format.Aggregate:
        ApplyTo: foo
        Modulators:
          - format.Base64Encode
          - format.Double
          - format.Envelope:
              Postfix: "\n"
    - format.Envelope:
        Postfix: "\n"
        ApplyTo: bar

# same config as
exampleConsumerB:
  Type: consumer.Console
  Streams: "bar"
  Modulators:
    - format.MetadataCopy:
        CopyToKeys: ["foo", "bar"]
    - format.Base64Encode:
        ApplyTo: foo
    - format.Double:
        ApplyTo: foo
    - format.Envelope:
        Postfix: "\n"
        ApplyTo: foo
    - format.Envelope:
        Postfix: "\n"
        ApplyTo: bar

func (*Aggregate) ApplyFormatter added in v0.5.0 

func (format *Aggregate) ApplyFormatter(msg *core.Message) error

ApplyFormatter execute the formatter

func (*Aggregate) Configure added in v0.5.0 

func (format *Aggregate) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Base64Decode 

type Base64Decode struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Base64Decode formatter plugin

Base64Decode is a formatter that decodes base64 encoded messages. If a message is not or only partly base64 encoded an error will be logged and the decoded part is returned. RFC 4648 is expected.

Parameters

- Base64Dictionary: This value defines the 64-character base64 lookup dictionary to use. When left empty, a dictionary as defined by RFC4648 is used. By default this parameter is set to "".

Examples

This example expects base64 strings from the console and decodes them before transmitting the message payload. 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.Base64Decode

func (*Base64Decode) ApplyFormatter added in v0.5.0 

func (format *Base64Decode) ApplyFormatter(msg *core.Message) error

ApplyFormatter execute the formatter

func (*Base64Decode) Configure 

func (format *Base64Decode) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Base64Encode 

type Base64Encode struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Base64Encode formatter

Base64Encode is a formatter that decodes Base64 encoded strings. Custom dictionaries are supported, by default RFC 4648 standard encoding is used.

Parameters

- Base64Dictionary: Defines the 64-character base64 lookup dictionary to use. When left empty a RFC 4648 standard encoding is used. By default this parameter is set to "".

Examples

This example uses RFC 4648 URL encoding to format incoming data. 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - formatter.Base64Encode
      Dictionary: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"

func (*Base64Encode) ApplyFormatter added in v0.5.0 

func (format *Base64Encode) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Base64Encode) Configure 

func (format *Base64Encode) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Clear added in v0.4.4 

type Clear struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
}

Clear formatter

This formatter erases the message payload or deletes a metadata key.

Examples

This example removes the "pipe" key from the metadata produced by consumer.Console. 

exampleConsumer:
  Type: consumer.Console
  Streams: stdin
  Modulators:
    - format.Clear
      ApplyTo: pipe

func (*Clear) ApplyFormatter added in v0.5.0 

func (format *Clear) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Clear) Configure added in v0.4.4 

func (format *Clear) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type CollectdToInflux08 added in v0.4.0 

type CollectdToInflux08 struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
}

CollectdToInflux08 formatter

This formatter transforms JSON data produced by collectd to InfluxDB 0.8.x. Trailing and leading commas are removed from the Collectd message.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - formatter.CollectdToInflux08

func (*CollectdToInflux08) ApplyFormatter added in v0.5.0 

func (format *CollectdToInflux08) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*CollectdToInflux08) Configure added in v0.4.0 

func (format *CollectdToInflux08) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type CollectdToInflux09 added in v0.4.0 

type CollectdToInflux09 struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
}

CollectdToInflux09 formatter

This formatter transforms JSON data produced by collectd to InfluxDB 0.9.0. Trailing and leading commas are removed from the Collectd message.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - formatter.CollectdToInflux09

func (*CollectdToInflux09) ApplyFormatter added in v0.5.0 

func (format *CollectdToInflux09) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*CollectdToInflux09) Configure added in v0.4.0 

func (format *CollectdToInflux09) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type CollectdToInflux10 added in v0.4.1 

type CollectdToInflux10 struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

CollectdToInflux10 formatter

This formatter transforms JSON data produced by collectd to InfluxDB 0.9.1 or later. Trailing and leading commas are removed from the Collectd message.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - formatter.CollectdToInflux10

func (*CollectdToInflux10) ApplyFormatter added in v0.5.0 

func (format *CollectdToInflux10) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*CollectdToInflux10) Configure added in v0.4.1 

func (format *CollectdToInflux10) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Double added in v0.5.0 

type Double struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Double formatter plugin

Double is a formatter that appends a delimiter string and a second copy of the message's contents to the message. Independent sets of formatters may be applied to both duplicates.

Parameters

- Separator: This value sets the separator string placed between both parts. This parameter is set to ":" by default.

- UseLeftStreamID: Use the stream id of the left side as the final stream id for the message if this value is "true". This parameter is set to "false" by default.

- Left: An optional list of formatters. The first copy of the message (left of the delimiter) is passed through these filters. This parameter is set to an empty list by default.

- Left: An optional list of formatters. The second copy of the mssage (right of the delimiter) is passed through these filters. This parameter is set to an empty list by default.

Examples

This example creates a message of the form "<orig>|<base64>", where <orig> is the original console input and <base64> its Base64-encoded equivalent. 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.Double:
      Separator: "|"
      Right:
        - format.Base64Encode

func (*Double) ApplyFormatter added in v0.5.0 

func (format *Double) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Double) Configure added in v0.5.0 

func (format *Double) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Envelope 

type Envelope struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Envelope formatter

This formatter adds content to the beginning and/or end of a message.

Parameters

- Prefix: Defines a string that is added to the front of the message. Special characters like \n \r or \t can be used without additional escaping. By default this parameter is set to "".

- Postfix: Defines a string that is added to the end of the message. Special characters like \n \r or \t can be used without additional escaping. By default this parameter is set to "\n".

Examples

This example adds a line number and a newline character to each message printed to the console. 

exampleProducer:
  Type: producer.Console
  Streams: "*"
  Modulators:
    - format.Sequence
    - format.Envelope

func (*Envelope) ApplyFormatter added in v0.5.0 

func (format *Envelope) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Envelope) Configure 

func (format *Envelope) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type ExtractJSON added in v0.4.3 

type ExtractJSON struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

ExtractJSON formatter

This formatter extracts a specific value from a JSON payload and writes it back as a new payload or as a metadata field.

Parameters

- Field: Defines the JSON key to extract. If the field does not exist an empty string is returned. Field paths can be defined in a format accepted by tgo.MarshalMap.Path. By default this parameter is set to "".

- TrimValues: Enables trimming of whitespaces at the beginning and end of the extracted value. By default this parameter is set to true.

- Precision: Defines the number of decimal places to use when converting Numbers into strings. If this parameter is set to -1 the shortest possible number of decimal places will be used. By default this parameter is set to -1.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - formatter.ExtractJSON
      Field: host
      ApplyTo: host

func (*ExtractJSON) ApplyFormatter added in v0.5.0 

func (format *ExtractJSON) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*ExtractJSON) Configure added in v0.4.3 

func (format *ExtractJSON) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type GrokToJSON added in v0.5.0 

type GrokToJSON struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

GrokToJSON formatter plugin

GrokToJSON is a formatter that applies regex filters to messages. It works by combining text patterns into something that matches your logs. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics for more information about Grok.

The output format is JSON.

Parameters

- RemoveEmptyValues: When set to true, empty captures will not be returned. By default this parameter is set to "true".

- NamedCapturesOnly: When set to true, only named captures will be returned. By default this parameter is set to "true".

- SkipDefaultPatterns: When set to true, standard grok patterns will not be included in the list of patterns. By default this parameter is set to "true".

- Patterns: A list of grok patterns that will be applied to messages. The first matching pattern will be used to parse the message.

Examples

This example transforms unstructured input into a structured json output. Input: 

us-west.servicename.webserver0.this.is.the.measurement 12.0 1497003802

Output: 

{
  "datacenter": "us-west",
  "service": "servicename",
  "host": "webserver0",
  "measurement": "this.is.the.measurement",
  "value": "12.0",
  "time": "1497003802"
}

Config: 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.GrokToJSON:
      Patterns:
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.gauge-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_gauge:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.latency-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_latency:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.derive-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_derive:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value:float}\s*%{INT:time}

func (*GrokToJSON) ApplyFormatter added in v0.5.0 

func (format *GrokToJSON) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*GrokToJSON) Configure added in v0.5.0 

func (format *GrokToJSON) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Hostname 

type Hostname struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Hostname formatter

This formatter prefixes the message or metadata with the hostname of the machine gollum is running on.

Parameters

- Separator: Defines the separator string placed between hostname and data. By default this parameter is set to ":".

Examples

This example inserts the hostname into an existing JSON payload. 

exampleProducer:
  Type: producer.Console
  Streams: "*"
  Modulators:
    - format.Trim:
      LeftSeparator: "{"
      RightSeparator: "}"
    - format.Hostname
      Separator: ","
    - format.Envelope:
      Prefix: "{\"host\":"
      Postfix: "}"

func (*Hostname) ApplyFormatter added in v0.5.0 

func (format *Hostname) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Hostname) Configure 

func (format *Hostname) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Identifier 

type Identifier struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Identifier formatter

This formatter generates a (mostly) unique 64 bit identifier number from the message payload, timestamp and/or sequence number. The number is be converted to a human readable form.

Parameters

- Generator: Defines which algorithm to use when generating the identifier. This my be one of the following values. By default this parameter is set to "time"

  • hash: The message payload will be hashed using fnv1a and returned as hex.

  • time: The id will be formatted YYMMDDHHmmSSxxxxxxx where x denotes the current sequence number modulo 10000000. I.e. 10.000.000 messages per second are possible before a collision occurs.

  • seq: The sequence number will be used.

  • seqhex: The hex encoded sequence number will be used.

Examples

This example will generate a payload checksum and store it to a metadata field called "checksum". 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - formatter.Identifier
      Generator: hash
      ApplyTo: checksum

func (*Identifier) ApplyFormatter added in v0.5.0 

func (format *Identifier) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Identifier) Configure 

func (format *Identifier) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type JSONToArray added in v0.5.0 

type JSONToArray struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

JSONToArray formatter plugin

JSONToArray "flattens" a JSON object by selecting specific fields and creating a delimiter-separated string of their values.

A JSON input like `{"foo":"value1","bar":"value2"}` can be transformed into a list like `value1,value2`.

Parameters

- Fields: List of strings specifying the JSON keys to retrieve from the input

- Separator: The delimited string to insert between each value in the generated string. By default this parameter is set to ",".

Examples

This example get the `foo` and `bar` fields from a json document and create a payload of `foo_value:bar_value`: 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.JSONToArray
      Separator: ;
      Fields:
        - foo
        - bar

func (*JSONToArray) ApplyFormatter added in v0.5.0 

func (format *JSONToArray) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*JSONToArray) Configure added in v0.5.0 

func (format *JSONToArray) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type JSONToInflux10 added in v0.5.0 

type JSONToInflux10 struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

JSONToInflux10 formatter

JSONToInflux10 provides a transformation from arbitrary JSON data to InfluxDB 0.9.1+ compatible line protocol data.

Parameters

- TimeField: Specifies the JSON field that holds the timestamp of the message. The timestamp is formatted as defined by TimeFormat. If the field is not found the current timestamp is assumed. By default this parameter is set to "time".

- TimeFormat: Specifies the format of the time field as in go's time.Parse or "unix" if the field contains a valid unix timestamp. By default this parameter is set to "unix".

- Measurement: Specifies the JSON field that holds the measurements in this message. If the field doesn't exist, the message is discarded. By default this parameter is set to "measurement".

- Ignore: May contain a list of JSON fields that should be ignored and not sent to InfluxDB. By default this parameter is set to an empty list.

- Tags: May contain a list of JSON fields to send to InfluxDB as tags. The InfluxDB 0.9 convention is that values that do not change by every request are to be considered metadata and given as tags.

Examples 

metricsToInflux:
  Type: producer.InfluxDB
  Streams: metrics
  Host: "influx01:8086"
  Database: "metrics"
  Modulators:
    - format.JSONToInflux10
      TimeField: timestamp
      Measurement: metrics
      Tags:
        - tags
        - service
        - host
        - application

func (*JSONToInflux10) ApplyFormatter added in v0.5.0 

func (format *JSONToInflux10) ApplyFormatter(msg *core.Message) error

ApplyFormatter updates the message payload

func (*JSONToInflux10) Configure added in v0.5.0 

func (format *JSONToInflux10) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type MetadataCopy added in v0.5.0 

type MetadataCopy struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

MetadataCopy formatter plugin

This formatter sets metadata fields by copying data from the message's payload or from other metadata fields.

Parameters

- Key: Defines the key to copy, i.e. the "source". ApplyTo will define the target of the copy, i.e. the "destination". An empty string will use the message payload as source. By default this parameter is set to an empty string (i.e. payload).

- Mode: Defines the copy mode to use. This can be one of "append", "prepend" or "replace". By default this parameter is set to "replace".

- Separator: When using mode prepend or append, defines the characters inserted between source and destination. By default this parameter is set to an empty string.

- CopyToKeys: DEPRECATED. A list of meta data keys to copy the payload or metadata content to. If this field contains at least one value, mode is set to replace and the key field is ignored. By default this parameter is set to an empty list.

Examples

This example copies the payload to the field key and applies a hash on it contain a hash over the complete payload. 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.MetadataCopy:
      ApplyTo: key
    - formatter.Identifier
      Generator: hash
      ApplyTo: key

func (*MetadataCopy) ApplyFormatter added in v0.5.0 

func (format *MetadataCopy) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*MetadataCopy) Configure added in v0.5.0 

func (format *MetadataCopy) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type ProcessJSON added in v0.4.2 

type ProcessJSON struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

ProcessJSON formatter

This formatter allows modification of JSON encoded data. Each field can be processed by different directives and the result of all directives will be stored back to the original location.

Parameters

- GeoIPFile: Defines a GeoIP file to load. This enables the "geoip" directive. If no file is loaded IPs will not be resolved. Files can be found e.g. at http://dev.maxmind.com/geoip/geoip2/geolite2/. By default this parameter is set to "".

- TrimValues: Allows trimming of whitespaces from the beginning and end of each value after processing took place. By default this parameter is set to true.

- Directives: Defines an array of actions to be applied to the JSON encoded data. Directives are processed in order of their appearance. Directives start with the name of the field, followed by an action followed by additional parameters if necessary. Parameters, key and action are separated by using the ":" character. By default this parameter is set to an empty list.

  • split: <delimiter> {<key>, <key>, ...} Split the field's value by the given delimiter, store the results to the fields listed after the delimiter.

  • replace: <string> <new string> Replace a given string inside the field's value with a new one.

  • trim: <characters> Remove the given characters from the start and end of the field's value.

  • rename: <new key> Rename a given field

  • remove: {<value>, <value>, ...}` Remove a given field. If additional parameters are given, the value is expected to be an array. The given strings will be removed from that array.

  • pick: <index> <key> Pick a specific index from an array and store it to the given field.

  • time: <from fromat> <to format> Read a timestamp with a given format compatible to time.Parse and transform it into another format compatible with time.Format.

  • unixtimestamp: <unit> <to format> Read a unix timestamp with a given unit ("s","ms" or "ns") and transform it it into another format compatible with time.Format.

  • flatten: {<delimiter>} Move all keys from a nested object to new fields named field + delimiter + subfield. If no delimiter is given "." will be used.

  • agent: <prefix> {<field>, <field>, ...} Parse the field's value as a user agent string and extract the given fields into new fields named prefix + "_" + field. If no fields are given all fields are returned.

  • mozilla: mozilla version

  • platform: the platform used

  • os: the operating system used

  • localization: the language used

  • engine: codename of the browser engine

  • engine_version: version of the browser engine

  • browser: name of the browser

  • version: version of the browser

  • ip: Parse the field as an array of strings and remove all values that cannot be parsed as a valid IP. Single-string fields are supported, too, but will be converted to an array.

  • geoip: {<field>, <field>, ...} Parse the field as an IP and extract the given fields into new fields named prefix + "_" + field. This action requires a valid GeoIP file to be loaded. If no fields are given all fields are returned.

  • country: the contry code of the IP. Generates country, countryCode.

  • city: the city of the IP

  • continent: the continent of the IP. Generates continent, continentCode.

  • timezone: the timezome of the IP

  • proxy: name of the proxy if applying Generates proxy, satellite.

  • location: the geolocation of this IP. Generates geocoord, geohash.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.ProcessJSON:
      Directives:
        - "host:split: :host:@timestamp"
        - "@timestamp:time:20060102150405:2006-01-02 15\\:04\\:05"
        - "client:ip"
        - "client:geoip:location:country"
        - "ua:agent:ua:os:engine:engine_version"

func (*ProcessJSON) ApplyFormatter added in v0.5.0 

func (format *ProcessJSON) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*ProcessJSON) Configure added in v0.4.2 

func (format *ProcessJSON) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type ProcessTSV added in v0.4.4 

type ProcessTSV struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

ProcessTSV formatter

This formatter allows modification of TSV encoded data. Each field can be processed by different directives and the result of all directives will be stored back to the original location.

Parameters

- Delimiter: Defines the separator used to split values. By default this parameter is set to "\t".

- QuotedValue: When set to true, values that start and end with a quotation mark are not scanned for delimiter characters. I.e. those values will not be split even if they contain delimiter characters. By default this parameter is set to false.

- Directives: Defines an array of actions to apply to the TSV encoded data. Directives are processed in order of appearance. Directives start with the index of the field, followed by an action followed by additional parameters if necessary. Parameters, key and action are separated by the ":" character. By default this parameter is set to an empty list.

  • replace: <string> <new string> Replace a given string inside the field's value with another one.

  • prefix: <string> Prepend the given string to the field's value

  • postfix: <string> Append the given string to the field's value

  • trim: <characters> Remove the given characters from the start and end of the field's value.

  • quote: Surround the field's value with quotation marks after all directives have been processed.

  • time: <from fromat> <to format> Read a timestamp in the specified time.Parse-compatible format and transform it into another format compatible with time.Format.

  • remove Removes the field from the result

  • agent: {<field>, <field>, ...} Parse the field's value as a user agent string and insert the given fields into the TSV after the given index. If no fields are given all fields are returned.

  • mozilla: mozilla version

  • platform: the platform used

  • os: the operating system used

  • localization: the language used

  • engine: codename of the browser engine

  • engine_version: version of the browser engine

  • browser: name of the browser

  • version: version of the browser

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.processTSV:
      Delimiter: ","
      Directives:
        - "0:time:20060102150405:2006-01-02 15\\:04\\:05"
        - "2:remove"
        - "11:agent:os:engine:engine_version"

func (*ProcessTSV) ApplyFormatter added in v0.5.0 

func (format *ProcessTSV) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*ProcessTSV) Configure added in v0.4.4 

func (format *ProcessTSV) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type RegExp added in v0.5.0 

type RegExp struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

RegExp formatter

This formatter parses a message using a regular expression, performs string (template) replacement and returns the result.

Parameters

- Posix: Set to true to compile the regular expression using posix semantics. By default this parameter is set to true.

- Expression: Defines the regular expression used for parsing. For details on the regexp syntax see https://golang.org/pkg/regexp/syntax. By default this parameter is set to "(.*)"

- Template: Defines the result string. Regexp matching groups can be referred to using "${n}", with n being the group's index. For other possible reference semantics, see https://golang.org/pkg/regexp/#Regexp.Expand. By default this parameter is set to "${1}"

Examples

This example extracts time and host from an imaginary log message format. 

exampleConsumer:
  Type: consumer.Console
  Streams: stding
  Modulators:
    - format.RegExp:
      Expression: "^(\\d+) (\\w+): "
      Template: "time: ${1}, host: ${2}"

func (*RegExp) ApplyFormatter added in v0.5.0 

func (format *RegExp) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*RegExp) Configure added in v0.5.0 

func (format *RegExp) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Runlength 

type Runlength struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Runlength formatter plugin

Runlength is a formatter that prepends the length of the message, followed by a ":". The actual message is formatted by a nested formatter.

Parameters

- Separator: This value is used as separator. By default this parameter is set to ":".

- StoreRunlengthOnly: If this value is set to "true" only the runlength will stored. This option is useful to e.g. create metadata fields only containing the length of the payload. When set to "true" the Separator parameter will be ignored. By default this parameter is set to false.

Examples

This example will store the length of the payload in a separate metadata field. 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.MetadataCopy:
      CopyToKeys: ["length"]
    - format.Runlength:
      ApplyTo: length
      StoreRunlengthOnly: true

func (*Runlength) ApplyFormatter added in v0.5.0 

func (format *Runlength) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Runlength) Configure 

func (format *Runlength) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Sequence 

type Sequence struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Sequence formatter

This formatter prefixes data with a sequence number managed by the formatter. All messages passing through an instance of the formatter will get a unique number. The number is not persisted, i.e. it restarts at 0 after each restart of gollum.

Parameters

- Separator: Defines the separator string placed between number and data. By default this parameter is set to ":".

Examples

This example will insert the sequence number into an existing JSON payload. 

exampleProducer:
  Type: producer.Console
  Streams: "*"
  Modulators:
    - format.Trim:
      LeftSeparator: "{"
      RightSeparator: "}"
    - format.Sequence
      Separator: ","
    - format.Envelope:
      Prefix: "{\"seq\":"
      Postfix: "}"

func (*Sequence) ApplyFormatter added in v0.5.0 

func (format *Sequence) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Sequence) Configure 

func (format *Sequence) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Serialize added in v0.4.0 

type Serialize struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
}

Serialize formatter plugin

Serialize is a formatter that serializes a message for later retrieval. The formatter uses the internal protobuf based function from msg.Serialize().

Examples

This example serializes all consumed messages: 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.Serialize

func (*Serialize) ApplyFormatter added in v0.5.0 

func (format *Serialize) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Serialize) Configure added in v0.4.0 

func (format *Serialize) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type SplitPick added in v0.4.5 

type SplitPick struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

SplitPick formatter

This formatter splits data into an array by using the given delimiter and extracts the given index from that array. The value of that index will be written back.

Parameters

- Delimiter: Defines the delimiter to use when splitting the data. By default this parameter is set to ":"

- Index: Defines the index to pick. By default this parameter is set to 0.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.SplitPick:
      Index: 2
      Delimiter: ":"

func (*SplitPick) ApplyFormatter added in v0.5.0 

func (format *SplitPick) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*SplitPick) Configure added in v0.4.5 

func (format *SplitPick) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type SplitToJSON added in v0.4.2 

type SplitToJSON struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

SplitToJSON formatter plugin

SplitToJSON is a formatter that splits a message by a given token and creates a JSON object of the split values by assigning each value to a predefined property.

Parameters

- Keys: This value defines an array of JSON keys to which the split message's parts should be assigned to. The keys are applied to the resulting token array by index.

- SplitBy: This value defines the separator character to use when splitting a message. By default this parameter is set to "|".

- KeepJSON: This value can be set to "false" to escape JSON payload texts as regualar strings. Otherwise JSON payload will be taken as-is and set to the corresponding key. By default this parameter is set to "true".

Examples

This example will format a input of `value1,value2,value3` to a json string of `{"foo":"value1", "bar":"value2"}`: 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.SplitToJSON:
      SplitBy: ","
      Keys:
        - foo
        - bar

func (*SplitToJSON) ApplyFormatter added in v0.5.0 

func (format *SplitToJSON) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*SplitToJSON) Configure added in v0.4.2 

func (format *SplitToJSON) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type StreamName added in v0.4.0 

type StreamName struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

StreamName formatter

This formatter prefixes data with the name of the current or previous stream.

Parameters

- UsePrevious: Set to true to use the name of the previous stream. By default this parameter is set to false.

- Separator: Defines the separator string used between stream name and data. By default this parameter is set to ":".

Examples

This example prefixes the message with the most recent routing history. 

exampleProducer:
  Type: producer.Console
  Streams: "*"
  Modulators:
    - format.StreamName:
      Separator: ", "
      UsePrevious: true
    - format.StreamName:
      Separator: ": "

func (*StreamName) ApplyFormatter added in v0.5.0 

func (format *StreamName) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*StreamName) Configure added in v0.4.0 

func (format *StreamName) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type StreamRevert added in v0.4.0 

type StreamRevert struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
}

StreamRevert formatter

This formatter gets the previously used stream from a message and sets it as the new target stream.

Examples 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.StreamRevert

func (*StreamRevert) ApplyFormatter added in v0.5.0 

func (format *StreamRevert) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*StreamRevert) Configure added in v0.4.0 

func (format *StreamRevert) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type StreamRoute added in v0.4.0 

type StreamRoute struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

StreamRoute formatter plugin

StreamRoute is a formatter that modifies a message's stream by reading a prefix from the message's data (and discarding it). The prefix is defined as everything before a given delimiter in the message. If no delimiter is found or the prefix is empty the message stream is not changed.

Parameters

- Delimiter: This value defines the delimiter to search when extracting the stream name. By default this parameter is set to ":".

- StreamModulator: A list of further modulators to format and filter the extracted stream name. By default this parameter is "empty".

Examples

This example sets the stream name for messages like `<error>:a message string` to `error` and `a message string` as payload: 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.StreamRoute:
      Delimiter: ":"
      StreamModulator:
        - format.Trim:
          LeftSeparator: <
          RightSeparator: >

func (*StreamRoute) ApplyFormatter added in v0.5.0 

func (format *StreamRoute) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*StreamRoute) Configure added in v0.4.0 

func (format *StreamRoute) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type TemplateJSON added in v0.4.5 

type TemplateJSON struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

TemplateJSON formatter

This formatter unmarshals the given data as JSON and applies the results to the given go template. The JSON data will be replaced with the rendered template result. The template language is described in the go documentation: https://golang.org/pkg/text/template/#hdr-Actions

Parameters

- Template: Defines the go template to execute with the received JSON data. If the template cannot be parsed or the JSON payload cannot be unmarshaled, the incoming JSON data is preserved. By default this parameter is set to "".

Examples

This example extracts the fields "Name" and "Surname" from a JSON encoded payload and writes them both back as a plain text result. 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.TemplateJSON:
      Template: "{{.Name}} {{.Surname}}"

func (*TemplateJSON) ApplyFormatter added in v0.5.0 

func (format *TemplateJSON) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*TemplateJSON) Configure added in v0.4.5 

func (format *TemplateJSON) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type TextToJSON added in v0.5.0 

type TextToJSON struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

TextToJSON formatter

This formatter uses a state machine to parse arbitrary text data and transform it to JSON.

Parameters

- StartState: Defines the name of the initial state when parsing a message. When set to an empty string the first state from the directives array will be used. By default this parameter is set to "".

- TimestampRead: Defines a time.Parse compatible format string used to read time fields when using the "dat" directive. By default this parameter is set to "20060102150405".

- TimestampWrite: Defines a time.Format compatible format string used to write time fields when using the "dat" directive. By default this parameter is set to "2006-01-02 15:04:05 MST".

- UnixTimestampRead: Defines the unix timestamp format expected from fields that are parsed using the "dat" directive. Valid valies are "s" for seconds, "ms" for milliseconds, or "ns" for nanoseconds. This parameter is ignored unless TimestampRead is set to "". By default this parameter is set to "".

- Directives: Defines an array of directives used to parse text data. Each entry must be of the format: "State:Token:NextState:Flags:Function". State denotes the name of the state owning this entry. Multiple entries per state are allowed. Token holds a string that triggers a state transition. NextState holds the target of the state transition. Flags is an optional field and is used to trigger special parser behavior. Flags can be comma separated if you need to use more than one. Function defines an action that is triggered upon state transition. Spaces will be stripped from all fields but Token. If a fields requires a colon it has to be escaped with a backslash. Other escape characters supported are \n, \r and \t. By default this parameter is set to an empty list.

- Directive rules: There are some special cases which will cause the parser to do additional actions.

* When writing a value without a key, the state name will become the key. * If two keys are written in a row the first key will hold a null value. * Writing a key while writing array elements will close the array.

- Directive flags: Flags can modify the parser behavior and can be used to store values on a stack across multiple directives.

  • continue: Prepend the token to the next match.

  • append: Append the token to the current match and continue reading.

  • include: Append the token to the current match.

  • push: Push the current state to the stack.

  • pop: Pop the stack and use the returned state if possible.

- Directive actions: Actions are used to write text read since the last transition to the JSON object.

  • key: Write the parsed section as a key.

  • val: Write the parsed section as a value without quotes.

  • esc: Write the parsed section as a escaped string value.

  • dat: Write the parsed section as a timestamp value.

  • arr: Start a new array.

  • obj: Start a new object.

  • end: Close an array or object.

  • arr+val: arr followed by val.

  • arr+esc: arr followed by esc.

  • arr+dat: arr followed by dat.

  • val+end: val followed by end.

  • esc+end: esc followed by end.

  • dat+end: dat followed by end.

Examples

The following example parses JSON data. 

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.TextToJSON:
      Directives:
        - "findKey   :\":  key       :      :        "
        - "findKey   :}:             : pop  : end    "
        - "key       :\":  findVal   :      : key    "
        - "findVal   :\\:: value     :      :        "
        - "value     :\":  string    :      :        "
        - "value     :[:   array     : push : arr    "
        - "value     :{:   findKey   : push : obj    "
        - "value     :,:   findKey   :      : val    "
        - "value     :}:             : pop  : val+end"
        - "string    :\":  findKey   :      : esc    "
        - "array     :[:   array     : push : arr    "
        - "array     :{:   findKey   : push : obj    "
        - "array     :]:             : pop  : val+end"
        - "array     :,:   array     :      : val    "
        - "array     :\":  arrString :      :        "
        - "arrString :\":  array     :      : esc    "

func (*TextToJSON) ApplyFormatter added in v0.5.0 

func (format *TextToJSON) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*TextToJSON) Configure added in v0.5.0 

func (format *TextToJSON) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Timestamp 

type Timestamp struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Timestamp formatter plugin

Timestamp is a formatter that allows prefixing messages with a timestamp (time of arrival at gollum). The timestamp format is freely configurable and can e.g. contain a delimiter sequence at the end.

Parameters

- Timestamp: This value defines a Go time format string that is used to f ormat the timestamp. By default this parameter is set to "2006-01-02 15:04:05 MST | ".

Examples

This example will set a time string to the meta data field `time`: 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.Timestamp:
      Timestamp: "2006-01-02T15:04:05.000 MST"
      ApplyTo: time

func (*Timestamp) ApplyFormatter added in v0.5.0 

func (format *Timestamp) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Timestamp) Configure 

func (format *Timestamp) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

type Trim added in v0.5.0 

type Trim struct {
	core.SimpleFormatter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Trim formatter

This formatter searches for separator strings and removes all data left or right of this separator.

Parameters

- LeftSeparator: The string to search for. Searching starts from the left side of the data. If an empty string is given this parameter is ignored. By default this parameter is set to "".

- RightSeparator: The string to search for. Searching starts from the right side of the data. If an empty string is given this parameter is ignored. By default this parameter is set to "".

- LeftOffset: Defines the search start index when using LeftSeparator. By default this parameter is set to 0.

- RightOffset: Defines the search start index when using RightSeparator. Counting starts from the right side of the message. By default this parameter is set to 0.

Examples

This example will reduce data like "foo[bar[foo]bar]foo" to "bar[foo]bar". 

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.Trim:
      LeftSeparator: "["
      RightSeparator: "]"

func (*Trim) ApplyFormatter added in v0.5.0 

func (format *Trim) ApplyFormatter(msg *core.Message) error

ApplyFormatter update message payload

func (*Trim) Configure added in v0.5.0 

func (format *Trim) Configure(conf core.PluginConfigReader)

Configure initializes this formatter with values from a plugin config.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.