decoder

Documentation

Index

• type AutoMultilineHandler
  • func NewAutoMultilineHandler(outputFn func(*Message), lineLimit, linesToAssess int, matchThreshold float64, ...) *AutoMultilineHandler
• type Decoder
  • func InitializeDecoder(source *config.LogSource, parser parsers.Parser) *Decoder
  • func New(InputChan chan *Input, OutputChan chan *Message, ...) *Decoder
  • func NewDecoderFromSource(source *config.LogSource) *Decoder
  • func NewDecoderFromSourceWithPattern(source *config.LogSource, multiLinePattern *regexp.Regexp) *Decoder
  • func NewDecoderWithEndLineMatcher(source *config.LogSource, parser parsers.Parser, ...) *Decoder
  • func (d *Decoder) GetDetectedPattern() *regexp.Regexp
  • func (d *Decoder) GetLineCount() int64
  • func (d *Decoder) Start()
  • func (d *Decoder) Stop()
• type DetectedPattern
  • func (d *DetectedPattern) Get() *regexp.Regexp
  • func (d *DetectedPattern) Set(pattern *regexp.Regexp)
• type Input
  • func NewInput(content []byte) *Input
• type LineHandler
• type LineParser
• type Message
  • func NewMessage(content []byte, status string, rawDataLen int, timestamp string) *Message
• type MultiLineHandler
  • func NewMultiLineHandler(outputFn func(*Message), newContentRe *regexp.Regexp, ...) *MultiLineHandler
• type MultiLineParser
  • func NewMultiLineParser(outputFn func(*Message), flushTimeout time.Duration, parser parsers.Parser, ...) *MultiLineParser
• type SingleLineHandler
  • func NewSingleLineHandler(outputFn func(*Message), lineLimit int) *SingleLineHandler
• type SingleLineParser
  • func NewSingleLineParser(outputFn func(*Message), parser parsers.Parser) *SingleLineParser

Types

type AutoMultilineHandler

type AutoMultilineHandler struct {
	// contains filtered or unexported fields
}

AutoMultilineHandler can attempts to detect a known/commob pattern (a timestamp) in the logs and will switch to a MultiLine handler if one is detected and the thresholds are met.

func NewAutoMultilineHandler

func NewAutoMultilineHandler(
	outputFn func(*Message),
	lineLimit, linesToAssess int,
	matchThreshold float64,
	matchTimeout time.Duration,
	flushTimeout time.Duration,
	source *config.LogSource,
	additionalPatterns []*regexp.Regexp,
	detectedPattern *DetectedPattern,
) *AutoMultilineHandler

NewAutoMultilineHandler returns a new AutoMultilineHandler.

type Decoder

type Decoder struct {
	InputChan  chan *Input
	OutputChan chan *Message
	// contains filtered or unexported fields
}

Decoder translates a sequence of byte buffers (such as from a file or a network socket) into log messages.

Decoder is structured as an actor with InputChan of type *decoder.Input and OutputChan of type *decoder.Message.

The Decoder's run() takes data from InputChan, uses a LineBreaker to break it into lines. The LineBreaker passes that data to a LineParser, which uses a Parser to convert it to parsers.Message, converts that to decoder.Message, and passes that to the LineHandler.

The LineHandler processes the messages it as necessary (as single lines, multiple lines, or auto-detecting the two), and sends the result to the Decoder's output channel.

func InitializeDecoder

func InitializeDecoder(source *config.LogSource, parser parsers.Parser) *Decoder

InitializeDecoder returns a properly initialized Decoder

func New

func New(InputChan chan *Input, OutputChan chan *Message, lineBreaker *breaker.LineBreaker, lineParser LineParser, lineHandler LineHandler, detectedPattern *DetectedPattern) *Decoder

New returns an initialized Decoder

func NewDecoderFromSource

func NewDecoderFromSource(source *config.LogSource) *Decoder

NewDecoderFromSource creates a new decoder from a log source

func NewDecoderFromSourceWithPattern

func NewDecoderFromSourceWithPattern(source *config.LogSource, multiLinePattern *regexp.Regexp) *Decoder

NewDecoderFromSourceWithPattern creates a new decoder from a log source with a multiline pattern

func NewDecoderWithEndLineMatcher

func NewDecoderWithEndLineMatcher(source *config.LogSource, parser parsers.Parser, matcher breaker.EndLineMatcher, multiLinePattern *regexp.Regexp) *Decoder

NewDecoderWithEndLineMatcher initialize a decoder with given endline strategy.

func (*Decoder) GetDetectedPattern

func (d *Decoder) GetDetectedPattern() *regexp.Regexp

GetDetectedPattern returns a detected pattern (if any)

func (*Decoder) GetLineCount

func (d *Decoder) GetLineCount() int64

GetLineCount returns the number of decoded lines

func (*Decoder) Start

func (d *Decoder) Start()

Start starts the Decoder

func (*Decoder) Stop

func (d *Decoder) Stop()

Stop stops the Decoder

type DetectedPattern

type DetectedPattern struct {
	sync.Mutex
	// contains filtered or unexported fields
}

DetectedPattern is a container to safely access a detected multiline pattern

func (*DetectedPattern) Get

func (d *DetectedPattern) Get() *regexp.Regexp

Get gets the pattern

func (*DetectedPattern) Set

func (d *DetectedPattern) Set(pattern *regexp.Regexp)

Set sets the pattern

type Input

type Input struct {
	// contains filtered or unexported fields
}

Input represents a chunk of line.

func NewInput

func NewInput(content []byte) *Input

NewInput returns a new input.

type LineHandler

type LineHandler interface {
	// contains filtered or unexported methods
}

LineHandler handles raw lines to form structured lines.

type LineParser

type LineParser interface {
	// contains filtered or unexported methods
}

LineParser handles decoded lines, parsing them into decoder.Message's using an embedded parsers.Parser.

type Message

type Message struct {
	Content            []byte
	Status             string
	RawDataLen         int
	Timestamp          string
	IngestionTimestamp int64
}

Message represents a structured line.

func NewMessage

func NewMessage(content []byte, status string, rawDataLen int, timestamp string) *Message

NewMessage returns a new output.

type MultiLineHandler

type MultiLineHandler struct {
	// contains filtered or unexported fields
}

MultiLineHandler makes sure that multiple lines from a same content are properly put together.

func NewMultiLineHandler

func NewMultiLineHandler(outputFn func(*Message), newContentRe *regexp.Regexp, flushTimeout time.Duration, lineLimit int) *MultiLineHandler

NewMultiLineHandler returns a new MultiLineHandler.

type MultiLineParser

type MultiLineParser struct {
	// contains filtered or unexported fields
}

MultiLineParser makes sure that chunked lines are properly put together.

func NewMultiLineParser

func NewMultiLineParser(
	outputFn func(*Message),
	flushTimeout time.Duration,
	parser parsers.Parser,
	lineLimit int,
) *MultiLineParser

NewMultiLineParser returns a new MultiLineParser.

type SingleLineHandler

type SingleLineHandler struct {
	// contains filtered or unexported fields
}

SingleLineHandler takes care of tracking the line length and truncating them when they are too long.

func NewSingleLineHandler

func NewSingleLineHandler(outputFn func(*Message), lineLimit int) *SingleLineHandler

NewSingleLineHandler returns a new SingleLineHandler.

type SingleLineParser

type SingleLineParser struct {
	// contains filtered or unexported fields
}

SingleLineParser makes sure that multiple lines from a same content are properly put together.

func NewSingleLineParser

func NewSingleLineParser(
	outputFn func(*Message),
	parser parsers.Parser) *SingleLineParser

NewSingleLineParser returns a new SingleLineParser.