parsesyslog

README

go-parsesyslog - a Go library to parse syslog messages

Supported formats

BSD syslog format (RFC3164)

go-parsesyslog fully implements the RFC3164 format including timestamp parsing and optional tags.

Please note: the RFC is not providing any message length definition and explicity states that there is "no ending delimiter to this part" for this reason we are using the newline (\n (ASCII: 10)) as delimiter. This will therefore truncate messages that have a newline in it. Additionally the RFC does specify a timestamp format that has not provide any information about the year. For this reason, we will interpret the year for the message as the current year.

Available fields in the LogMsg:

• AppName: this represents the TAG part of TAG[pid]: format (if given in the message) that is often used for the name off the application or process logging
• ProcID: this represents the pid part of TAG[pid]: format (if given in the message) that is often used for the process ID
• HostName: this represents the hostname part of the RFC3164 message
• Priority: The Priority part of the message
• Facility: The facility calculated from the Priority part of the message
• Severity: The severity calculated from the Priority part of the message
• Timestamp: The parsed timestamp of the RFC3164 message as time.Time representation
• Message: The message part of the log message as bytes.Buffer
• MsgLength: The length of the Message (not including any header part)
• Type: This will be always set to RFC3164

IETF-syslog

go-parsesyslog is also fully (RFC5424) compliant. All available fields are parsed and represented accordingly in the LogMsg fields. Although the RFC5424 mandates a maximum length of 2048 bytes for a log message, go-parsesyslog does only obey the message length given in the header of the message.

Available fields in the LogMsg:

• Priority: this represents the PRI field of the header
• ProtoVersion: this represents the VERSION field of the header
• Timestamp: this represents the TIMESTAMP field of the header
• Hostname: this represents the HOSTNAME field of the header
• AppName: this represents the APP-NAME field of the header
• ProcID: this represents the PROCID field of the header
• MsgID: this represents the MSGID field of the header
• StructuredData this represents fully parsed structured data as described in the STRUCTURED-DATA section of the RFC
• HasBOM: is set to true if the log message starts with a BOM
• Facility: The facility calculated from the Priority part of the message
• Severity: The severity calculated from the Priority part of the message
• Message: The message part of the log message as bytes.Buffer
• MsgLength: The length of the Message (not including any header part)
• Type: This will be always set to RFC5424

Usage

go-parsesyslog implements an interface for various syslog formats, which makes it easy to extend your own log parser. As long as the Parser interface is satisfied, go-parsesyslog will be able to work.

The interface looks as following:

type Parser interface {
  ParseReader(io.Reader) (LogMsg, error)
  ParseString(string) (LogMsg, error)
}

Parsing logs

As you can see, the ParseReader() method expects an io.Reader interface as argument. This allows you to easily parse your logs from any kind of source (STDIN, a file, a network socket...). ParseString() instead takes a string and parses it accordingly.

Parsing RFC3164

This example code show how to parse a RFC3164 conformant message:

package main

import (
	"fmt"
	"github.com/wneessen/go-parsesyslog"
	"github.com/wneessen/go-parsesyslog/rfc3164"
	"os"
)

func main() {
	msg := "<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8\n"
	p, err := parsesyslog.New(rfc3164.Type)
	if err != nil {
		fmt.Printf("failed to create RFC3164 parser: %s", err)
		os.Exit(1)
	}
	lm, err := p.ParseString(msg)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Log message: %+v", lm)
}

Parsing RFC5424

This example code show how to parse a RFC5424 conformant message:

package main

import (
	"fmt"
	"github.com/wneessen/go-parsesyslog"
	"github.com/wneessen/go-parsesyslog/rfc5424"
	"os"
)

func main() {
	msg := `197 <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][foo@1234 foo="bar" blubb="bluh"] \xEF\xBB\xBFAn application event log entry..."`
	p, err := parsesyslog.New(rfc5424.Type)
	if err != nil {
		fmt.Printf("failed to create RFC3164 parser: %s", err)
		os.Exit(1)
	}
	lm, err := p.ParseString(msg)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Log message: %+v", lm)
}

An example implementation can be found in cmd/stdin-parser

$ $ echo -ne '197 <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][foo@1234 foo="bar" blubb="bluh"] \xEF\xBB\xBFAn application event log entry...' | go run github.com/wneessen/go-parsesyslog/cmd/stdin-parser

This command will output:

Log message details:
+ Log format:         RFC5424
+ Header:
  - Priority:         165 (Facility: LOCAL4 / Severity: NOTICE)
  - Protocol Version: 1
  - Hostname:         mymachine.example.com
  - AppName:          evntslog
  - ProcID:
  - MsgID:            ID47
  - Timestamp (UTC):  2003-10-11 22:14:15.003 +0000 UTC
+ Structured Data:
  - ID:               exampleSDID@32473
    + Param 0:
      - Name:         iut
      - Value:        3
    + Param 1:
      - Name:         eventSource
      - Value:        Application
    + Param 2:
      - Name:         eventID
      - Value:        1011
  - ID:               foo@1234
    + Param 0:
      - Name:         foo
      - Value:        bar
    + Param 1:
      - Name:         blubb
      - Value:        bluh
+ Message has BOM:    true
+ Message Length:     25
+ Message:            An application event l

Log parsed in 18.745µs

Benchmark

As the main intention of this library was for me to use it in a network service that parses incoming syslog messages, quite some work has been invested to make go-parsesyslog fast and memory efficient. We are trying to allocate as less as possible and make use of buffered I/O where possible.

$ go test -run=X -bench=.\*ParseReader -benchtime=5s ./...
goos: linux
goarch: amd64
pkg: github.com/wneessen/go-parsesyslog
cpu: AMD Ryzen 9 3950X 16-Core Processor
BenchmarkRFC3164Msg_ParseReader-2        7971660               748.9 ns/op            96 B/op          4 allocs/op
BenchmarkRFC5424Msg_ParseReader-2        3458671              1734 ns/op            1144 B/op         16 allocs/op
PASS

Documentation

Overview

Package parsesyslog implements a syslog message parser for different RFC log formats

Index

• Constants
• Variables
• func Atoi(b []byte) (int, error)
• func FacilityStringFromPrio(p Priority) string
• func ParsePriority(r *bufio.Reader, buf *bytes.Buffer, lm *LogMsg) error
• func ReadBytesUntilSpace(r *bufio.Reader) ([]byte, int, error)
• func ReadBytesUntilSpaceOrNilValue(r *bufio.Reader, buf *bytes.Buffer) (int, error)
• func ReadMsgLength(r *bufio.Reader) (int, error)
• func Register(t ParserType, fn func() (Parser, error))
• func SeverityStringFromPrio(p Priority) string
• type Facility
  • func FacilityFromPrio(p Priority) Facility
  • func (f Facility) String() string
• type LogMsg
• type LogMsgType
• type Parser
  • func New(t ParserType) (Parser, error)
• type ParserType
• type Priority
• type ProtoVersion
• type Severity
  • func SeverityFromPrio(p Priority) Severity
  • func (s Severity) String() string
• type StructuredDataElement
• type StructuredDataParam

Constants

const SeverityMask = 0x07

SeverityMask is the bitmask representing the Severity in the Priority

Variables

var (
	// ErrInvalidPrio should be used if the PRI part of the message is not following the log format
	ErrInvalidPrio = errors.New("PRI header not a valid priority string")
	// ErrInvalidProtoVersion should be used if the protocol version part of the header is not following the log format
	ErrInvalidProtoVersion = errors.New("protocol version string invalid")
	// ErrInvalidTimestamp should be used if it was not possible to parse the timestamp of the log message
	ErrInvalidTimestamp = errors.New("timestamp does not conform the logging format")
	// ErrParserTypeUnknown is returned if a Parser is requested via New() which is not registered
	ErrParserTypeUnknown = errors.New("unknown parser type")
	// ErrPrematureEOF should be used in case a log message ends before the provided length
	ErrPrematureEOF = errors.New("log message is shorter than the provided length")
	// ErrWrongFormat should be used if a log messages does not comply with the logging format definitions
	ErrWrongFormat = errors.New("log message does not conform the logging format")
	// ErrWrongSDFormat should be used in case the structured data is not parsable
	ErrWrongSDFormat = errors.New("structured data does not conform the format")
)

Functions

func Atoi

func Atoi(b []byte) (int, error)

Atoi performs allocation free ASCII number to integer conversion

func FacilityStringFromPrio

func FacilityStringFromPrio(p Priority) string

FacilityStringFromPrio returns a string representation of the Facility of a given Priority

func ParsePriority

func ParsePriority(r *bufio.Reader, buf *bytes.Buffer, lm *LogMsg) error

ParsePriority will try to parse the priority part of the RFC3164 header See: https://tools.ietf.org/search/rfc3164#section-4.1.1

func ReadBytesUntilSpace

func ReadBytesUntilSpace(r *bufio.Reader) ([]byte, int, error)

ReadBytesUntilSpace is a helper method that takes a io.Reader and reads all bytes until it hits a Space character. It returns the read bytes, the amount of bytes read and an error if one occurred

func ReadBytesUntilSpaceOrNilValue

func ReadBytesUntilSpaceOrNilValue(r *bufio.Reader, buf *bytes.Buffer) (int, error)

ReadBytesUntilSpaceOrNilValue is a helper method that takes a io.Reader and reads all bytes until it hits a Space character or the NILVALUE ("-"). It returns the read bytes, the amount of bytes read and an error if one occurred

func ReadMsgLength

func ReadMsgLength(r *bufio.Reader) (int, error)

ReadMsgLength reads the first bytes of the log message which represent the total length of the log message

func Register

func Register(t ParserType, fn func() (Parser, error))

Register registers a new ParserType with its corresponding Parser function.

func SeverityStringFromPrio

func SeverityStringFromPrio(p Priority) string

SeverityStringFromPrio returns a string representation of the Severity of a given Priority

Types

type Facility

type Facility int

Facility represents the facility part of the Syslog PRI header

func FacilityFromPrio

func FacilityFromPrio(p Priority) Facility

FacilityFromPrio extracts the Facility from a given Priority

func (Facility) String

func (f Facility) String() string

String satisfies the fmt.Stringer interface for the Facility type

type LogMsg

type LogMsg struct {
	AppName  string
	Facility Facility
	HasBOM   bool
	Hostname string
	// Message        []byte
	Message        bytes.Buffer
	MsgLength      int
	MsgID          string
	Priority       Priority
	ProcID         string
	ProtoVersion   ProtoVersion
	Severity       Severity
	StructuredData []StructuredDataElement
	Timestamp      time.Time
	Type           LogMsgType
}

LogMsg represents a parsed syslog message

type LogMsgType

type LogMsgType string

LogMsgType represents the type of message

const (
	RFC3164 LogMsgType = "RFC3164" // RFC3164: legacy BSD-syslog
	RFC5424 LogMsgType = "RFC5424" // RFC5424: modern IETF-syslog
)

LogMsgTypes

type Parser

type Parser interface {
	ParseReader(io.Reader) (LogMsg, error)
	ParseString(s string) (LogMsg, error)
}

Parser is an interface for parsing log messages.

func New

func New(t ParserType) (Parser, error)

New returns a Parser of the specified ParserType and an error. It looks up the ParserType in the types map and if found, calls the corresponding Parser function to create a new Parser instance.

If the ParserType is not found in the map, it returns nil and ErrParserTypeUnknown.

type ParserType

type ParserType string

ParserType is a type of parser for logs messages

type Priority

type Priority int

Priority represents the Syslog PRI header

const (
	Emergency Priority = iota // System is unusable
	Alert                     // Action must be taken immediately
	Crit                      // Critical conditions
	Error                     // Error conditions
	Warning                   // Warning conditions
	Notice                    // Normal but significant conditions
	Info                      // Informational messages
	Debug                     // Debug-level messages
)

Severities

const (
	Kern        Priority = iota << 3 // Kernel messages
	User                             // User-level messages
	Mail                             // Mail system
	Daemon                           // System daemons
	Auth                             // Security/authentication messages
	Syslog                           // Messages generated internally by the syslog daemon
	LPR                              // Printer subsystem
	News                             // Network News subsystem
	UUCP                             // UUCP subsystem
	Cron                             // Cron subsystem
	AuthPriv                         // Security/authentication messages
	FTP                              // FTP daemon
	NTP                              // NTP subsystem
	Security                         // Log audit
	Console                          // Log alert
	SolarisCron                      // Scheduling daemon
	Local0                           // Locally used facilities
	Local1                           // Locally used facilities
	Local2                           // Locally used facilities
	Local3                           // Locally used facilities
	Local4                           // Locally used facilities
	Local5                           // Locally used facilities
	Local6                           // Locally used facilities
	Local7                           // Locally used facilities
)

Facilities

type ProtoVersion

type ProtoVersion int

ProtoVersion represents the version of message

type Severity

type Severity int

Severity represents the serverity part of the Syslog PRI header

func SeverityFromPrio

func SeverityFromPrio(p Priority) Severity

SeverityFromPrio extracts the Facility from a given Priority

func (Severity) String

func (s Severity) String() string

String satisfies the fmt.Stringer interface for the Severity type

type StructuredDataElement

type StructuredDataElement struct {
	ID    string
	Param []StructuredDataParam
}

StructuredDataElement represents a structured data elements as defined in RFC5424 See: https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.1

type StructuredDataParam

type StructuredDataParam struct {
	Name  string
	Value string
}

StructuredDataParam represents a structured data param pair as defined in RFC5424 See: https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3