systemlogmonitor

README

System Log Monitor

System Log Monitor is a problem daemon in node problem detector. It monitors specified system daemon log and detects problems following predefined rules.

The System Log Monitor matches problems according to a set of predefined rule list in the configuration files. ( config/kernel-monitor.json as an example). The rule list is extensible.

Limitations

• System Log Monitor only supports file based log and journald now, but it is easy to extend it with new log watcher

Add New NodeConditions

To support new node conditions, you can extend the conditions field in the configuration file with new condition definition:

{
  "type": "NodeConditionType",
  "reason": "CamelCaseDefaultNodeConditionReason",
  "message": "arbitrary default node condition message"
}

Detect New Problems

To detect new problems, you can extend the rules field in the configuration file with new rule definition:

{
  "type": "temporary/permanent",
  "condition": "NodeConditionOfPermanentIssue",
  "reason": "CamelCaseShortReason",
  "message": "regexp matching the issue in the log"
}

Log Watchers

System log monitor supports different log management tools with different log watchers:

• filelog: Log watcher for arbitrary file based log.
• journald: Log watcher for journald. Set plugin in the configuration file to specify log watcher.

Plugin Configuration

Log watcher specific configurations are configured in pluginConfig.

• journald
  • source: The SYSLOG_IDENTIFIER of the log to watch.
• filelog:
  • timestamp: The regular expression used to match timestamp in the log line. Submatch is supported, but only the last result will be used as the actual timestamp.
  • message: The regular expression used to match message in the log line. Submatch is supported, but only the last result will be used as the actual message.
  • timestampFormat: The format of the timestamp. The format string is the time 2006-01-02T15:04:05Z07:00 in the expected format. (See golang timestamp format)

Change Log Path

Log on different OS distros may locate in different path. The logPath field in the configurtion file is the log path. You can always configure logPath to match your OS distro.

• filelog: logPath is the path of log file, e.g. /var/log/kern.log for kernel log.
• journald: logPath is the journal log directory, usually /var/log/journal.

New Log Watcher

System log monitor uses Log Watcher to support different log management tools. It is easy to implement a new log watcher.

Documentation

Index

• func NewLogBuffer(maxLines int) *logBuffer
• type LogBuffer
• type LogMonitor
  • func NewLogMonitorOrDie(configPath string) LogMonitor
• type MonitorConfig

Functions

func NewLogBuffer

func NewLogBuffer(maxLines int) *logBuffer

NewLogBuffer creates log buffer with max line number limit. Because we only match logs in the log buffer, the max buffer line number is also the max pattern line number we support. Smaller buffer line number means less memory and cpu usage, but also means less lines of patterns we support.

Types

type LogBuffer

type LogBuffer interface {
	// Push pushes log into the log buffer.
	Push(*types.Log)
	// Match with regular expression in the log buffer.
	Match(string) []*types.Log
	// String returns a concatenated string of the buffered logs.
	String() string
}

LogBuffer buffers the logs and supports match in the log buffer with regular expression.

type LogMonitor

type LogMonitor interface {
	// Start starts the log monitor.
	Start() (<-chan *types.Status, error)
	// Stop stops the log monitor.
	Stop()
}

LogMonitor monitors the log and reports node problem condition and event according to the rules.

func NewLogMonitorOrDie

func NewLogMonitorOrDie(configPath string) LogMonitor

NewLogMonitorOrDie create a new LogMonitor, panic if error occurs.

type MonitorConfig

type MonitorConfig struct {
	// WatcherConfig is the configuration of log watcher.
	watchertypes.WatcherConfig
	// BufferSize is the size (in lines) of the log buffer.
	BufferSize int `json:"bufferSize"`
	// Source is the source name of the log monitor
	Source string `json:"source"`
	// DefaultConditions are the default states of all the conditions log monitor should handle.
	DefaultConditions []types.Condition `json:"conditions"`
	// Rules are the rules log monitor will follow to parse the log file.
	Rules []logtypes.Rule `json:"rules"`
}

MonitorConfig is the configuration of log monitor.