core

package
v1.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2025 License: BSD-2-Clause Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MaxNumLinesDefault is a default for QueryLogsParams.MaxNumLines below.
	MaxNumLinesDefault = 250
)
View Source 
const SpecialFilenameJournalctl = "journalctl"

Variables

View Source 
var ErrBusyWithAnotherQuery = errors.Errorf("busy with another query")
View Source 
var ErrNotYetConnected = errors.Errorf("not connected to all lstreams yet")
View Source 
var ValidSudoModes = map[SudoMode]struct{}{
	SudoModeNone: {},
	SudoModeFull: {},
}

Functions

func DetectTimeLayout 

func DetectTimeLayout(logLine string) string

DetectTimeLayout tries to detect a time format from a log line.

TODO: it's pretty simplistic and could be improved, even to avoid having a predefined set of known formats, but good enough for now.

func InferYear 

func InferYear(t time.Time) time.Time

InferYear infers year from the month of the given timestamp, and the current time. Resulting timestamp (with the year populated) is then returned.

Most of the time it just uses the current year, but on the year boundary it can return previous or next year.

Types

type BootstrapDetails 

type BootstrapDetails struct {
	// Err is an error message from the last bootstrap attempt.
	Err string

	// WarnJournalctlNoAdminAccess is set to true if journalctl is used and the
	// user doesn't have access to all the system logs. It's a separate bool
	// instead of a generic warning message to make it possible to suppress it
	// with a flag.
	WarnJournalctlNoAdminAccess bool
}

type BootstrapIssue 

type BootstrapIssue struct {
	LStreamName string
	Err         string

	// WarnJournalctlNoAdminAccess is set to true if journalctl is used and the
	// user doesn't have access to all the system logs. It's a separate bool
	// instead of a generic warning message to make it possible to suppress it
	// with a flag.
	WarnJournalctlNoAdminAccess bool
}

type BusyStage 

type BusyStage struct {
	// Num is just a stage number. Its meaning depends on the kind of command the
	// host is executing, but a general rule is that this number starts from 1
	// and then increases, as the process goes on, so we can compare different
	// nodes and e.g. find the slowest one.
	Num int

	// Title is just a human-readable description of the stage.
	Title string

	// ExtraInfo might contains some additional context, also just a
	// human-readable string.
	ExtraInfo string

	// Percentage is a percentage of the current stage.
	Percentage int
}

type ConfigHost 

type ConfigHost struct {
	// Addr is the address to connect to, in the same format which is used by
	// net.Dial. To copy-paste some docs from net.Dial: the address has the form
	// "host:port". The host must be a literal IP address, or a host name that
	// can be resolved to IP addresses. The port must be a literal port number or
	// a service name.
	//
	// Examples: "golang.org:http", "192.0.2.1:http", "198.51.100.1:22".
	Addr string
	// User is the username to authenticate as.
	User string
}

func (*ConfigHost) Key 

func (ch *ConfigHost) Key() string

type ConfigLogStream 

type ConfigLogStream struct {
	// HostAddr is the actual host to connect to.
	//
	// If empty, we'll resort to the ssh config, and if there's no info for that
	// host either, we'll try to get host addr from the key in
	// ConfigLogStreams.LogStreams.
	Hostname string `yaml:"hostname"`

	// Port is the actual port to connect to. If empty, the same overriding rules
	// apply.
	Port string `yaml:"port"`

	// User is the user to authenticate as. If empty, same overriding rules
	// apply.
	User string `yaml:"user"`

	// LogFiles contains a list of files which are part of the logstream, like
	// ["/var/log/syslog", "/var/log/syslog.1"]. The [0]th item is the latest log
	// file [1]st is the previous one, etc.
	//
	// During the final usage (after resolving everything), it must contain at
	// least a single item, otherwise LogStream is invalid. However in the configs,
	// it's optional (and eventually, if empty, will be set to default values by
	// the LStreamsResolver).
	LogFiles []string `yaml:"log_files"`

	Options ConfigLogStreamOptions `yaml:"options"`
}

type ConfigLogStreamOptions added in v1.2.0 

type ConfigLogStreamOptions struct {
	// Sudo is a shortcut for SudoMode: if Sudo is true, it's an equivalent of
	// setting SudoMode to SudoModeFull.
	Sudo bool `yaml:"sudo"`

	// SudoMode can be used to configure nerdlog to read log files with "sudo -n".
	// See constants for the SudoMode type for more details.
	SudoMode SudoMode `yaml:"sudo_mode"`
}

ConfigLogStreamOptions contains additional options for a particular logstream.

func (ConfigLogStreamOptions) EffectiveSudoMode added in v1.2.0 

func (opts ConfigLogStreamOptions) EffectiveSudoMode() SudoMode

EffectiveSudoMode returns the SudoMode considering all fields that can affect it: Sudo and SudoMode.

type ConfigLogStreamWKey 

type ConfigLogStreamWKey struct {
	// Key is the key at which the corresponding ConfigLogStream was
	// stored in the ConfigLogStreams map.
	Key string

	ConfigLogStream
}

type ConfigLogStreams 

type ConfigLogStreams map[string]ConfigLogStream

func (ConfigLogStreams) Keys 

func (lss ConfigLogStreams) Keys() []string

type ConnDetails 

type ConnDetails struct {
	// Err is an error message from the last connection attempt.
	Err string
}

type LStreamClient 

type LStreamClient struct {
	// contains filtered or unexported fields
}

func NewLStreamClient 

func NewLStreamClient(params LStreamClientParams) *LStreamClient

func (*LStreamClient) Close 

func (lsc *LStreamClient) Close(changeName string)

Close initiates the shutdown. It doesn't wait for the shutdown to complete; client code needs to wait for the corresponding event (with TornDown: true).

If changeName is non-empty, the LStreamClient's Name will be updated; it's useful to distinguish this LStreamClient from potentially-existing another one with the same (old) name.

func (*LStreamClient) EnqueueCmd 

func (lsc *LStreamClient) EnqueueCmd(cmd lstreamCmd)

func (*LStreamClient) Reconnect 

func (lsc *LStreamClient) Reconnect()

func (*LStreamClient) SendFoo 

func (lsc *LStreamClient) SendFoo()

type LStreamClientParams 

type LStreamClientParams struct {
	LogStream LogStream

	Logger *log.Logger

	// ClientID is just an arbitrary string (should be filename-friendly though)
	// which will be appended to the nerdlog_agent.sh and its index filenames.
	//
	// Needed to make sure that different clients won't get conflicts over those
	// files when using the tool concurrently on the same nodes.
	ClientID string

	UpdatesCh chan<- *LStreamClientUpdate
}

type LStreamClientState 

type LStreamClientState string
const (
	LStreamClientStateDisconnected  LStreamClientState = "disconnected"
	LStreamClientStateConnecting    LStreamClientState = "connecting"
	LStreamClientStateDisconnecting LStreamClientState = "disconnecting"
	LStreamClientStateConnectedIdle LStreamClientState = "connected_idle"
	LStreamClientStateConnectedBusy LStreamClientState = "connected_busy"
)

type LStreamClientUpdate 

type LStreamClientUpdate struct {
	Name string

	State *LStreamClientUpdateState

	ConnDetails      *ConnDetails
	BootstrapDetails *BootstrapDetails
	BusyStage        *BusyStage

	// If TornDown is true, it means it's the last update from that client.
	TornDown bool
}

LStreamClientUpdate represents an update from logstream client. Name is always populated and it's the logstream's name, and from all the other fields, exactly one field must be non-nil.

type LStreamClientUpdateState 

type LStreamClientUpdateState struct {
	OldState LStreamClientState
	NewState LStreamClientState
}

type LStreamsManager 

type LStreamsManager struct {
	// contains filtered or unexported fields
}

func NewLStreamsManager 

func NewLStreamsManager(params LStreamsManagerParams) *LStreamsManager

func (*LStreamsManager) Close 

func (lsman *LStreamsManager) Close()

Close initiates the shutdown. It doesn't wait for the shutdown to complete; use Wait for it.

func (*LStreamsManager) Disconnect 

func (lsman *LStreamsManager) Disconnect()

func (*LStreamsManager) Ping 

func (lsman *LStreamsManager) Ping()

func (*LStreamsManager) QueryLogs 

func (lsman *LStreamsManager) QueryLogs(params QueryLogsParams)

func (*LStreamsManager) Reconnect 

func (lsman *LStreamsManager) Reconnect()

func (*LStreamsManager) SetLStreams 

func (lsman *LStreamsManager) SetLStreams(logStreamsSpec string) error

func (*LStreamsManager) Wait 

func (lsman *LStreamsManager) Wait()

Wait waits for the LStreamsManager to tear down. Typically used after calling Close().

type LStreamsManagerParams 

type LStreamsManagerParams struct {
	// ConfigLogStreams contains nerdlog-specific config, typically coming from
	// ~/.config/nerdlog/logstreams.yaml.
	ConfigLogStreams ConfigLogStreams

	// SSHConfig contains the general ssh config, typically coming from
	// ~/.ssh/config.
	SSHConfig *ssh_config.Config

	Logger *log.Logger

	InitialLStreams string

	// ClientID is just an arbitrary string (should be filename-friendly though)
	// which will be appended to the nerdlog_agent.sh and its index filenames.
	//
	// Needed to make sure that different clients won't get conflicts over those
	// files when using the tool concurrently on the same nodes.
	ClientID string

	UpdatesCh chan<- LStreamsManagerUpdate
}

type LStreamsManagerState 

type LStreamsManagerState struct {
	NumLStreams int

	LStreamsByState map[LStreamClientState]map[string]struct{}

	// NumConnected is how many nodes are actually connected
	NumConnected int

	// NoMatchingLStreams is true when there are no matching lstreams.
	NoMatchingLStreams bool

	// Connected is true when all matching lstreams (which should be more than 0)
	// are connected.
	Connected bool

	// Busy is true when a query is in progress.
	Busy bool

	ConnDetailsByLStream map[string]ConnDetails
	BusyStageByLStream   map[string]BusyStage

	// TearingDown contains logstream names whic are in the process of teardown.
	TearingDown []string
}

type LStreamsManagerUpdate 

type LStreamsManagerUpdate struct {
	State   *LStreamsManagerState
	LogResp *LogRespTotal

	BootstrapIssue *BootstrapIssue
}

type LStreamsResolver 

type LStreamsResolver struct {
	// contains filtered or unexported fields
}

func NewLStreamsResolver 

func NewLStreamsResolver(params LStreamsResolverParams) *LStreamsResolver

func (*LStreamsResolver) Resolve 

func (r *LStreamsResolver) Resolve(lstreamsStr string) (map[string]LogStream, error)

Resolve parses the given logstream spec, and returns the mapping from LogStream.Name to the corresponding LogStream. Examples of logstream spec are:

- "myuser@myserver.com:22:/var/log/syslog" - "myuser@myserver.com:22" - "myuser@myserver.com" - "myserver.com"

type LStreamsResolverParams 

type LStreamsResolverParams struct {
	// CurOSUser is the current OS username, it's used as the last resort when
	// determining the user for a particular host connection.
	CurOSUser string

	// ConfigLogStreams is the nerdlog-specific config, typically coming from
	// ~/.config/nerdlog/logstreams.yaml.
	ConfigLogStreams ConfigLogStreams

	// SSHConfig is the general SSH config, typically coming from ~/.ssh/config
	SSHConfig *ssh_config.Config
}

type LogLevel 

type LogLevel string
const LogLevelDebug LogLevel = "debug"
const LogLevelError LogLevel = "error"
const LogLevelInfo LogLevel = "info"
const LogLevelUnknown LogLevel = ""
const LogLevelWarn LogLevel = "warn"

type LogMsg 

type LogMsg struct {
	Time               time.Time
	DecreasedTimestamp bool

	// LogFilename and LogLinenumber are file ane line number in that file
	LogFilename   string
	LogLinenumber int

	// CombinedLinenumber is the line number in pseudo-file: all (actually just
	// two) log files concatenated. This is the linenumbers output by the
	// nerdlog_agent.sh for every "msg:" line, and this is the linenumber
	// which should be used for --lines-until param.
	CombinedLinenumber int

	Msg     string
	Context map[string]string
	Level   LogLevel

	OrigLine string
}

type LogResp 

type LogResp struct {
	// MinuteStats is a map from the unix timestamp (in seconds) to the stats for
	// the minute starting at this timestamp.
	MinuteStats map[int64]MinuteStatsItem

	Logs []LogMsg

	// NumMsgsTotal is the total number of messages in the time range (and
	// included in MinuteStats). This number is usually larger than len(Logs).
	NumMsgsTotal int
}

LogResp is a log response from a single logstream

type LogRespTotal 

type LogRespTotal struct {
	// If LoadedEarlier is true, it means we've just loaded more logs instead of replacing
	// the logs (the Logs slice still contains everything though).
	LoadedEarlier bool

	// MinuteStats is a map from the unix timestamp (in seconds) to the stats for
	// the minute starting at this timestamp.
	MinuteStats map[int64]MinuteStatsItem

	Logs []LogMsg

	// NumMsgsTotal is the total number of messages in the time range (and
	// included in MinuteStats). This number is usually larger than len(Logs).
	NumMsgsTotal int

	Errs []error

	// QueryDur shows how long the query took.
	QueryDur time.Duration
}

LogRespTotal is a log response from a LStreamsManager. It's merged from multiple LogResp's and it also contains some extra field(s), e.g. LoadedEarlier.

type LogStream 

type LogStream struct {
	// Name is an arbitrary string which will be included in log messages as the
	// "lstream" context tag; it must uniquely identify the LogStream.
	Name string

	Host     ConfigHost
	Jumphost *ConfigHost

	// LogFiles contains a list of files which are part of the logstream, like
	// ["/var/log/syslog", "/var/log/syslog.1"]. The [0]th item is the latest log
	// file [1]st is the previous one, etc.
	//
	// It must contain at least a single item, otherwise LogStream is invalid.
	LogFiles []string

	Options LogStreamOptions
}

func (LogStream) LogFileLast 

func (ls LogStream) LogFileLast() string

func (LogStream) LogFilePrev 

func (ls LogStream) LogFilePrev() (string, bool)

type LogStreamOptions added in v1.2.0 

type LogStreamOptions struct {
	SudoMode SudoMode
}

type MinuteStatsItem 

type MinuteStatsItem struct {
	NumMsgs int
}

type QueryLogsParams 

type QueryLogsParams struct {
	// maxNumLines is how many log lines the nerdlog_agent.sh will return at
	// most.
	MaxNumLines int

	From time.Time
	To   time.Time

	Query string

	// If LoadEarlier is true, it means we're only loading the logs _before_ the ones
	// we already had.
	LoadEarlier bool

	// If DontAddHistoryItem is true, the browser-like history will not be
	// populated with a new item (it should be used exactly when we're navigating
	// this browser-like history back and forth)
	DontAddHistoryItem bool
}

type SudoMode added in v1.2.0 

type SudoMode string

SudoMode can be used to configure nerdlog to read log files with "sudo -n". See constants below for more details. 

const (
	// SudoModeNone is the same as an empty string, and it obviously means that
	// no sudo will be used. It exists as a separate mode to make it possible to
	// override the mode to not use sudo even if some config specifies some
	// non-empty sudo mode.
	SudoModeNone SudoMode = "none"

	// SudoModeFull means that the whole nerdlog_agent.sh script will be executed
	// with "sudo -n". Useful for cases when the log files are owned by root but
	// sudo doesn't require a password.
	SudoModeFull SudoMode = "full"
)

type TimeFormatAWKExpr 

type TimeFormatAWKExpr struct {
	// Month is an AWK expression to get month number as a string, from "01" to
	// "12". It may use `monthByName`, which is a map from a 3-char string like
	// "Jan" to the corresponding string like "01".
	//
	// So e.g. for the traditional syslog format "Jan _2 15:04:05", it should be
	// "monthByName[$1]".
	//
	// For the format "2006-01-02T15:04:05.000000Z07:00", it should rather be
	// "substr($0, 6, 2)".
	Month string

	// Year is an AWK expression to get year string like "2024". If the format
	// doesn't contain the year, it may use the already-computed `month` and
	// `yearByMonth`, which is a mapping from the month (from "01" to "12") to
	// the corresponding inferred year.
	//
	// So e.g. for the traditional syslog format "Jan _2 15:04:05", it should be
	// "yearByMonth[month]".
	//
	// For the format "2006-01-02T15:04:05.000000Z07:00", it should rather be
	// "substr($0, 1, 4)".
	Year string

	// Day is an AWK expression to get the day string like "05"; note the leading
	// zero, it's important (TODO: make it possible to support spaces; it'd mean
	// having spaces in the index and in the --from and --to args, which has issues)
	//
	// So e.g. for the traditional syslog format "Jan _2 15:04:05", it should be
	// `(length($2) == 1) ? "0" $2 : $2`.
	//
	// For the format "2006-01-02T15:04:05.000000Z07:00", it should rather be
	// "substr($0, 9, 2)".
	Day string

	// HHMM is an AWK expression to get the hours and minutes string like "14:38".
	//
	// So e.g. for the traditional syslog format "Jan _2 15:04:05", it should be
	// "substr($3, 1, 5)".
	//
	// For the format "2006-01-02T15:04:05.000000Z07:00", it should rather be
	// "substr($0, 12, 5)".
	HHMM string

	// MinuteKey is an AWK expression to get a string covering all timestamp
	// components from minute and larger. It'll be used as a key to identify a
	// particular minute (in the mapping from the minute to the amount of logs in
	// that minute), hence the name; so it should not include seconds, and it
	// should include minute+hour+day+month, maybe even year but that's optional,
	// since Nerdlog is not designed to look at logs spanning more than one year.
	//
	// So e.g. for the traditional syslog format "Jan _2 15:04:05", it should be
	// "substr($0, 1, 12)".
	//
	// For the format "2006-01-02T15:04:05.000000Z07:00", it should rather be
	// "substr($0, 1, 16)" (to include the year) or "substr($0, 6, 11)" (to not
	// include the year).
	MinuteKey string
}

TimeFormatAWKExpr contains all the awk expressions which will be used by the nerdlog_agent.sh script to get the time components from logs.

type TimeFormatDescr 

type TimeFormatDescr struct {
	// TimestampLayout is a Go-style time layout which should parse the entire
	// timestamp in the log line, e.g. "Jan _2 15:04:05" or
	// "2006-01-02T15:04:05.000000Z07:00".
	TimestampLayout string

	// MinuteKeyLayout is a Go-style time layout which should parse the time
	// captured by the awk expression `TimeFormatAWKExpr.MinuteKey` (read there
	// what "minute key" means in the first place).
	//
	// So e.g. for the traditional syslog format "Jan _2 15:04:05", it should be
	// "Jan _2 15:04".
	//
	// For the format "2006-01-02T15:04:05.000000Z07:00", it should rather be
	// "2006-01-02T15:04" (if we decided to include the year) or "01-02T15:04"
	// (if we decided to not include the year).
	MinuteKeyLayout string

	// AWKExpr contains all the awk expressions which will be used by the
	// nerdlog_agent.sh script to get the time components from logs.
	AWKExpr TimeFormatAWKExpr
}

TimeFormatDescr contains all data necessary for Nerdlog to parse timestamps in the particular logstream. It contains info for both the awk script and Go client.

func GenerateTimeDescr 

func GenerateTimeDescr(layout string) (*TimeFormatDescr, error)

GenerateTimeDescr takes a Go-style time layout, and returns the full time format descriptor to be used for parsing all logs.

func GetTimeFormatDescrFromLogLines 

func GetTimeFormatDescrFromLogLines(logLines []string) (*TimeFormatDescr, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.