Documentation
¶
Overview ¶
Package nanlogger collects `graph.Node` objects to monitor for `NaN` ("not-a-number") or `Inf` (infinity) values.
It does that by implementing `graph.LoggerFn` and hooking to the `graph.Exec` that executes the graph. If at the end of a graph.Exec call, if a `NaN` value is found, the first node where it appears (often `NaN` values spread through the graph) is reported back.
The report includes a stack trace and an optional user set scoped context.
Example:
func train() {
…
nanLogger := nanlogger.New()
trainer := train.NewTrainer(…)
trainer.OnExecCreation(func(exec *context.Exec, _ train.GraphType) {
nanLogger.Attach(exec)
})
…
}
func ModelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node {) {
…
for ii := 0; ii < numBlocks; ii++ {
name := fmt.Sprintf("Residual-%d", ii+1)
nanLogger.PushScope(name)
x = ResidualBlock(ctx.In(name), x, lastNumChannels)
nanLogger.Trace(x)
nanLogger.PopScope()
}
…
}
Index ¶
Constants ¶
const UniqueMessageId = "#nanlogger"
Variables ¶
This section is empty.
Functions ¶
func DefaultHandler ¶
DefaultHandler when a `NaN` or `Inf` is observed: it prints all out all the information and exits.
Types ¶
type ExecWithLogger ¶
type ExecWithLogger interface {
SetNodeLogger(loggerFn graph.LoggerFn)
GetNodeLogger() graph.LoggerFn
}
ExecWithLogger represents any of the executors in GoMLX (or future): `graph.Exec` and `context.Exec`. What is required is that it supports setting the logger and reading the current logger.
type NanLogger ¶
type NanLogger struct {
// contains filtered or unexported fields
}
NanLogger uses the logger infrastructure to monitor for NaN (and Inf) values in your graph. You manually select the nodes you want to monitor, and it saves the stack where it was called along with user provided scope information. If during the execution any NaN appears, it panics with the stack trace where the monitor was set, along with the scope. Alternatively, instead of panicking, one can set a handler to be called if/when a NaN is observed.
See example in package documentation.
func New ¶
func New() *NanLogger
New creates a NanLogger that can be used to debug where NaN happen in graphs. See NanLogger for details.
func (*NanLogger) Attach ¶
func (l *NanLogger) Attach(exec ExecWithLogger)
Attach will set the NanLogger as the default logger in exec. NanLogger acts as a pass-through logger, anything that is not marked as nanlogger.UniqueMessageId is passed through to whatever was the previous logger configured in exec.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) PopScope ¶
func (l *NanLogger) PopScope()
PopScope removes the last entry in the current scope stack. These values are added by default to any new Trace.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) PushScope ¶
PushScope to current scope stack. These values are added by default to any new Trace.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) SetHandler ¶
SetHandler sets the function called when a `NaN` is observed. The default is DefaultHandler which prints out all information on the node and exits.
func (*NanLogger) Trace ¶
Trace the given node. This means the node is monitored and whenever a NaN is observed, the trace is printed and the program exit. Alternatively, a handler is called, see SetHandler.
A user-provided scope can be given. If none is given, then it uses the current NanLogger scope. These are two different methods of providing scope, both optional -- it's also fine not to provide any.
A nil NanLogger is valid, and it will simply be a no-op.
type Trace ¶
type Trace struct {
// StackTrace of where the monitored node was created, stored as an error that can be printed.
StackTrace error
// Scope saved when monitor node was created.
Scope []string
}
Trace information of a node that is set to monitor. This is what printed out when a `NaN` is found, or passed to a handler function, if one is set.
Source Files