README
¶
nanolog
"Nanosecond scale" logger inspired by https://github.com/PlatformLab/NanoLog
Why?
It's about 3.75x faster than the equivalent stdlib log package usage and the output log files are about 1/2 the size. These ratios should increase and decrease, respectively, as the amount of unchanging data in each log line increases.
The AddLogger method returns a nanolog.Handle to an internal data structure that keeps track of the data required to ensure proper operation. This handle is just a simple number identifier. You can think of AddLogger like adding a prepared statement in a database. You supply the unchanging information up front, and the system holds on to that while you give it to the changing data. Overall this is much more efficient because less data is transferred.
Usage
Logging at runtime
Add loggers by registering them in an init function in any package using AddLogger. The main package should set the writer for the logging system (using the SetWriter method) before doing much of anything else, as log writes are buffered in memory until the writer is set. Writes include the data AddLogger generates, so by the time main gets started there's likely data waiting. Log lines are written using the Log method.
At the end of the main method in your program, you should ensure that you call nanolog.Flush() to ensure that the data that has been logged is sent to the writer you supplied. Otherwise, some data may get lost.
package main
import (
"os"
"github.com/ScottMansfield/nanolog"
)
var h nanolog.Handle
func init() {
nanolog.SetWriter(os.Stderr)
h = nanolog.AddLogger("Example %i32 log %{s} line %c128")
}
func main() {
nanolog.Log(h, int32(4), "this is a string", 4i)
nanolog.Flush()
}
Inflating the logs
The logs are written in an efficient format and are thus not human-readable. In order to be able to read them, you will need to "inflate" them. Each log file is self-contained, so the tooling doesn't need any external information to parse the file.
First, compile the inflate tool, then use it on the log output file. The tool outputs to stdout, so if you want to save the output for later, make sure to direct it to another file. The following example assumes your log output is stored in foo.clog.
$ go build github.com/ScottMansfield/nanolog/cmd/inflate
$ ./inflate -f foo.clog > foo-inflated.log
Format
The logger is created with a string format. The interpolation tokens are prefixed using a percentage sign (%) and surrounded by optional curly braces when you need to disambiguate. This can be useful if you want to interpolate an int but for some reason need to put a number after it that might confuse the system, like a 1, 3, or 6.
nanolog.AddLogger("Disambiguate this: %{i}32")
In order to output a literal %, you use two of them in a row to escape the second one.
Types
The types that can be interpolated are limited, for now, to those in the following table. The corresponding interpolation tokens are listed next to each type.
| Type | Token |
|---|---|
| Bool | b |
| Int | i |
| Int8 | i8 |
| Int16 | i16 |
| Int32 | i32 |
| Int64 | i64 |
| Uint | u |
| Uint8 | u8 |
| Uint16 | u16 |
| Uint32 | u32 |
| Uint64 | u64 |
| Float32 | f32 |
| Float64 | f64 |
| Complex64 | c64 |
| Complex128 | c128 |
| String | s |
The logging system is strict when it comes to types. For example, an int16 will not work in a slot meant for an int.
Benchmark
This benchmark is in the nanolog_test.go file. It compares the following log line time to log for both nanolog and the stdlib log package.
nanolog:
foo thing bar thing %i64. Fubar %s foo. sadfasdf %u32 sdfasfasdfasdffds %u32.
stdlib:
foo thing bar thing %d. Fubar %s foo. sadfasdf %d sdfasfasdfasdffds %d.
$ go test -bench CompareToStdlib -count 100 >> bench
$ benchstat bench
name time/op
CompareToStdlib/Nanolog-8 120ns ± 3%
CompareToStdlib/Stdlib-8 452ns ± 3%
Documentation
¶
Overview ¶
Package nanolog is a package to speed up your logging.
The format string is inspired by the full fledged fmt.Fprintf function. The codes are unique to this package, so normal fmt documentation is not be applicable.
The format string is similar to fmt in that it uses the percent sign (a.k.a. the modulo operator) to signify the start of a format code. The reader is greedy, meaning that the parser will attempt to read as much as it can for a code before it stops. E.g. if you have a generic int in the middle of your format string immediately followed by the number 1 and a space ("%i1 "), the parser may complain saying that it encountered an invalid code. To fix this, use curly braces after the percent sign to surround the code: "%{i}1 ".
Kinds and their corresponding format codes:
Kind | Code --------------|------------- Bool | b Int | i Int8 | i8 Int16 | i16 Int32 | i32 Int64 | i64 Uint | u Uint8 | u8 Uint16 | u16 Uint32 | u32 Uint64 | u64 Uintptr | Float32 | f32 Float64 | f64 Complex64 | c64 Complex128 | c128 Array | Chan | Func | Interface | Map | Ptr | Slice | String | s Struct | UnsafePointer |
The file format has two categories of data:
- Log line information to reconstruct logs later
- Actual log entries
The differentiation is done with the entryType, which is prefixed on to the record.
The log line records are formatted as follows:
- type: 1 byte - ETLogLine (1)
- id: 4 bytes - little endian uint32
- # of string segs: 4 bytes - little endian uint32
- kinds: (#segs - 1) bytes, each being a reflect.Kind
- segments:
- string length: 4 bytes - little endian uint32
- string data: ^length bytes
The log entry records are formatted as follows:
- type: 1 byte - ETLogEntry (2)
- line id: 4 bytes - little endian uint32
- data+: var bytes - all the corresponding data for the kinds in the log line entry
The data is serialized as follows:
Bool: 1 byte
False: 0 or True: 1
String: 4 + len(string) bytes
Length: 4 bytes - little endian uint32
String bytes: Length bytes
int family:
int: 8 bytes - int64 as little endian uint64
int8: 1 byte
int16: 2 bytes - int16 as little endian uint16
int32: 4 bytes - int32 as little endian uint32
int64: 8 bytes - int64 as little endian uint64
uint family:
uint: 8 bytes - little endian uint64
uint8: 1 byte
uint16: 2 bytes - little endian uint16
uint32: 4 bytes - little endian uint32
uint64: 8 bytes - little endian uint64
float32:
4 bytes as little endian uint32 from float32 bits
float64:
8 bytes as little endian uint64 from float64 bits
complex64:
Real: 4 bytes as little endian uint32 from float32 bits
Complex: 4 bytes as little endian uint32 from float32 bits
complex128:
Real: 8 bytes as little endian uint64 from float64 bits
Complex: 8 bytes as little endian uint64 from float64 bits
Index ¶
Constants ¶
const MaxLoggers = 10240
MaxLoggers is the maximum number of different loggers that are allowed
Variables ¶
This section is empty.
Functions ¶
Types ¶
type EntryType ¶
type EntryType byte
EntryType is an enum that represents the record headers in the output files to differentiate between log lines and log entries
type Handle ¶
type Handle uint32
Handle is a simple handle to an internal logging data structure LogHandles are returned by the AddLogger method and used by the Log method to actually log data.
type LogWriter ¶ added in v0.2.0
type LogWriter interface {
// SetWriter will set up efficient writing for the log to the output stream given.
// A raw IO stream is best. The first time SetWriter is called any logs that were
// created or posted before the call will be sent to the writer all in one go.
SetWriter(new io.Writer) error
// Flush ensures all log entries written up to this point are written to the underlying io.Writer
Flush() error
// AddLogger initializes a logger and returns a handle for future logging
AddLogger(fmt string) Handle
// Log logs to the output stream
Log(handle Handle, args ...interface{}) error
}
Source Files
Directories