nanolog

package module
Version: v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 1, 2017 License: Apache-2.0 Imports: 10 Imported by: 14

README

GoDoc GoReportCard Coverage Status

nanolog

"Nanosecond scale" logger inspired by https://github.com/PlatformLab/NanoLog

Why?

It's about 3.75x faster than the equivalent stdlib log package usage and the output log files are about 1/2 the size. These ratios should increase and decrease, respectively, as the amount of unchanging data in each log line increases.

The AddLogger method returns a nanolog.Handle to an internal data structure that keeps track of the data required to ensure proper operation. This handle is just a simple number identifier. You can think of AddLogger like adding a prepared statement in a database. You supply the unchanging information up front, and the system holds on to that while you give it to the changing data. Overall this is much more efficient because less data is transferred.

Usage

Logging at runtime

Add loggers by registering them in an init function in any package using AddLogger. The main package should set the writer for the logging system (using the SetWriter method) before doing much of anything else, as log writes are buffered in memory until the writer is set. Writes include the data AddLogger generates, so by the time main gets started there's likely data waiting. Log lines are written using the Log method.

At the end of the main method in your program, you should ensure that you call nanolog.Flush() to ensure that the data that has been logged is sent to the writer you supplied. Otherwise, some data may get lost.

package main

import (
	"os"
	"github.com/ScottMansfield/nanolog"
)

var h nanolog.Handle

func init() {
	nanolog.SetWriter(os.Stderr)
	h = nanolog.AddLogger("Example %i32 log %{s} line %c128")
}

func main() {
	nanolog.Log(h, int32(4), "this is a string", 4i)
	nanolog.Flush()
}

Inflating the logs

The logs are written in an efficient format and are thus not human-readable. In order to be able to read them, you will need to "inflate" them. Each log file is self-contained, so the tooling doesn't need any external information to parse the file.

First, compile the inflate tool, then use it on the log output file. The tool outputs to stdout, so if you want to save the output for later, make sure to direct it to another file. The following example assumes your log output is stored in foo.clog.

$ go build github.com/ScottMansfield/nanolog/cmd/inflate
$ ./inflate -f foo.clog > foo-inflated.log

Format

The logger is created with a string format. The interpolation tokens are prefixed using a percentage sign (%) and surrounded by optional curly braces when you need to disambiguate. This can be useful if you want to interpolate an int but for some reason need to put a number after it that might confuse the system, like a 1, 3, or 6.

nanolog.AddLogger("Disambiguate this: %{i}32")

In order to output a literal %, you use two of them in a row to escape the second one.

Types

The types that can be interpolated are limited, for now, to those in the following table. The corresponding interpolation tokens are listed next to each type.

Type Token
Bool b
Int i
Int8 i8
Int16 i16
Int32 i32
Int64 i64
Uint u
Uint8 u8
Uint16 u16
Uint32 u32
Uint64 u64
Float32 f32
Float64 f64
Complex64 c64
Complex128 c128
String s

The logging system is strict when it comes to types. For example, an int16 will not work in a slot meant for an int.

Benchmark

This benchmark is in the nanolog_test.go file. It compares the following log line time to log for both nanolog and the stdlib log package.

nanolog:

foo thing bar thing %i64. Fubar %s foo. sadfasdf %u32 sdfasfasdfasdffds %u32.

stdlib:

foo thing bar thing %d. Fubar %s foo. sadfasdf %d sdfasfasdfasdffds %d.
$ go test -bench CompareToStdlib -count 100 >> bench
$ benchstat bench
name                       time/op
CompareToStdlib/Nanolog-8  120ns ± 3%
CompareToStdlib/Stdlib-8   452ns ± 3%

Documentation

Overview

Package nanolog is a package to speed up your logging.

The format string is inspired by the full fledged fmt.Fprintf function. The codes are unique to this package, so normal fmt documentation is not be applicable.

The format string is similar to fmt in that it uses the percent sign (a.k.a. the modulo operator) to signify the start of a format code. The reader is greedy, meaning that the parser will attempt to read as much as it can for a code before it stops. E.g. if you have a generic int in the middle of your format string immediately followed by the number 1 and a space ("%i1 "), the parser may complain saying that it encountered an invalid code. To fix this, use curly braces after the percent sign to surround the code: "%{i}1 ".

Kinds and their corresponding format codes:

Kind          | Code
--------------|-------------
Bool          | b
Int           | i
Int8          | i8
Int16         | i16
Int32         | i32
Int64         | i64
Uint          | u
Uint8         | u8
Uint16        | u16
Uint32        | u32
Uint64        | u64
Uintptr       |
Float32       | f32
Float64       | f64
Complex64     | c64
Complex128    | c128
Array         |
Chan          |
Func          |
Interface     |
Map           |
Ptr           |
Slice         |
String        | s
Struct        |
UnsafePointer |

The file format has two categories of data:

  1. Log line information to reconstruct logs later
  2. Actual log entries

The differentiation is done with the entryType, which is prefixed on to the record.

The log line records are formatted as follows:

  • type: 1 byte - ETLogLine (1)
  • id: 4 bytes - little endian uint32
  • # of string segs: 4 bytes - little endian uint32
  • kinds: (#segs - 1) bytes, each being a reflect.Kind
  • segments:
  • string length: 4 bytes - little endian uint32
  • string data: ^length bytes

The log entry records are formatted as follows:

  • type: 1 byte - ETLogEntry (2)
  • line id: 4 bytes - little endian uint32
  • data+: var bytes - all the corresponding data for the kinds in the log line entry

The data is serialized as follows:

  • Bool: 1 byte

  • False: 0 or True: 1

  • String: 4 + len(string) bytes

  • Length: 4 bytes - little endian uint32

  • String bytes: Length bytes

  • int family:

  • int: 8 bytes - int64 as little endian uint64

  • int8: 1 byte

  • int16: 2 bytes - int16 as little endian uint16

  • int32: 4 bytes - int32 as little endian uint32

  • int64: 8 bytes - int64 as little endian uint64

  • uint family:

  • uint: 8 bytes - little endian uint64

  • uint8: 1 byte

  • uint16: 2 bytes - little endian uint16

  • uint32: 4 bytes - little endian uint32

  • uint64: 8 bytes - little endian uint64

  • float32:

  • 4 bytes as little endian uint32 from float32 bits

  • float64:

  • 8 bytes as little endian uint64 from float64 bits

  • complex64:

  • Real: 4 bytes as little endian uint32 from float32 bits

  • Complex: 4 bytes as little endian uint32 from float32 bits

  • complex128:

  • Real: 8 bytes as little endian uint64 from float64 bits

  • Complex: 8 bytes as little endian uint64 from float64 bits

Index

Constants

View Source
const MaxLoggers = 10240

MaxLoggers is the maximum number of different loggers that are allowed

Variables

This section is empty.

Functions

func Flush

func Flush() error

Flush calls LogWriter.Flush on the default log writer.

func Log

func Log(handle Handle, args ...interface{}) error

Log calls LogWriter.Log on the default log writer.

func SetWriter

func SetWriter(new io.Writer) error

SetWriter calls LogWriter.SetWriter on the default log writer.

Types

type EntryType

type EntryType byte

EntryType is an enum that represents the record headers in the output files to differentiate between log lines and log entries

const (
	// ETInvalid is an invalid EntryType
	ETInvalid EntryType = iota

	// ETLogLine means the log line data for a single call to AddLogger is ahead
	ETLogLine

	// ETLogEntry means the log data for a single call to Log is ahead
	ETLogEntry
)

type Handle

type Handle uint32

Handle is a simple handle to an internal logging data structure LogHandles are returned by the AddLogger method and used by the Log method to actually log data.

func AddLogger

func AddLogger(fmt string) Handle

AddLogger calls LogWriter.AddLogger on the default log writer.

type LogWriter added in v0.2.0

type LogWriter interface {
	// SetWriter will set up efficient writing for the log to the output stream given.
	// A raw IO stream is best. The first time SetWriter is called any logs that were
	// created or posted before the call will be sent to the writer all in one go.
	SetWriter(new io.Writer) error
	// Flush ensures all log entries written up to this point are written to the underlying io.Writer
	Flush() error
	// AddLogger initializes a logger and returns a handle for future logging
	AddLogger(fmt string) Handle
	// Log logs to the output stream
	Log(handle Handle, args ...interface{}) error
}

func New added in v0.2.0

func New() LogWriter

New creates a new LogWriter

type Logger

type Logger struct {
	Kinds []reflect.Kind
	Segs  []string
}

Logger is the internal struct representing the runtime state of the loggers. The Segs field is not used during logging; it is only used in the inflate utility

Source Files

Directories

Path Synopsis
cmd
pkg

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL