journal

README

journal

Package journal is intended to be a very simple, but somewhat flexible logging facility for golang services.

Main attributes:

• Selection of relevant columns for local logs
• Built-in or custom error messages
• Output to file and optionally stdout
• File rotation (daily, weekly, monthly, annually)
• Old log compression
• Remote logging/mirroring via grpc
• Tab-delimited or JSON-formatted output (output to stdout is always tab-delimited)

Logging locally

import log "github.com/vaitekunas/log"

logger := log.New(&log.Config{


})

notify := logger.Logfunc("Caller 1")

notify(0, "This is a simple message")
if err := notify(1, "Something went wrong: %s","Some general error"); err != nil {
  notify(999, "Yup, things really went south: %s (%s)", "Should exit the program now","Nothing to be done here")
}

go func(){
    notify := logger.Logfunc("Coroutine 1")
    notify(0, "Starting coroutine")
    notify(3, "")
}()

<- time.After(500*time.Millisecond)

notify(0, "Exiting")

Starting a remote logging facility and logging remotely

Mirroring local logs remotely is a nice way of aggregating information about a set of your services/instances in one place.

logserver start-service \
          --host=127.0.0.1 --port=37746 \
          --folder=$HOME/logs/ --file=myservice --stdout=true \
          --rotation=daily --compress=true \
          --cert=$HOME/logger/cert.pem --key=$HOME/logger/key.pem
          --tokens=$HOME/logger/tokens.db

Authentication token management

In order to accept incoming connections we need to create authentication tokens for each service/instance of the service we wish to allow to connect to this server:

logserver add-token myservice myinstance

This creates a random 64-character token that the myinstance instance of service myservice can use to connect to this server. We can also retrieve the token for the instance:

logserver show-token myservice myinstance

or all instances of a service:

logserver show-tokens myservice

tokens can also be revoked:

logserver revoke-token myservice myinstance

We can also revoke all tokens and close all incoming connections for a given service:

logserver revoke-tokens myservice

revoking the token(s) closes any associated sessions and denies future connections.

Remote logging

import log "github.com/vaitekunas/log"

logger := log.New(&log.Config{


})

notify := logger.Logfunc("Caller 1")

// Connect to the log server
host := "127.0.0.1"
port := 37746
token := "8018A51319ADA07AF801595E18EABE438FF5C330241661B93C9C32ED36D60C00"

if err:= logger.Connect(host, port, instance, token, 5*time.Second); err != nil {
  notify(1, "Could not connect to log server: %s", err.Error())  
}else{
  notify(0, "Connection to logserver (%s:%d) established.", host, port)
}

Local output:

DATA

Remote output:

DATA

The logging facility server will write all the incoming messages to the same file (and optionally stdout). In the logfile all the columns are present, even though incoming messages might have selected differing columns.

Remote logging is done asynchronously. The request will timeout according to the defined timeout duration.

Build

log imports the logrpc subpackage, which is generated from a protobuf definition, so that in order to build log you first need to generate the golang stubs for it:

protoc --go_out=plugins=grpc:. protobuf/log.proto

TODO

• Implement journal.ConnectToKafka

Documentation

Overview

Package journal implements a very simple logging facility with only one shared logging method, namely "Log".

Index

• Constants
• func ConnectToJournald(host string, port int, service, instance, token string, timeout time.Duration) (io.WriteCloser, error)
• func ConnectToKafka(host string, port int, topic string) (io.WriteCloser, error)
• type Code
• type Config
• type Logger
  • func New(config *Config) (*Logger, error)
  • func (l *Logger) AddDestination(name string, writer io.Writer) error
  • func (l *Logger) ListDestinations() []string
  • func (l *Logger) Log(caller string, code int, msg string, format ...interface{}) error
  • func (l *Logger) LogFields(caller string, code int, msg map[string]interface{}) error
  • func (l *Logger) NewCaller(caller string) func(int, string, ...interface{}) error
  • func (l *Logger) NewCallerWithFields(caller string) func(int, map[string]interface{}) error
  • func (l *Logger) Quit()
  • func (l *Logger) RawEntry(entry map[int64]string) error
  • func (l *Logger) RemoveDestination(name string) error
  • func (l *Logger) UseCustomCodes(codes map[int]Code)

Examples

• Logger.Log

Constants

const (
	ROT_NONE     = 0
	ROT_DAILY    = 1
	ROT_WEEKLY   = 2
	ROT_MONTHLY  = 3
	ROT_ANNUALLY = 4
)

File rotation frequency

const (
	OUT_FILE            = 0
	OUT_STDOUT          = 1
	OUT_FILE_AND_STDOUT = 2
)

Output selection

const (
	COL_DATE_YYMMDD             = 0
	COL_DATE_YYMMDD_HHMMSS      = 1
	COL_DATE_YYMMDD_HHMMSS_NANO = 2
	COL_TIMESTAMP               = 3
	COL_SERVICE                 = 4
	COL_INSTANCE                = 5
	COL_CALLER                  = 6
	COL_MSG_TYPE_SHORT          = 7
	COL_MSG_TYPE_INT            = 8
	COL_MSG_TYPE_STR            = 9
	COL_MSG                     = 10
	COL_FILE                    = 11
	COL_LINE                    = 12
)

Log columns

Functions

func ConnectToJournald

func ConnectToJournald(host string, port int, service, instance, token string, timeout time.Duration) (io.WriteCloser, error)

ConnectToJournald connects to a log server backend

func ConnectToKafka

func ConnectToKafka(host string, port int, topic string) (io.WriteCloser, error)

ConnectToKafka connects to a kafka backend as a producer

Types

type Code

type Code struct {
	Error bool
	Type  string
}

Code contains a single message type with an indicator of whether this message should be treated as an error.

type Config

type Config struct {
	Service  string  // Service name
	Instance string  // Instance name
	Folder   string  // Folder to store logfiles (can be empty if logging to stdout only)
	Filename string  // Filename of the logfiles (without date suffix and file extension. Can be empty if logging to stdout only)
	Rotation int     // Logfile rotation frequency
	Out      int     // Logger output type
	Headers  bool    // Should the logfile contain column headers?
	JSON     bool    // Should each entry be written as a JSON-formatted string?
	Compress bool    // Should old logfiles be compressed?
	Columns  []int64 // List of relevant columns (can be empty if default columns should be used)
}

Config contains all the necessary settings to create a new local logging facility

type Logger

type Logger struct {
	// contains filtered or unexported fields
}

Logger is the main loggger struct

func New

func New(config *Config) (*Logger, error)

New creates a new logging facility

func (*Logger) AddDestination

func (l *Logger) AddDestination(name string, writer io.Writer) error

AddDestination adds a (remote) destination to send logs to

func (*Logger) ListDestinations

func (l *Logger) ListDestinations() []string

ListDestinations lists all (remote) destinations

func (*Logger) Log

func (l *Logger) Log(caller string, code int, msg string, format ...interface{}) error

Log logs a simple message and returns nil or error, depending on the code

Example

Simple example of local logging

package main

import (
	"fmt"
	"io/ioutil"
	"os"
	"testing"
)

// Test/example setup
func setup(t *testing.T) (tempdir string, teardown func()) {

	dir, err := ioutil.TempDir(os.Getenv("HOME"), "example")
	if err != nil {
		t.Errorf("Could not create tempdir: %s", err.Error())
	}

	return dir, func() {
		os.RemoveAll(dir)
	}

}

// Simple example of local logging
func main() {

	// Create temporary folder and teardown function
	tempdir, teardown := setup(&testing.T{})
	defer teardown()

	// Instantiate logger
	logger, err := New(&Config{
		Service:  "MyService",
		Instance: "MyInstance",
		Folder:   tempdir,
		Filename: "myservice",
		Rotation: ROT_DAILY,
		Out:      OUT_FILE_AND_STDOUT,
		Headers:  true,
		JSON:     false,
		Compress: true,
		Columns:  []int64{}, // Use default columns
	})

	if err != nil {
		fmt.Printf("Could not start logger: %s", err.Error())
		os.Exit(1)
	}

	// Log messages
	notify := logger.NewCaller("Example 1")

	notify(0, "Hello, World!")
}

func (*Logger) LogFields

func (l *Logger) LogFields(caller string, code int, msg map[string]interface{}) error

LogFields encodes the message (not the whole log) in JSON and writes to log

func (*Logger) NewCaller

func (l *Logger) NewCaller(caller string) func(int, string, ...interface{}) error

NewCaller is a wrapper for the Logger.Log function

func (*Logger) NewCallerWithFields

func (l *Logger) NewCallerWithFields(caller string) func(int, map[string]interface{}) error

NewCallerWithFields is a wrapper for the Logger.LogFields function

func (*Logger) Quit

func (l *Logger) Quit()

Quit stops all Logger coroutines and closes files

func (*Logger) RawEntry

func (l *Logger) RawEntry(entry map[int64]string) error

RawEntry writes a raw log entry (map of strings) into the ledger. The raw entry must contain columns COL_DATE_YYMMDD_HHMMSS_NANO to COL_LINE

func (*Logger) RemoveDestination

func (l *Logger) RemoveDestination(name string) error

RemoveDestination removes a (remote) destination to send logs to

func (*Logger) UseCustomCodes

func (l *Logger) UseCustomCodes(codes map[int]Code)

UseCustomCodes Replaces loggers default message codes with custom ones