logwriter

package module
Version: v0.0.0-...-2242458 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 27, 2017 License: MIT Imports: 7 Imported by: 0

README

logwriter

Golang package logwriter automates routine related to logging into files.

GoDoc Build Status Coverage Status

Initial version finished. Stabilization, testing and benchmarkign are going.

Concepts

Hot and Cold Log Files

There is a single hot log file. Usually file name is similar to daemon/service name and located in /var/log/servicename/. There are cold log files. In accordance to rules specified by logwriter.Config, logwriter freezes content of hot file by moving content to new cold file.

Using sync.Mutex

If you don't need buffering (logwriter.Config.BufferSize==0) you can believe that file write executes synchronously.

Error Handling

There is set of background routines what can face critital i/o errors. Specify callback function f(error) in NewLogWriter() what will be called every time then background routines catch an error. Possible cases:

  • No free space at config.ColdPath
  • No free space at config.HotPath
  • Permission denied
  • ...

You error handler function calls in sync mode and blocks log writing. Please, do not do long operations there.

Stop! It's not a *unix way

Oh nooo. Not everyone develops Facebook (c) or smth similar daily :)

Features

  • Folders for hot and cold log files configurable
  • Using fixed name of file with latest log items
  • Support module running mode
    • Production - writes into the file only
    • Debug - writes into file and os.Stdout
  • Support hot file freezing rules:
    • By max file size
    • Every time.Duration
    • Every midnight
    • Manually
    • Freeze when your application starts
  • File write buffering
    • Configurable buffer size
    • Flush buffer every time.Duration
    • Flush buffer manually
  • Update configuration on the fly
  • Cold log files compression
  • Log items re-ordering before persisting
  • Log items re-ordering on freezing stage
  • Cold files cleaning
  • Cold log files round robin
  • Tracing option. Saving some of log items in separate .trc files
  • Ability to freeze hot file several times per second

Tasks

  • Add benchmarks
  • Add tests
  • Add examples

Examples

Using standard log package

package main

import (
	"github.com/regorov/logwriter"
	"log"
	"time"
)

func main() {
	cfg := &logwriter.Config{
		BufferSize:       0,                  // no buffering
		FreezeInterval:   1 * time.Hour,      // freeze log file every hour
		HotMaxSize:       100 * logwriter.MB, // 100 MB max file size
		CompressColdFile: true,               // compress cold file
		HotPath:          "/var/log/mywebserver",
		ColdPath:         "/var/log/mywebserver/arch",
		Mode:             logwriter.ProductionMode, // write to file only
	}

	lw, err := logwriter.NewLogWriter("mywebserver",
		cfg,
		true, // freeze hot file if exists
		nil)

	if err != nil {
		panic(err)
	}

	logger := log.New(lw, "mywebserver", log.Ldate|log.Ltime)
	logger.Println("Module started")

	if err := lw.Close(); err != nil {
		// Error handling
	}

	return
}

Using github.com/Sirupsen/logrus

package main

import (
  "time"
  "github.com/Sirupsen/logrus"
  "github.com/regorov/logwriter"
)

func errHandler(err error) {

	// send SMS or Smth
	return
}

func main() {

	lw, err := logwriter.NewLogWriter("mywebserver",
	                                  &logwriter.Config{
									      BufferSize: 1024 * 1024, // 1 MB
	                                      BufferFlushInterval : 3 * time.Second, // flush buffer every 3 sec
	                                      FreezeInterval : 1 * time.Hour, // create new log every hour
							              HotMaxSize : 100 * 1024 * 1024, // or when hot file size over 100 MB
	                                      HotPath: "/var/log/myweb",
	                                      ColdPath: "/var/log/myweb/arch",
	                                      Mode: logwriter.ProductionMode,
										},
					                  false, // do not freeze hot file if exists
					                  errHandler))
	if err != nil {
		// Error handling
	}

	var log = logrus.New()
  	log.Out = lw

	log.WithFields(logrus.Fields{"animal": "walrus",
        	                     "size":   10,
  	}).Info("A group of walrus emerges from the ocean")

	if err := lw.Close(); err != nil {
        // Error handling
    }

	return
}

License

MIT

Documentation

Overview

Package logwriter offers a rich log file writing tools.

There is single 'hot' log file per LogWriter. Usually file name is similar to servicename name and located in /var/log/servicename. All log items goes into 'hot' file.

There are "cold" log files. In accordance to rules specified by Config, logwiter freezes content of 'hot' file by moving content to 'cold' files.

Index

Examples

Constants

View Source
const (
	KB = 1024
	MB = 1024 * 1204
	GB = 1024 * 1024 * 1024
)

Size helpers

Variables

View Source
var (
	// HotFileExtension holds extension for 'hot' log file.
	HotFileExtension = "log"

	// ColdFileExtension holds extension for 'cold' log files.
	ColdFileExtension = "log"

	// CompressedColdFileExtension holds extension for compressed 'cold' files.
	CompressedColdFileExtension = "tz"

	// TraceFileExtension holds extension for trace files. (Not implemented yet)
	TraceFileExtension = "trc"
)

It is allowed to change default values listed above. Change it before calling NewLogWriter(). It's even safe to change if you already have running LogWriter instance.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Current running mode
	Mode RunningMode

	// Output buffer size. Buffering disabled if value == 0
	BufferSize int

	// Flush buffer to disk every BufferFlushInterval (works if BufferSize > 0)
	BufferFlushInterval time.Duration

	// Freeze hot file when size reaches HotMaxSize (value in bytes)
	HotMaxSize int64

	// Freeze hot file every FreezeInterval if value > 0
	FreezeInterval time.Duration

	// Freeze hot file at midnight
	FreezeAtMidnight bool

	// Folder where to open/create hot log file
	HotPath string

	// Folder where to copy cold file (frozen hot file)
	ColdPath string

	// CompressColdFile compresses cold file
	CompressColdFile bool
}

Config holds parameters of LogWriter instance.

type LogWriter

type LogWriter struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

LogWriter wraps io.Writer to automate routine with log files.

func NewLogWriter

func NewLogWriter(uid string, cfg *Config, freezeExisting bool, errHanldler func(error)) (*LogWriter, error)

NewLogWriter creates new LogWriter, opens/creates hot file "%uid%.log". Hot file freezes immediately if freezeExisting is true and non-empty file size > 0.

Example
package main

import (
	"github.com/regorov/logwriter"
	"log"
	"time"
)

func main() {

	cfg := &logwriter.Config{
		BufferSize: 2 * logwriter.MB, // write buffering enabled
		HotPath:    "/var/log/example",
		ColdPath:   "/var/log/example/arch",
		Mode:       logwriter.ProductionMode} // write into a file only

	lw, err := logwriter.NewLogWriter("mywebserver",
		cfg,
		true, // freeze log file if it exists
		nil)

	if err != nil {
		panic(err)
	}

	l := log.New(lw, "mywebserver ", log.Ldate|log.Ltime|log.Lmicroseconds)

	l.Println("Mywebserer started at ", time.Now())

	if err := lw.Close(); err != nil {
		panic(err)
	}
}
Output:

func (*LogWriter) Close

func (lw *LogWriter) Close() error

Close stops timers, flushes buffers and closes hot file. Please call this function at the end of your program.

func (*LogWriter) FlushBuffer

func (lw *LogWriter) FlushBuffer() error

FlushBuffer flushes buffer if buffering enabled and buffer is not empty

func (*LogWriter) FreezeHotFile

func (lw *LogWriter) FreezeHotFile() error

FreezeHotFile freezes hot file. Freeze steps: flush buffer, close file, rename hot file to temporary file in the same folder, rename/move temp file to cold file (async), create new hot file.

func (*LogWriter) SetColdNameFormatter

func (lw *LogWriter) SetColdNameFormatter(f func(string, string, time.Duration) string)

SetColdNameFormatter replaces default 'cold' file name generator. Default format is "$uid-20060102-150405[.00000].log" implemented by function defaultColdNameFormatter().

func (*LogWriter) SetConfig

func (lw *LogWriter) SetConfig(cfg *Config) error

SetConfig updates LogWriter config parameters. Func stops timers, flushes buffer, applies new Config, recreate buffer if need, starts timers.

func (*LogWriter) SetErrorFunc

func (lw *LogWriter) SetErrorFunc(f func(error))

SetErrorFunc assigns callback function to be called when BACKGROUND i/o fails. See running() as instance. logwriter public functions return error withoug calling specified function. Please be carefull, specified user function calls synchronously!

func (*LogWriter) SetMode

func (lw *LogWriter) SetMode(mode RunningMode)

SetMode changes LogWriter running mode. Default value is ProductionMode Default value can be overwritten in NewLogWriter() or changed later by SetMode

func (*LogWriter) Write

func (lw *LogWriter) Write(p []byte) (n int, err error)

Write 'overrides' the underlying io.Writer's Write method.

type RunningMode

type RunningMode int

RunningMode represents application running mode

const (
	// DebugMode orders to write log items into "hot" file and os.Stdout
	DebugMode RunningMode = 0

	// ProductionMode orders to wrire log items to the "hot" file only
	ProductionMode RunningMode = 1
)

Supported running mode options

Source Files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL