logicalfile

README

go-logicalfile

A logical file library that abstracts away physical file management for Go.

go-logicalfile provides a logical io.Writer whose underlying physical files are managed automatically. It separates application-level file writing from concerns such as file rotation, archiving, retention, compression, and file naming.

Application
    ↓
io.Writer
    ↓
go-logicalfile
    ↓
active physical file
    ↓
archive / rotation / compression

Features

• Logical file abstraction
  • Write to a logical file without managing physical files directly.
  • Automatically rotates and replaces the underlying physical writer when necessary.
• Automatic file rotation
  • Rotate files based on maximum file size.
  • Rotate files at a specified time.
• Archived file management
  • MaxAge — Remove archived files older than the specified age.
  • MaxHistory — Remove archived files when the number of archives exceeds the specified limit.
  • MaxTotalBytes — Remove archived files when their total size exceeds the specified limit.
• Optional compression of archived files.
• Flexible file naming
  • Configure independent patterns for active and archived files.
  • Generate file names using timestamps, indexes, host information, process IDs, and other values.

// FileConfig is the configuration for a logical file.
// Use [NewFile] to create a new logical file.
type FileConfig struct {
  // ManagerConfig is the configuration of physical file manager.
  ManagerConfig
  // MaxBytes is the maximum size in bytes.
  // Target writer will be renewed before exceeding the size.
  // Zero or negative means no limit, or no writer rotation.
  MaxBytes uint64
  // RotateAt optionally specifies the time to rotate.
  // RotateAt can be used for rotating physical file .
  // RotateAt must return the future time.
  RotateAt func() time.Time
  // HandleError optionally handles errors occurred in a logical file.
  HandleError func(error)
  // Fallback is the function that called when writing to the
  // current target io writer failed.
  // If nil, os.Stderr.Write is used by default.
  Fallback func([]byte) (int, error)
}

// ManagerConfig is the configuration for the physical file manager.
type ManagerConfig struct {
  // MaxAge is the maximum age to keep the archived files.
  // Age is calculated based on the last modified time
  // obtained by [io/fs.FileInfo].ModTime().
  // Set UseParsedAge to true to calculate ages using
  // the time parsed from archived files.
  MaxAge time.Duration
  // MaxHistory is the maximum number of archived files to keep.
  // Zero or negative value means no limit.
  MaxHistory int
  // MaxTotalBytes is the maximum total byte size to keep archived files.
  // Zero or negative value means no limit.
  MaxTotalBytes uint64
  // UseParsedAge, if true, calculate file ages based on the time
  // parsed from archived file names.
  // Note that time information must be contained in the Pattern.
  // Missing time components are initialized to their minimum values.
  // If %Y is not in the Pattern, it will be 1.
  // If %M is not in the Pattern, it will be 1 (January).
  // If %D is not in the Pattern, it will be 1.
  // If %h is not in the Pattern, it will be 00.
  // If %m is not in the Pattern, it will be 00.
  // If %s is not in the Pattern, it will be 00.
  UseParsedAge bool
  // CompressLv specifies the compression level.
  // Zero means no compression.
  CompressLv int
  // SrcDir is the directory containing source files.
  // If empty, DstDir is used.
  // If both SrcDir and DstDir are empty, current directory is used.
  SrcDir string
  // DstDir is the directory containing archived files.
  // If empty, SrcDir is used.
  // If both DstDir and SrcDir are empty, current directory is used.
  DstDir string
  // Pattern is the file name pattern to be manged.
  // Pattern must not contain directories.
  // Pattern must not be empty when ActiveFile is not specified.
  // A ".%i" will be added when no specifiers are found.
  // Following specifiers can be used in the pattern.
  //   %Y : YYYY 4 digits year. 0 <= YYYY
  //   %M : MM 2 digits month. 1 <= MM <= 12
  //   %D : DD 2 digits day of month. 1 <= DD <= 31
  //   %h : hh 2 digits hour. 0 <= hh <= 23
  //   %m : mm 2 digits minute. 0 <= mm <= 59
  //   %s : ss 2 digits second. 0 <= ss <= 59
  //   %u : unix second with free digits. 0 <= unix
  //   %i : index with free digits. 0 <= index
  //   %H : hostname
  //   %U : user id. "-1" on windows.
  //   %G : user group id. "-1" on windows.
  //   %p : pid (process id)
  //   %P : ppid (parent process id)
  Pattern string
  // ActiveFile is the file name that will be used for write target.
  // ActiveFile is optional and can be used to use fixed file name.
  // ActiveFile must not contain directories.
  // Following specifiers can be used in the name.
  //   %H : hostname
  //   %U : user id. "-1" on windows.
  //   %G : user group id. "-1" on windows.
  //   %p : pid (process id)
  //   %P : ppid (parent process id)
  ActiveFile string
}

The Pattern is the archived file name patterns and the ActiveFile is the file name of current write target. If ActiveFile is not set, the Pattern is used to create a new physical file.

ActiveFile @ SrcDir
  application.log
      │
      │ rotation
      ▼
Archive files @ DstDir
  - application.0.log
  - application.1.log
  - application.2.log

Usages

MaxAge

Use MaxAge to remove archived files older than the specified age.

For UseParsedAge=false, file ages are calculated from the file's last modification time. For UseParsedAge=true, file ages are calculated from timestamp parsed from the file names. The Pattern must contain timestamp when UseParsedAge=true.

• UseParsedAge=false: age is calculated from the file modification time.
• UseParsedAge=true: age is calculated from the timestamp parsed from the filename.

Examples:

// Use last modified time to calculate file ages.
config1 = &logicalfile.FileConfig{
  DstDir:       "./logs",
  Pattern:      "application.%i.log",
  MaxAge:       30 * time.Second,
  UseParsedAge: false, // Use last modified time.
  MaxBytes:     500,   // for single file
}

// Use timestamp parsed from file name to calculate file ages.
// Timestamp points the time when files are archived.
var config2 = &logicalfile.FileConfig{
  DstDir:       "./logs",
  Pattern:      "application.%Y-%M-%D.%h-%m-%s.log",
  ActiveFile:   "application.log", // set non-empty
  MaxAge:       30 * time.Second,
  UseParsedAge: true, // Parse timestamp from filename.
  MaxBytes:     500,  // for single file
}

// Use timestamp parsed from file name to calculate file ages.
// Timestamp points the time when files are created.
var config3 = &logicalfile.FileConfig{
  DstDir:       "./logs",
  Pattern:      "application.%Y-%M-%D.%h-%m-%s.log",
  ActiveFile:   "", // do not set
  MaxAge:       30 * time.Second,
  UseParsedAge: true, // Parse timestamp from filename.
  MaxBytes:     500,  // for single file
}

MaxTotalBytes

MaxTotalBytes limits the total size of archived files.

Example:

config := &logicalfile.FileConfig{
  DstDir:        "./logs",
  Pattern:       "application.%i.log",
  ActiveFile:    "application.log",
  MaxTotalBytes: 4 * 500, // 4 files
  MaxBytes:      500,     // for single file
}

MaxHistory

MaxHistory limits the number of archived files.

Example:

config := &logicalfile.FileConfig{
  DstDir:     "./logs",
  Pattern:    "application.%i.log",
  ActiveFile: "application.log",
  MaxHistory: 3,
  MaxBytes:   500, // for single file
}

Use all configs

If multiple retention limits are configured, an archived file is removed when any of the configured retention limits is exceeded:

MaxAge exceeded OR MaxHistory exceeded OR MaxTotalBytes exceeded

config := &logicalfile.FileConfig{
  SrcDir:        "./src",
  DstDir:        "./dst",
  Pattern:       "application.%i.log",
  ActiveFile:    "application.log",
  MaxBytes:      500, // for single file
  MaxAge:        30 * time.Second,
  MaxHistory:    3,
  MaxTotalBytes: 4 * 500,
}

Rotate at specific time

RotateAt can specify the time when the active file is rotated next time.

This example is the configuration to rotate files every 10 seconds.

now := time.Now()

config := &logicalfile.FileConfig{
  DstDir:     "./logs",
  Pattern:    "application.%h-%m-%s.log",
  ActiveFile: "application.log",
  RotateAt: func() time.Time {
    next := now.Add(10 * time.Second)
    now = next
    return next
  },
}

No rotations

If no rotation limits are configured, the physical file will not be rotated.

// Create config with no physical file rotations.
config := &logicalfile.FileConfig{
  ActiveFile: "application.log",
}

f, err := logicalfile.NewFile(config)
if err != nil {
  panic(err)
}
defer f.Close()

// Use logical file for log output.
log.SetOutput(f)

for {
  log.Println("hello, gopher !!")
  time.Sleep(time.Second)
}

Docs & Examples

• GoDoc: https://pkg.go.dev/github.com/aileron-projects/go-logicalfile
• Examples:
  • MaxAge: examples/max-age/
  • MaxHistory: examples/max-history/
  • MaxTotalBytes: examples/max-total/
  • All limits: examples/all/
  • Rotate at specific time: examples/cron/
  • No rotation: examples/no-rotations/

References

• github.com/natefinch/lumberjack
• github.com/DeRuina/timberjack
• github.com/easyCZ/logrotate
• github.com/lestrrat-go/file-rotatelogs
• Rolling file - logback
• Rolling file appenders -log4j 2

Documentation

Index

• Constants
• Variables
• func ReplaceCompressor(ext string, compress func(src, dst string, level int) error) error
• type Error
  • func (e *Error) Error() string
  • func (e *Error) Is(target error) bool
  • func (e *Error) Unwrap() error
• type File
  • func NewFile(c *FileConfig) (*File, error)
  • func (f *File) Close() error
  • func (f *File) Rotate() error
  • func (f *File) Write(b []byte) (int, error)
• type FileConfig
• type FileProvider
• type ManagerConfig

Constants

const (
	OpCompress   = "compress"
	OpListFiles  = "list files"
	OpRemove     = "remove file "
	OpOpen       = "open file "
	OpClose      = "close file "
	OpRename     = "rename file "
	OpArchive    = "archive"
	OpNewManager = "new manager"
)

Variables

var (
	ErrFileClosed = errors.New("go-logicalfile/logicalfile: file already closed")
	ErrEmptyExt   = errors.New("go-logicalfile/logicalfile: file extension must no be empty")
	ErrNilFunc    = errors.New("go-logicalfile/logicalfile: compression function must no be nil")
	ErrInplace    = errors.New("go-logicalfile/logicalfile: inplace compression is not supported")
	ErrNilWriter  = errors.New("go-logicalfile/logicalfile: inner writer is nil. cannot write")
)

Functions

func ReplaceCompressor

func ReplaceCompressor(ext string, compress func(src, dst string, level int) error) error

ReplaceCompressor replaces default compression method. It changes global state. ext should be file extension such as ".gz". Empty is not allowed. compress is the function that compress src file to dst file with the level. compress must not be nil.

Types

type Error

type Error struct {
	Inner  error  // Inner is the inner error.
	Op     string // Op is the operation.
	Msg    string // Msg is the error message.
	Detail string // Details is the error detail.
}

Error is the error type.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Is

func (e *Error) Is(target error) bool

func (*Error) Unwrap

func (e *Error) Unwrap() error

type File

type File struct {
	// contains filtered or unexported fields
}

File is a logical file.

func NewFile

func NewFile(c *FileConfig) (*File, error)

NewFile returns a new File.

func (*File) Close

func (f *File) Close() error

Close closes the logical file. Once Close is called, this file cannot be used to write. It returns nil when the file is already closed.

func (*File) Rotate

func (f *File) Rotate() error

Rotate renew the underlying writer which means file rotation. Calling Rotate forces file rotation. It returns ErrFileClosed if the file has already been closed.

func (*File) Write

func (f *File) Write(b []byte) (int, error)

Write writes the given data in to the file. It will be immediately written to the underlying physical file. Write implements io.Writer.Write. Write is safe for concurrent call.

type FileConfig

type FileConfig struct {
	// ManagerConfig is the configuration of physical file manager.
	ManagerConfig
	// MaxBytes is the maximum size in bytes.
	// Target writer will be renewed before exceeding the size.
	// Zero or negative means no limit, or no writer rotation.
	MaxBytes uint64
	// RotateAt optionally specifies the time to rotate.
	// RotateAt can be used for rotating physical file .
	// RotateAt must return the future time.
	RotateAt func() time.Time
	// HandleError optionally handles errors occurred in a logical file.
	HandleError func(error)
	// Fallback is the function that called when writing to the
	// current target io writer failed.
	// If nil, os.Stderr.Write is used by default.
	Fallback func([]byte) (int, error)
}

FileConfig is the configuration for a logical file. Use NewFile to create a new logical file.

type FileProvider

type FileProvider interface {
	Put(io.Writer) error
	Get() (io.Writer, int64, error)
	Wait()
}

FileProvider provides file as io writer.

type ManagerConfig

type ManagerConfig struct {
	// MaxAge is the maximum age to keep the archived files.
	// Age is calculated based on the last modified time
	// obtained by [io/fs.FileInfo].ModTime().
	// Set UseParsedAge to true to calculate ages using
	// the time parsed from archived files.
	MaxAge time.Duration
	// MaxHistory is the maximum number of archived files to keep.
	// Zero or negative value means no limit.
	MaxHistory int
	// MaxTotalBytes is the maximum total byte size to keep archived files.
	// Zero or negative value means no limit.
	MaxTotalBytes uint64
	// UseParsedAge, if true, calculate file ages based on the time
	// parsed from archived file names.
	// Note that time information must be contained in the Pattern.
	// Missing time components are initialized to their minimum values.
	// If %Y is not in the Pattern, it will be 1.
	// If %M is not in the Pattern, it will be 1 (January).
	// If %D is not in the Pattern, it will be 1.
	// If %h is not in the Pattern, it will be 00.
	// If %m is not in the Pattern, it will be 00.
	// If %s is not in the Pattern, it will be 00.
	UseParsedAge bool
	// CompressLv specifies the compression level.
	// Zero means no compression.
	CompressLv int
	// SrcDir is the directory containing source files.
	// If empty, DstDir is used.
	// If both SrcDir and DstDir are empty, current directory is used.
	SrcDir string
	// DstDir is the directory containing archived files.
	// If empty, SrcDir is used.
	// If both DstDir and SrcDir are empty, current directory is used.
	DstDir string
	// Pattern is the file name pattern to be manged.
	// Pattern must not contain directories.
	// Pattern must not be empty when ActiveFile is not specified.
	// A ".%i" will be added when no specifiers are found.
	// Following specifiers can be used in the pattern.
	//	%Y : YYYY 4 digits year. 0 <= YYYY
	//	%M : MM 2 digits month. 1 <= MM <= 12
	//	%D : DD 2 digits day of month. 1 <= DD <= 31
	//	%h : hh 2 digits hour. 0 <= hh <= 23
	//	%m : mm 2 digits minute. 0 <= mm <= 59
	//	%s : ss 2 digits second. 0 <= ss <= 59
	//	%u : unix second with free digits. 0 <= unix
	//	%i : index with free digits. 0 <= index
	//	%H : hostname
	//	%U : user id. "-1" on windows.
	//	%G : user group id. "-1" on windows.
	//	%p : pid (process id)
	//	%P : ppid (parent process id)
	Pattern string
	// ActiveFile is the file name that will be used for write target.
	// ActiveFile is optional and can be used to use fixed file name.
	// ActiveFile must not contain directories.
	// Following specifiers can be used in the name.
	//	%H : hostname
	//	%U : user id. "-1" on windows.
	//	%G : user group id. "-1" on windows.
	//	%p : pid (process id)
	//	%P : ppid (parent process id)
	ActiveFile string
}

ManagerConfig is the configuration for the physical file manager.