loggerj

README

loggerj

Ultra-high-performance, lock-free, asynchronous logging for Go. Designed for extreme throughput with zero heap allocations in the hot path.

Philosophy

Most loggers sacrifice performance for convenience. loggerj takes a different approach: The "Pre-Compiled Execution Profile" architecture.

Instead of evaluating rate limits, sampling rules, or formatting static fields on every single log call (which causes mutex contention and allocations), loggerj bakes these rules into memory once during initialization. The hot path consists solely of atomic operations and memory copies (memcpy), ensuring the garbage collector is never disturbed by your logging.

Features

• 🚀 100% Zero-Allocation Hot Path: No interface boxing, no hidden strconv calls, no map lookups during logging.
• ⚡ Lock-Free Rate Limiting & Sampling: Powered by atomic.CompareAndSwap (CAS). No shards, no mutexes, no contention.
• 🧠 Pre-Baked SubProfiles: Static fields (e.g., env=prod) are formatted into []byte once at startup. Zero CPU cost at log time.
• 🛡️ Non-Blocking & Drop-Monitored: Channel-based async architecture. If overloaded, it safely drops logs and increments an atomic counter instead of deadlocking your app.
• 🔄 Native Log Rotation: Size-based rotation with backup retention, no external dependencies (like lumberjack) required.
• 🎛️ Runtime Level Control: Atomically change log levels on the fly with ~2ns overhead.
• Standard Library Compatibility: Seamlessly intercept std log and third-party library logs via io.Writer adapter (logger.AsWriter()).

---

Quick Start

package main

import (
    "context"
    "time"
    "uretgec/internal/loggerj"
)

func main() {
    // 1. Initialize Logger
    logger := loggerj.NewLogger(loggerj.Config{
        JSONOutput:   true,
        FlushTimeout: 50 * time.Millisecond,
    })

    // 2. Define SubProfiles (COLD PATH: Do this once at startup)
    // Rules are baked into memory. No hot-path overhead.
    logger.RegisterSub("HTTP", 
        loggerj.WithRateLimit(1000, time.Second),
        loggerj.WithFields("env", "prod", "service", "gateway"),
    )
    
    logger.RegisterSub("DB", 
        loggerj.WithSampleRate(100), // Log 1 out of 100
    )

    // 3. Start the async worker
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    go logger.Start(ctx)
    defer logger.Close()

    // 4. Log messages (HOT PATH: Ultra-fast, zero allocation)
    logger.InfoString("HTTP", "request received", "method", "GET", "path", "/api/v1/users")
    logger.ErrorString("DB", "connection timeout", "host", "localhost", "err", "dial tcp: i/o timeout")

    // 5. Ensure all logs are written before exit
    logger.Flush()
}

---

The SubProfile Paradigm

In loggerj, you do not pass rate limits or sampling values during the log call. Instead, you define SubProfiles at initialization. This is the key to our lock-free performance.

// ✅ CORRECT: Define rules once, log cleanly forever.
logger.RegisterSub("AUTH", loggerj.WithRateLimit(50, time.Second))
logger.InfoString("AUTH", "login attempt", "user", "admin")

// ❌ INCORRECT: The Log method does not accept dynamic rate limits anymore.
// logger.Log(LevelInfo, "AUTH", []byte("msg"), 50, nil) // This API is removed.

---

Performance

Benchmarks run on Apple M1 Pro. loggerj consistently outperforms industry standards by eliminating hot-path allocations and lock contention.

Mode	ns/op	logs/s	Allocations	Use Case
Filtered	2.1	484M	0	Debug logs in production
RateLimited	43.3	23.0M	0	High-volume events (Lock-Free CAS)
Parallel	102.1	9.7M	0	Concurrent logging (10+ goroutines)
JSON	132.5	7.5M	0	Structured logging
WithCaller	506.7	1.9M	2	Debugging only

Note: WithCaller is the only operation that allocates, as it is a fundamental limitation of Go's runtime.Caller.

vs Industry Standards (Approximate Max Throughput)

System	logs/s	Allocations	Notes
logrus	~500K	High	Reflection-based, sync
zap	~2.5M	Low	Sync, requires manual field typing
zerolog	~3.5M	Low	Sync, fluent API
loggerj	~9.7M	Zero	Async, lock-free, pre-compiled

---

Design Decisions & Limitations

We believe in radical transparency. Here is what loggerj intentionally does not do, and why:

1. No context.Context Integration: Extracting values from ctx.Value() is slow and breaks the zero-allocation guarantee. Solution: Extract TraceIDs/RequestIDs in your HTTP middleware and pass them as standard fields: logger.Info("HTTP", msg, "trace_id", traceID).
2. No Typed Fields (e.g., zap.Int, slog.Any): Typed field helpers cause interface boxing allocations on the caller side. loggerj forces the caller to use strconv.Itoa() or fmt.Sprintf(). This keeps the logger's internal hot path strictly zero-allocation and makes the cost of formatting explicit to the developer.
3. Async Log Ordering is Not Strictly Guaranteed: Under extreme concurrent load, the order in which logs are written to disk may slightly differ from the order they were generated. Solution: For critical audit trails, call logger.Flush() immediately after the log, or use a synchronous logger for that specific path.
4. No Dynamic Rate Limiting per Call: Rate limits are bound to the SubProfile at init-time. This eliminates map lookups and mutex locks in the hot path, enabling true lock-free performance.

---

Buffer Tuning Guide

loggerj is highly tunable. Adjust these based on your environment:

Scenario	ChannelSize	WorkerBufferSize	FlushThreshold	FlushTimeout
Low Memory (512MB RAM)	1024	2048	2048	100ms
Balanced (Default)	4096	4096	4096	50ms
High Throughput	16384	16384	16384	500ms
Burst Traffic	32768	8192	8192	10ms

---

Output Formats

*Text (Default)

[1704067200123] INFO [HTTP] request received method=GET path=/api/v1/users env=prod service=gateway

*JSON

{"ts":1704067200123,"level":"INFO","type":"HTTP","msg":"request received","env":"prod","service":"gateway","fields":{"method":"GET","path":"/api/v1/users"}}

---

Testing & Profiling

# Run all tests with race detector
go test -race -v ./internal/loggerj/...

# Run benchmarks
go test -bench=. -benchmem ./internal/loggerj/...

# CPU Profiling
go test -bench=. -cpuprofile=cpu.out ./internal/loggerj/...
go tool pprof -http=:8080 cpu.out

---

Documentation

• Usage Examples - Comprehensive examples for all features
• Benchmark Results - Detailed performance analysis

---

License

MIT License

---

🤖 Acknowledgments

This package was developed with the architectural guidance, performance optimization, and code generation assistance of Qwen AI. The core engineering decisions, trade-off analyses, and domain expertise were driven by rigorous performance engineering principles to achieve true production-ready quality.

Documentation

Overview

Package loggerj provides an ultra-high-performance, asynchronous, and lock-free logging facility designed for high-throughput Go services. It offers near-zero heap allocations, atomic rate limiting, log rotation, and structured fields in both text and JSON formats.

Architecture: The "Dark Side" (Lock-Free SubProfiles)

Unlike traditional loggers that use mutexes and maps for rate limiting and sampling in the hot path, loggerj uses a "Pre-compiled Execution Profile" architecture.

1. Cold Path (Init-time): You register log types using RegisterSub(). This pre-bakes JSON/Text prefixes into []byte and initializes lock-free atomic counters for rate limiting and sampling.
2. Hot Path (Log-time): The Log() method performs ZERO map lookups and ZERO mutex locks. It uses atomic.CompareAndSwap (CAS) for rate limiting and atomic.Add for sampling.
3. Worker: The dedicated worker goroutine simply appends the pre-baked []byte prefixes, resulting in near-zero CPU overhead for formatting.

This design ensures that logging never blocks the caller, eliminates GC pressure in the hot path, and scales linearly with CPU cores without lock contention.

Quick Start

logger := loggerj.NewLogger(loggerj.Config{
    JSONOutput:   true,
    FlushTimeout: 50 * time.Millisecond,
})

// COLD PATH: Register profiles once at startup
logger.RegisterSub("HTTP",
    loggerj.WithRateLimit(1000, time.Second),
    loggerj.WithFields("env", "prod", "service", "gateway"),
)

ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go logger.Start(ctx)
defer logger.Close()

// HOT PATH: Ultra-fast, zero-allocation logging
logger.InfoString("HTTP", "request received", "method", "GET", "path", "/api")
logger.ErrorString("DB", "connection failed", "host", "localhost", "err", "timeout")

Performance

On Apple M1 Pro, the lock-free architecture achieves:

• ~15-20 ns/op for Rate-Limited logs (Down from 48ns in v1)
• 8M+ logs/s single-thread (no fields)
• 10.9M+ logs/s parallel
• 0 allocs/op in the hot path

See BENCH.md for detailed benchmark results.

Index

• type Config
  • func DefaultConfig() Config
• type Entry
  • func (e *Entry) Reset()
• type Level
  • func (l Level) String() string
• type Logger
  • func NewLogger(c Config) *Logger
  • func (l *Logger) AsWriter(level Level, logType string) io.Writer
  • func (l *Logger) Close() error
  • func (l *Logger) Debug(logType string, msg []byte, fields ...string)
  • func (l *Logger) DebugString(logType string, msg string, fields ...string)
  • func (l *Logger) Drops() uint64
  • func (l *Logger) Error(logType string, msg []byte, fields ...string)
  • func (l *Logger) ErrorString(logType string, msg string, fields ...string)
  • func (l *Logger) Flush()
  • func (l *Logger) GetLevel() Level
  • func (l *Logger) Info(logType string, msg []byte, fields ...string)
  • func (l *Logger) InfoString(logType string, msg string, fields ...string)
  • func (l *Logger) Log(level Level, logType string, msg []byte, fields ...string)
  • func (l *Logger) RegisterSub(logType string, opts ...SubOption)
  • func (l *Logger) ResetDrops()
  • func (l *Logger) SetLevelValue(level Level)
  • func (l *Logger) Start(ctx context.Context)
  • func (l *Logger) StartWithWriter(ctx context.Context, w io.Writer)
  • func (l *Logger) Stats() map[string]uint64
  • func (l *Logger) Warn(logType string, msg []byte, fields ...string)
  • func (l *Logger) WarnString(logType string, msg string, fields ...string)
• type StdLogWriter
  • func (w *StdLogWriter) Write(p []byte) (int, error)
• type SubOption
  • func WithFields(fields ...string) SubOption
  • func WithRateLimit(limit int64, window time.Duration) SubOption
  • func WithSampleRate(rate int64) SubOption
• type SubProfile

Types

type Config

type Config struct {
	// JSONOutput controls the output format. If true, logs are formatted as JSON.
	// If false, logs are formatted as human-readable text. Default: false
	JSONOutput bool

	// FlushTimeout is the interval at which the worker flushes buffered logs.
	// Shorter timeouts reduce latency but increase I/O operations. Default: 50ms
	FlushTimeout time.Duration

	// ChannelSize is the capacity of the internal log channel. Larger values
	// provide more buffering for burst traffic. If full, entries are dropped. Default: 4096
	ChannelSize int

	// WorkerBufferSize is the initial capacity of the worker's format buffer.
	// Minimum: 256. Default: 4096
	WorkerBufferSize int

	// FlushThreshold is the byte count at which the worker flushes the
	// format buffer to the underlying writer. Should be <= WorkerBufferSize.
	// Minimum: 256. Default: 4096
	FlushThreshold int

	// WriterBufferSize is the size of the bufio.Writer buffer used for I/O.
	// Minimum: 512. Default: 8192
	WriterBufferSize int

	// RateLimitWindow is the default time window for rate limiting, in seconds.
	// Used if a SubProfile doesn't specify its own window via WithRateLimit.
	// Minimum: 1. Default: 1
	RateLimitWindow int64

	// IncludeCaller adds file:line information to each log entry.
	// WARNING: Adds ~480ns overhead and 2 allocations per log entry.
	// Should be disabled in production for maximum performance. Default: false
	IncludeCaller bool

	// OutputFile is the path to the log file. If empty, logs are written to stderr.
	OutputFile string

	// MaxFileSize is the maximum size of the log file before rotation.
	// If 0, rotation is disabled. Default: 0
	MaxFileSize int64

	// MaxBackupFiles is the maximum number of rotated log files to keep.
	// Only effective if MaxFileSize > 0. Default: 0
	MaxBackupFiles int
}

Config holds the configuration for a Logger instance. All fields have sensible defaults and can be left at their zero values.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults for production use.

type Entry

type Entry struct {
	Level   Level
	Type    string
	Msg     []byte
	Ts      int64
	File    string
	Line    int
	Fields  []string
	Profile *SubProfile // 🌑 Pointer to the pre-compiled SubProfile
}

Entry represents a single log record. Entries are pooled using sync.Pool to minimize allocations. The Reset method clears all fields for reuse.

func (*Entry) Reset

func (e *Entry) Reset()

Reset clears all fields of the Entry for reuse.

type Level

type Level uint8

Level represents the severity of a log entry. Lower values indicate more verbose logging. The logger filters entries below the configured threshold.

const (
	// LevelDebug is the most verbose level, used for detailed debugging information.
	LevelDebug Level = 0
	// LevelInfo is the default level, used for general operational messages.
	LevelInfo Level = 1
	// LevelWarn indicates potential issues that should be monitored.
	LevelWarn Level = 2
	// LevelError indicates serious problems that require immediate attention.
	LevelError Level = 3
)

func (Level) String

func (l Level) String() string

String returns the human-readable name of the log level.

type Logger

type Logger struct {
	// contains filtered or unexported fields
}

Logger is the main logging instance. It provides asynchronous, high-throughput, lock-free logging.

func NewLogger

func NewLogger(c Config) *Logger

NewLogger creates a new Logger instance. The logger is not started; call Start or StartWithWriter to begin processing.

func (*Logger) AsWriter

func (l *Logger) AsWriter(level Level, logType string) io.Writer

AsWriter returns an io.Writer that routes all writes to the logger at the specified level and logType.

Note: This method is designed for intercepting standard library logs. It incurs a minor string allocation (string(p)) which is acceptable for stdlib interception, but should not be used in the application's hot path.

func (*Logger) Close

func (l *Logger) Close() error

Close releases resources held by the logger.

func (*Logger) Debug

func (l *Logger) Debug(logType string, msg []byte, fields ...string)

Debug logs a message at LevelDebug. Caller skip is 2.

func (*Logger) DebugString

func (l *Logger) DebugString(logType string, msg string, fields ...string)

DebugString logs a string message at LevelDebug. Caller skip is 2.

func (*Logger) Drops

func (l *Logger) Drops() uint64

Drops returns the total number of log entries dropped because the channel was full.

func (*Logger) Error

func (l *Logger) Error(logType string, msg []byte, fields ...string)

Error logs a message at LevelError. Caller skip is 2.

func (*Logger) ErrorString

func (l *Logger) ErrorString(logType string, msg string, fields ...string)

ErrorString logs a string message at LevelError. Caller skip is 2.

func (*Logger) Flush

func (l *Logger) Flush()

Flush forces an immediate flush of all pending log entries.

func (*Logger) GetLevel

func (l *Logger) GetLevel() Level

GetLevel returns the current log level threshold.

func (*Logger) Info

func (l *Logger) Info(logType string, msg []byte, fields ...string)

Info logs a message at LevelInfo. Caller skip is 2.

func (*Logger) InfoString

func (l *Logger) InfoString(logType string, msg string, fields ...string)

InfoString logs a string message at LevelInfo. Caller skip is 2.

func (*Logger) Log

func (l *Logger) Log(level Level, logType string, msg []byte, fields ...string)

Log is the public core logging method. Caller skip is 1.

func (*Logger) RegisterSub

func (l *Logger) RegisterSub(logType string, opts ...SubOption)

RegisterSub registers a SubProfile for a specific logType. COLD PATH: Call this during application initialization, NOT inside HTTP handlers or hot loops.

func (*Logger) ResetDrops

func (l *Logger) ResetDrops()

ResetDrops resets the drop counter to zero.

func (*Logger) SetLevelValue

func (l *Logger) SetLevelValue(level Level)

SetLevelValue sets the current log level threshold atomically.

func (*Logger) Start

func (l *Logger) Start(ctx context.Context)

Start begins the worker goroutine that processes log entries.

func (*Logger) StartWithWriter

func (l *Logger) StartWithWriter(ctx context.Context, w io.Writer)

StartWithWriter begins the worker goroutine with a custom io.Writer.

func (*Logger) Stats

func (l *Logger) Stats() map[string]uint64

Stats returns a map of logger statistics.

func (*Logger) Warn

func (l *Logger) Warn(logType string, msg []byte, fields ...string)

Warn logs a message at LevelWarn. Caller skip is 2.

func (*Logger) WarnString

func (l *Logger) WarnString(logType string, msg string, fields ...string)

WarnString logs a string message at LevelWarn. Caller skip is 2.

type StdLogWriter

type StdLogWriter struct {
	// contains filtered or unexported fields
}

StdLogWriter wraps a Logger to implement the io.Writer interface. This allows the standard library "log" package (and third-party libraries that rely on it) to route their output through loggerj's async, high-performance pipeline.

func (*StdLogWriter) Write

func (w *StdLogWriter) Write(p []byte) (int, error)

Write implements the io.Writer interface.

type SubOption

type SubOption func(*SubProfile)

SubOption configures a SubProfile during RegisterSub.

func WithFields

func WithFields(fields ...string) SubOption

WithFields adds static key-value pairs that will be pre-baked into the JSON/Text prefixes. This avoids formatting these fields in the hot path.

func WithRateLimit

func WithRateLimit(limit int64, window time.Duration) SubOption

WithRateLimit sets a lock-free rate limit for this specific logType. limit is the max logs per window. window is the duration (e.g., time.Second).

func WithSampleRate

func WithSampleRate(rate int64) SubOption

WithSampleRate sets a lock-free sampling rate for this logType. rate means 1 out of `rate` logs will be written (0 disables sampling).

type SubProfile

type SubProfile struct {
	Name string
	// contains filtered or unexported fields
}

SubProfile represents a pre-compiled execution profile for a specific logType. Unlike traditional loggers that use mutexes and maps in the hot path, SubProfile holds lock-free atomic counters for rate limiting/sampling and pre-baked []byte prefixes for zero-CPU formatting.