loggerj

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 19, 2026 License: MIT Imports: 12 Imported by: 0

README

Go Version Built with Qwen AI

loggerj

Ultra-high-performance, lock-free, asynchronous logging for Go. Designed for extreme throughput with zero heap allocations in the hot path.

Philosophy

Most loggers sacrifice performance for convenience. loggerj takes a different approach: The "Pre-Compiled Execution Profile" architecture.

Instead of evaluating rate limits, sampling rules, or formatting static fields on every single log call (which causes mutex contention and allocations), loggerj bakes these rules into memory once during initialization. The hot path consists solely of atomic operations and memory copies (memcpy), ensuring the garbage collector is never disturbed by your logging.

Features

  • 🚀 100% Zero-Allocation Hot Path: No interface boxing, no hidden strconv calls, no map lookups during logging. String-to-byte conversion uses unsafe zero-copy (Go 1.21+).
  • ⚡ Lock-Free Rate Limiting & Sampling: Powered by atomic.CompareAndSwap (CAS). No shards, no mutexes, no contention.
  • 🧠 Pre-Baked SubProfiles: Static fields (e.g., env=prod) are formatted into []byte once at startup. Zero CPU cost at log time.
  • 📋 Copy-on-Write Profile Registry: Profiles are stored in an immutable registry accessed via atomic.Pointer. Unlimited profiles, zero interface boxing, lock-free reads.
  • 🛡️ Non-Blocking & Drop-Monitored: Channel-based async architecture. If overloaded, it safely drops logs and increments an atomic counter instead of deadlocking your app. Optional SetOnDrop callback for real-time monitoring.
  • 🔄 Native Log Rotation: Size-based rotation with backup retention, no external dependencies (like lumberjack) required.
  • 🎛️ Runtime Level Control: Atomically change log levels on the fly with ~2ns overhead.
  • 🔗 Standard Library Compatibility: Seamlessly intercept std log and third-party library logs via io.Writer adapter (logger.AsWriter()).
  • 🌐 Context Integration (Opt-in): Extract trace_id, request_id, span_id from context.Context with zero cost when unused.
  • ⏱️ Worker-Side Timestamps: Timestamps are captured by the worker goroutine at format time, removing a vDSO syscall from the hot path.
  • ✅ Deterministic Flush: Flush() drains all pending channel entries before writing, guaranteeing no log loss on explicit flush.

Quick Start

package main

import (
    "context"
    "time"

    "github.com/uretgec/loggerj"
)

func main() {
    // 1. Initialize Logger
    logger := loggerj.NewLogger(loggerj.Config{
        JSONOutput:   true,
        FlushTimeout: 50 * time.Millisecond,
    })

    // 2. Define SubProfiles (COLD PATH: Do this once at startup)
    // Rules are baked into memory. No hot-path overhead.
    logger.RegisterSub("HTTP",
        loggerj.WithRateLimit(1000, time.Second),
        loggerj.WithFields("env", "prod", "service", "gateway"),
    )
    logger.RegisterSub("DB",
        loggerj.WithSampleRate(100), // Log 1 out of 100
    )

    // 3. Start the async worker
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    go logger.Start(ctx)
    defer logger.Close()

    // 4. Log messages (HOT PATH: Ultra-fast, zero allocation)
    logger.InfoString("HTTP", "request received", "method", "GET", "path", "/api/v1/users")
    logger.ErrorString("DB", "connection timeout", "host", "localhost", "err", "dial tcp: i/o timeout")

    // 5. Context-aware logging (opt-in, zero cost when unused)
    ctx = context.WithValue(ctx, loggerj.TraceIDKey, "abc-123")
    logger.InfoCtx(ctx, "HTTP", "traced request", "method", "POST")

    // 6. Ensure all logs are written before exit
    logger.Flush()
}

The SubProfile Paradigm

In loggerj, you do not pass rate limits or sampling values during the log call. Instead, you define SubProfiles at initialization. This is the key to our lock-free performance.

// ✅ CORRECT: Define rules once, log cleanly forever.
logger.RegisterSub("AUTH", loggerj.WithRateLimit(50, time.Second))
logger.InfoString("AUTH", "login attempt", "user", "admin")

// ❌ INCORRECT: The Log method does not accept dynamic rate limits anymore.
// logger.Log(LevelInfo, "AUTH", []byte("msg"), 50, nil) // This API is removed.

Performance

Benchmarks run on Apple M1 Pro (10 cores), Go 1.21+. loggerj consistently outperforms industry standards by eliminating hot-path allocations and lock contention.

Mode ns/op logs/s Allocs/op Use Case
Filtered 2.1 483M 0 Debug logs in production
Sampling 24 41M 0 High-volume sampled events
Dropped 19 53M 0 Channel-full backpressure
RateLimited 41 24M 0 High-volume events (Lock-Free CAS)
JSON 55 18M 0 Structured logging
StringAPI 63 16M 0 String messages (zero-copy)
NoFields 61 16M 0 Simple messages
Parallel 80 12.5M 0 Concurrent logging (10+ goroutines)
SubProfile Prefix 55 18M 0 Pre-baked static fields
WithCaller 463 2.2M 2 Debugging only
SyncEquivalent 1082 924K 3 Fair comparison with sync loggers

Note: WithCaller allocates due to Go's runtime.Caller — a fundamental limitation. SyncEquivalent forces Flush() after every log to simulate synchronous behavior; this is NOT the intended usage pattern.

vs Industry Standards (Approximate Max Throughput)
System logs/s Allocs/op Notes
logrus ~500K High Reflection-based, sync
zap ~2.5M Low Sync, requires manual field typing
zerolog ~3.5M Low Sync, fluent API
loggerj (async) ~16M Zero Async, lock-free, pre-compiled
loggerj (sync-equiv) ~924K 3 Forced Flush() per log (not intended usage)

Design Decisions & Limitations

We believe in radical transparency. Here is what loggerj intentionally does and does not do, and why:

Context Integration (Opt-in)

loggerj provides InfoCtx, DebugCtx, WarnCtx, ErrorCtx methods that extract known keys (TraceIDKey, RequestIDKey, SpanIDKey) from context.Context. This is opt-in: if you never call *Ctx methods, there is zero overhead. When used, only known string keys are extracted — no reflection, no fmt.Sprintf.

ctx = context.WithValue(ctx, loggerj.TraceIDKey, "abc-123")
logger.InfoCtx(ctx, "HTTP", "request", "method", "GET")
// Output includes: "trace_id":"abc-123"
No Typed Fields (e.g., zap.Int, slog.Any)

Typed field helpers cause interface boxing allocations on the caller side. loggerj forces the caller to use strconv.Itoa() or fmt.Sprintf(). This keeps the logger's internal hot path strictly zero-allocation and makes the cost of formatting explicit to the developer.

Async Log Ordering is Not Strictly Guaranteed

Under extreme concurrent load, the order in which logs are written to disk may slightly differ from the order they were generated. Timestamps are captured at format time (worker-side), not at call time. For critical audit trails, call logger.Flush() immediately after the log.

No Dynamic Rate Limiting per Call

Rate limits are bound to the SubProfile at init-time. This eliminates map lookups and mutex locks in the hot path, enabling true lock-free performance.

Buffer Tuning Guide

loggerj is highly tunable. Adjust these based on your environment:

Scenario ChannelSize WorkerBufferSize FlushThreshold FlushTimeout
Low Memory (512MB RAM) 1024 2048 2048 100ms
Balanced (Default) 4096 4096 4096 50ms
High Throughput 16384 16384 16384 500ms
Burst Traffic 32768 8192 8192 10ms

Output Formats

Text (Default)
[1704067200123] INFO [HTTP] request received method=GET path=/api/v1/users env=prod service=gateway
JSON
{"ts":1704067200123,"level":"INFO","type":"HTTP","msg":"request received","env":"prod","service":"gateway","fields":{"method":"GET","path":"/api/v1/users"}}

Testing & Profiling

# Run all tests with race detector
go test -race -v ./...

# Run benchmarks
go test -bench=. -benchmem ./...

# CPU Profiling
go test -bench=. -cpuprofile=cpu.out ./...
go tool pprof -http=:8080 cpu.out

Documentation

License

MIT License

🤖 Acknowledgments

This package was developed with the architectural guidance, performance optimization, and code generation assistance of Qwen AI. The core engineering decisions, trade-off analyses, and domain expertise were driven by rigorous performance engineering principles to achieve true production-ready quality.

Documentation

Overview

Package loggerj provides an ultra-high-performance, asynchronous, and lock-free logging facility designed for high-throughput Go services. It offers zero heap allocations in the hot path, atomic rate limiting, log rotation, and structured fields in both text and JSON formats.

Architecture: Pre-Compiled Execution Profiles

Unlike traditional loggers that use mutexes and maps for rate limiting and sampling in the hot path, loggerj uses a "Pre-compiled Execution Profile" architecture.

  1. Cold Path (Init-time): You register log types using RegisterSub(). This pre-bakes JSON/Text prefixes into []byte and initializes lock-free atomic counters for rate limiting and sampling. Profiles are stored in an immutable copy-on-write registry accessed via atomic.Pointer, ensuring zero interface boxing and zero map lookups in the hot path.
  2. Hot Path (Log-time): The Log() method performs ZERO map lookups, ZERO mutex locks, and ZERO heap allocations. It uses atomic.CompareAndSwap (CAS) for rate limiting and atomic.Add for sampling. String-to-byte conversion uses unsafe zero-copy (Go 1.21+). Timestamps are deferred to the worker goroutine, removing a vDSO syscall from the hot path.
  3. Worker: A dedicated goroutine formats entries, injects pre-baked []byte prefixes, and writes to the underlying io.Writer via bufio. Flush() drains the channel before writing, guaranteeing no log loss on explicit flush.

This design ensures that logging never blocks the caller, eliminates GC pressure in the hot path, and scales linearly with CPU cores without lock contention.

Quick Start 

logger := loggerj.NewLogger(loggerj.Config{
    JSONOutput:   true,
    FlushTimeout: 50 * time.Millisecond,
})

// COLD PATH: Register profiles once at startup
logger.RegisterSub("HTTP",
    loggerj.WithRateLimit(1000, time.Second),
    loggerj.WithFields("env", "prod", "service", "gateway"),
)

ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go logger.Start(ctx)
defer logger.Close()

// HOT PATH: Ultra-fast, zero-allocation logging
logger.InfoString("HTTP", "request received", "method", "GET", "path", "/api")
logger.ErrorString("DB", "connection failed", "host", "localhost", "err", "timeout")

// Context-aware logging (opt-in, zero cost when unused)
ctx = context.WithValue(ctx, loggerj.TraceIDKey, "abc-123")
logger.InfoCtx(ctx, "HTTP", "traced request", "method", "POST")

Performance

Benchmarks on Apple M1 Pro (10 cores), Go 1.21+: 

Filtered:    ~2.0 ns/op   (484M logs/s)   0 allocs/op
RateLimited: ~44 ns/op    (23M logs/s)    0 allocs/op
Parallel:    ~86 ns/op    (11.6M logs/s)  0 allocs/op
JSON:        ~65 ns/op    (15.3M logs/s)  0 allocs/op
StringAPI:   ~66 ns/op    (15.1M logs/s)  0 allocs/op
WithCaller:  ~461 ns/op   (2.2M logs/s)   2 allocs/op

See BENCH.md for detailed benchmark results and methodology.

Index

Constants

View Source 
const (
	// TraceIDKey is the context key for distributed trace identifiers.
	TraceIDKey contextKey = iota
	// RequestIDKey is the context key for request identifiers.
	RequestIDKey
	// SpanIDKey is the context key for span identifiers.
	SpanIDKey
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	// JSONOutput controls the output format. If true, logs are formatted as JSON.
	// If false, logs are formatted as human-readable text. Default: false
	JSONOutput bool

	// FlushTimeout is the interval at which the worker flushes buffered logs.
	// Shorter timeouts reduce latency but increase I/O operations. Default: 50ms
	FlushTimeout time.Duration

	// ChannelSize is the capacity of the internal log channel. Larger values
	// provide more buffering for burst traffic. If full, entries are dropped.
	// Default: 4096
	ChannelSize int

	// WorkerBufferSize is the initial capacity of the worker's format buffer.
	// Minimum: 256. Default: 4096
	WorkerBufferSize int

	// FlushThreshold is the byte count at which the worker flushes the
	// format buffer to the underlying writer. Should be <= WorkerBufferSize.
	// Minimum: 256. Default: 4096
	FlushThreshold int

	// WriterBufferSize is the size of the bufio.Writer buffer used for I/O.
	// Minimum: 512. Default: 8192
	WriterBufferSize int

	// RateLimitWindow is the default time window for rate limiting, in seconds.
	// Used if a SubProfile doesn't specify its own window via WithRateLimit.
	// Minimum: 1. Default: 1
	RateLimitWindow int64

	// IncludeCaller adds file:line information to each log entry.
	// WARNING: Adds ~460ns overhead and 2 allocations per log entry.
	// Should be disabled in production for maximum performance. Default: false
	IncludeCaller bool

	// OutputFile is the path to the log file. If empty, logs are written to stderr.
	OutputFile string

	// MaxFileSize is the maximum size of the log file before rotation.
	// If 0, rotation is disabled. Default: 0
	MaxFileSize int64

	// MaxBackupFiles is the maximum number of rotated log files to keep.
	// Only effective if MaxFileSize > 0. Default: 0
	MaxBackupFiles int
}

Config holds the configuration for a Logger instance. All fields have sensible defaults and can be left at their zero values.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults for production use.

type Entry 

type Entry struct {
	Level   Level
	Type    string
	Msg     []byte
	File    string
	Line    int
	Fields  []string
	Profile *SubProfile
}

Entry represents a single log record. Entries are pooled using sync.Pool to minimize allocations. The Reset method clears all fields for reuse.

Timestamps are not stored in the Entry; they are captured by the worker goroutine at format time, removing a vDSO syscall from the hot path.

func (*Entry) Reset 

func (e *Entry) Reset()

Reset clears all fields of the Entry for reuse. Large slices (Msg > 4096 bytes, Fields > 64 elements) are released to the garbage collector to prevent permanent memory retention in the pool.

type Level 

type Level uint8

Level represents the severity of a log entry. Lower values indicate more verbose logging. The logger filters entries below the configured threshold. 

const (
	// LevelDebug is the most verbose level, used for detailed debugging information.
	LevelDebug Level = 0
	// LevelInfo is the default level, used for general operational messages.
	LevelInfo Level = 1
	// LevelWarn indicates potential issues that should be monitored.
	LevelWarn Level = 2
	// LevelError indicates serious problems that require immediate attention.
	LevelError Level = 3
)

func (Level) String 

func (l Level) String() string

String returns the human-readable name of the log level. Unknown levels return "UNKNOWN" without allocation.

type Logger 

type Logger struct {
	// contains filtered or unexported fields
}

Logger is the main logging instance. It provides asynchronous, high-throughput, lock-free logging with zero heap allocations in the hot path.

func NewLogger 

func NewLogger(c Config) *Logger

NewLogger creates a new Logger instance. The logger is not started; call Start or StartWithWriter to begin processing.

func (*Logger) AsWriter 

func (l *Logger) AsWriter(level Level, logType string) io.Writer

AsWriter returns an io.Writer that routes all writes to the logger at the specified level and logType.

Usage with the standard library log package: 

log.SetFlags(0) // Disable std log timestamps; loggerj adds its own
log.SetOutput(logger.AsWriter(loggerj.LevelInfo, "STDLIB"))

Note: This adapter incurs a minor allocation (string(p)) per write. This is acceptable for intercepting legacy or third-party logs but should not be used for the application's primary high-throughput path.

func (*Logger) Close 

func (l *Logger) Close() error

Close releases resources held by the logger, including flushing the buffered writer and closing the log file handle.

func (*Logger) Debug 

func (l *Logger) Debug(logType string, msg []byte, fields ...string)

Debug logs a message at LevelDebug. Caller skip is 2.

func (*Logger) DebugCtx added in v1.1.0 

func (l *Logger) DebugCtx(ctx context.Context, logType string, msg string, fields ...string)

DebugCtx logs a message at LevelDebug with optional context extraction. If ctx is nil or contains no known keys, it behaves exactly like DebugString with zero additional cost.

func (*Logger) DebugString 

func (l *Logger) DebugString(logType string, msg string, fields ...string)

DebugString logs a string message at LevelDebug with zero-copy string-to-byte conversion. Caller skip is 2.

func (*Logger) Drops 

func (l *Logger) Drops() uint64

Drops returns the total number of log entries dropped because the internal channel was full.

func (*Logger) Error 

func (l *Logger) Error(logType string, msg []byte, fields ...string)

Error logs a message at LevelError. Caller skip is 2.

func (*Logger) ErrorCtx added in v1.1.0 

func (l *Logger) ErrorCtx(ctx context.Context, logType string, msg string, fields ...string)

ErrorCtx logs a message at LevelError with optional context extraction. If ctx is nil or contains no known keys, it behaves exactly like ErrorString with zero additional cost.

func (*Logger) ErrorString 

func (l *Logger) ErrorString(logType string, msg string, fields ...string)

ErrorString logs a string message at LevelError with zero-copy string-to-byte conversion. Caller skip is 2.

func (*Logger) Flush 

func (l *Logger) Flush()

Flush forces an immediate flush of all pending log entries. It signals the worker to drain the channel and write the buffer, then blocks until the worker confirms completion (or times out after 1 second).

If no worker is running (globalWriter is nil), Flush drains and discards all pending entries to prevent channel blockage.

func (*Logger) GetLevel 

func (l *Logger) GetLevel() Level

GetLevel returns the current log level threshold.

func (*Logger) Info 

func (l *Logger) Info(logType string, msg []byte, fields ...string)

Info logs a message at LevelInfo. Caller skip is 2.

func (*Logger) InfoCtx added in v1.1.0 

func (l *Logger) InfoCtx(ctx context.Context, logType string, msg string, fields ...string)

InfoCtx logs a message at LevelInfo with optional context extraction. If ctx is nil or contains no known keys, it behaves exactly like InfoString with zero additional cost.

func (*Logger) InfoString 

func (l *Logger) InfoString(logType string, msg string, fields ...string)

InfoString logs a string message at LevelInfo with zero-copy string-to-byte conversion. Caller skip is 2.

func (*Logger) Log 

func (l *Logger) Log(level Level, logType string, msg []byte, fields ...string)

Log is the public core logging method. Caller skip is 1.

func (*Logger) RegisterSub 

func (l *Logger) RegisterSub(logType string, opts ...SubOption)

RegisterSub registers a SubProfile for a specific logType.

COLD PATH: Call this during application initialization, NOT inside HTTP handlers or hot loops. The registry uses a copy-on-write strategy: a new immutable snapshot is created and swapped atomically, so concurrent hot-path reads are never blocked.

func (*Logger) ResetDrops 

func (l *Logger) ResetDrops()

ResetDrops resets the drop counter to zero.

func (*Logger) SetLevelValue 

func (l *Logger) SetLevelValue(level Level)

SetLevelValue sets the current log level threshold atomically. Entries below this level are discarded in ~2ns with zero allocations.

func (*Logger) SetOnDrop added in v1.1.0 

func (l *Logger) SetOnDrop(fn func(dropped uint64))

SetOnDrop registers a callback invoked whenever a log entry is dropped due to a full channel. The callback receives the current total drop count.

WARNING: This callback is invoked from the hot path. Keep it fast (e.g., atomic counter increment, metrics gauge update). Never perform I/O or acquire locks inside this callback.

func (*Logger) Start 

func (l *Logger) Start(ctx context.Context)

Start begins the worker goroutine that processes log entries. It writes to the configured OutputFile or stderr.

func (*Logger) StartWithWriter 

func (l *Logger) StartWithWriter(ctx context.Context, w io.Writer)

StartWithWriter begins the worker goroutine with a custom io.Writer.

Lifecycle synchronization:

  • The started channel is closed (via sync.Once) when the worker begins, allowing callers to wait for readiness without polling or sleeping.
  • The workerDone channel is closed when the worker exits (after draining remaining entries on context cancellation).

The Flush() method drains all pending channel entries before writing, guaranteeing no log loss on explicit flush.

func (*Logger) Stats 

func (l *Logger) Stats() map[string]uint64

Stats returns a snapshot of logger statistics.

func (*Logger) Warn 

func (l *Logger) Warn(logType string, msg []byte, fields ...string)

Warn logs a message at LevelWarn. Caller skip is 2.

func (*Logger) WarnCtx added in v1.1.0 

func (l *Logger) WarnCtx(ctx context.Context, logType string, msg string, fields ...string)

WarnCtx logs a message at LevelWarn with optional context extraction. If ctx is nil or contains no known keys, it behaves exactly like WarnString with zero additional cost.

func (*Logger) WarnString 

func (l *Logger) WarnString(logType string, msg string, fields ...string)

WarnString logs a string message at LevelWarn with zero-copy string-to-byte conversion. Caller skip is 2.

type StdLogWriter 

type StdLogWriter struct {
	// contains filtered or unexported fields
}

StdLogWriter wraps a Logger to implement the io.Writer interface. This allows the standard library "log" package (and third-party libraries that rely on it) to route their output through loggerj's async pipeline.

func (*StdLogWriter) Write 

func (w *StdLogWriter) Write(p []byte) (int, error)

Write implements the io.Writer interface. Trailing newlines from std log are trimmed for cleaner loggerj output.

type SubOption 

type SubOption func(*SubProfile)

SubOption configures a SubProfile during RegisterSub.

func WithFields 

func WithFields(fields ...string) SubOption

WithFields adds static key-value pairs that will be pre-baked into the JSON/Text prefixes. This avoids formatting these fields in the hot path.

func WithRateLimit 

func WithRateLimit(limit int64, window time.Duration) SubOption

WithRateLimit sets a lock-free rate limit for this specific logType. limit is the max logs per window. window is the duration (e.g., time.Second).

func WithSampleRate 

func WithSampleRate(rate int64) SubOption

WithSampleRate sets a lock-free sampling rate for this logType. rate means 1 out of `rate` logs will be written (0 disables sampling).

type SubProfile 

type SubProfile struct {
	Name string
	// contains filtered or unexported fields
}

SubProfile represents a pre-compiled execution profile for a specific logType. Unlike traditional loggers that use mutexes and maps in the hot path, SubProfile holds lock-free atomic counters for rate limiting/sampling and pre-baked []byte prefixes for zero-CPU formatting.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.