betterlogs

README

Better Logs

A lightweight, production‑ready logging toolkit designed for Go services that want structured logs, database persistence, and framework‑agnostic HTTP middleware, without relying on any external SaaS.

This README provides:

• Overview
• Why this package exists
• Install
• Basic usage
• HTTP middleware
• Postgres log core (batch writer)
• Context fields
• Integration examples
  • net/http
  • Echo
• Best practices (PII, redaction, retention)
• Technical choices

---

📌 Overview

Better Logs is a small, focused toolkit that provides:

• A Zap core that writes logs into PostgreSQL in batches
• A generic HTTP middleware compatible with any net/http router
• Context‑based structured logging fields (similar to logctx)
• Safe batching, backpressure and non‑blocking writes
• Plug‑and‑play usage: combine your stdout logger + pgcore

It is designed to be:

• Framework‑agnostic (Echo, Chi, net/http, Fiber* except Fiber doesn't use stdlib)
• Fully self‑hosted
• Simple to audit (no magic, no global state outside Zap)
• Low overhead

---

❓ Why This Package Exists

Most Go projects evolve into the same pattern:

• You start with stdout logs using Zap or Zerolog.
• Then you want logs searchable over time.
• But you do not want:
  • Loki + Grafana
  • Elasticsearch + Kibana
  • Cloud Logging / Datadog / Sentry (pricing, compliance, vendor lock‑in)

You just want:

• Keep stdout for local dev & simple operations
• Also write logs into a PostgreSQL logs table
• With retention
• And without blocking your handlers

Better Logs solves exactly that. Small footprint. No external dependencies. Just PostgreSQL + Zap.

---

📦 Install

go get github.com/ZiplEix/better-logs

---

🚀 Basic Usage

1. Initialize your logger

import (
    betterlogs "github.com/ZiplEix/better-logs"
    "go.uber.org/zap"
)

db := /* your *sql.DB */

cfg := betterlogs.DefaultConfig()
cfg.ServiceName = "my-service"
cfg.DB = db
cfg.EnablePostgres = true

logger, closeFn, err := betterlogs.New(cfg)
if err != nil {
    log.Fatalf("failed to init logger: %v", err)
}

zap.ReplaceGlobals(logger)

This returns:

• A *zap.Logger
• A cleanup function you call on shutdown
• An error if initialization failed

2. Use Zap everywhere

zap.L().Info("server started", zap.String("addr", ":8080"))

3. Close cleanly on shutdown

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

if err := closeFn(ctx); err != nil {
    log.Printf("logger cleanup error: %v", err)
}

---

🧱 Creating the logs table (CLI)

Better Logs ships a set of CLI utilities to manage the logs table in your PostgreSQL database.

Available commands:

• logs-migrate → Creates the table and indexes (idempotent)
• logs-drop → Drops the table
• logs-retention → Purges logs older than a given duration

---

🧱 1. Create the table — logs-migrate

go run github.com/ZiplEix/better-logs/cmd/logs-migrate@latest \
  -database-url "postgres://user:pass@host:5432/dbname?sslmode=disable"

This will:

• Connect to your PostgreSQL database
• Create the logs table and indexes if they do not exist
• Do nothing if the table already exists (idempotent)

Under the hood it runs:

• EnsureLogsTable

---

🗑️ 2. Drop the table — logs-drop

Useful for development or resetting logs.

go run github.com/ZiplEix/better-logs/cmd/logs-drop@latest \
  -database-url "postgres://user:pass@host:5432/dbname?sslmode=disable"

This will:

• Drop the logs table if it exists
• Fail silently if the table does not exist

Internally it runs:

DROP TABLE IF EXISTS logs;

---

🧹 3. Purge old logs — logs-retention

Deletes logs older than the provided duration.

Example: purge logs older than 7 days

go run github.com/ZiplEix/better-logs/cmd/logs-retention@latest \
  -database-url "$DATABASE_URL" \
  -older-than "168h"   # 7 days

Example: purge logs older than 30 days

go run github.com/ZiplEix/better-logs/cmd/logs-retention@latest \
  -database-url "$DATABASE_URL" \
  -older-than "720h"

Under the hood, the utility executes:

DELETE FROM logs WHERE ts < NOW() - ($1 || '' )::interval;

Where $1 is the parsed value of -older-than.

---

🌐 HTTP Middleware

Better Logs includes a fully generic net/http middleware.

Features

• Measures latency
• Captures status code
• Captures request body (optional)
• Captures remote IP (header or TCP)
• Injects context fields
• Logs every request as structured JSON

Add it to any router

handler := httpmw.Middleware(mux)
http.ListenAndServe(":8080", handler)

With config

cfg := httpmw.DefaultConfig()
cfg.LogRequestBody = true
cfg.RemoteIPHeader = "X-Real-IP"

handler := httpmw.WithConfig(cfg)(mux)

---

🗄️ PostgreSQL Log Core (pgcore)

The pgcore package implements a Zap core that writes logs to PostgreSQL.

Highlights

• Batched inserts using COPY for maximum throughput
• Backpressure channel buffer (default 10k entries)
• Timeout-based flush + size-based flush
• Non-blocking Write (logs dropped only if buffer is full)
• Automatic JSON parsing to extract request ID

Usage

pgCfg := pgcore.Config{
    Level:      zap.InfoLevel,
    BatchSize:  1000,
    MaxWait:    2 * time.Second,
    BufferSize: 10_000,
}

pgCore, closePg, err := pgcore.New(db, pgCfg)
if err != nil {
    log.Fatalf("failed to init pg core: %v", err)
}

core := zapcore.NewTee(stdoutCore, pgCore)
logger := zap.New(core)

PostgreSQL table schema

CREATE TABLE IF NOT EXISTS logs (
    id BIGSERIAL PRIMARY KEY,
    ts timestamptz NOT NULL DEFAULT now(),
    req_id text,
    raw jsonb NOT NULL
);

---

🧩 Context Fields

Sometimes you need to attach additional metadata to logs during a request.

Better Logs provides:

• WithFields(ctx, map[string]any)
• AddField(ctx, key, value)
• FieldsFrom(ctx)

The middleware automatically merges these into the final log entry.

Example

func UserHandler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    ctx = httpmw.AddField(ctx, "user_id", "abc123")
    ctx = httpmw.AddField(ctx, "scope", "admin")
    r = r.WithContext(ctx)

    // handler logic...
}

---

🔒 Best Practices

Sensitive Data (PII)

Never log:

• Passwords
• OAuth tokens
• Raw JWTs
• Bank information
• Personal identifiers unless necessary

Use a redaction helper if needed.

Log Retention

Implement a cron or background goroutine that removes logs older than X days:

DELETE FROM logs WHERE ts < NOW() - INTERVAL '7 days';

Batch Size

Keep batch sizes between 500–2000 for optimal throughput.

Avoid Logging Large Bodies

If you enable LogRequestBody, always:

• Set a MaxBodyBytes limit
• Avoid logging file uploads or binary data

---

⚙️ Technical Choices

Zap

Chosen because:

• Fastest widely used structured logger in Go
• Supported by Uber
• Has pluggable cores

PostgreSQL

Chosen because:

• Reliable
• Easy to query
• JSONB support for structured logs

Batching via COPY

This gives 10x+ better performance than individual INSERTs.

net/http Compatibility

The middleware is intentionally written for:

• net/http
• Echo
• Chi
• Httprouter
• Any stdlib-compatible router

Fiber is intentionally excluded because it does not use net/http.

---

🎉 Summary

Better Logs gives you:

• Structured logging
• Batching into PostgreSQL
• Nice HTTP middleware
• Context-based metadata
• Fully self-hosted architecture

Minimal, composable, production-friendly. Ready to drop into any Go project.

Documentation

Index

• func EnsureLogsTable(ctx context.Context, db *sql.DB) error
• func New(cfg Config) (*zap.Logger, func(context.Context) error, error)
• type Config
  • func DefaultConfig() Config

Functions

func EnsureLogsTable

func EnsureLogsTable(ctx context.Context, db *sql.DB) error

EnsureLogsTable creates the logs table (and indexes) if they don't already exist. It is safe to call multiple times.

func New

func New(cfg Config) (*zap.Logger, func(context.Context) error, error)

New initializes global logging based on the provided Config.

It returns:

• a *zap.Logger instance
• a cleanup func(ctx) error to flush buffers and close background workers
• an error if initialization fails

Types

type Config

type Config struct {
	// Identifies the service in logs (e.g. "todos-api", "auth-service").
	ServiceName string

	// Optional DB used for Postgres log sink.
	// If nil and EnablePostgres is true, New will return an error.
	DB *sql.DB

	// Global log level (applies to all cores).
	Level zapcore.Level

	// Whether to log to stdout in JSON (recommended: true).
	Stdout bool

	// Enable Postgres sink. If true, DB must not be nil.
	EnablePostgres bool

	// Postgres sink options.
	PG struct {
		BatchSize  int           // number of log lines per COPY batch.
		MaxWait    time.Duration // max wait before flushing batch.
		BufferSize int           // channel buffer size.
	}
}

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a sane default configuration.