slogsampling

README

Slog sampling policy

A middleware that samples incoming records which caps the CPU and I/O load of logging while attempting to preserve a representative subset of your logs.

Sampling fixes throughput by dropping repetitive log entries.

See also:

• slog-multi: slog.Handler chaining, fanout, routing, failover, load balancing...
• slog-formatter: slog attribute formatting
• slog-sampling: slog sampling policy
• slog-gin: Gin middleware for slog logger
• slog-echo: Echo middleware for slog logger
• slog-fiber: Fiber middleware for slog logger
• slog-datadog: A slog handler for Datadog
• slog-rollbar: A slog handler for Rollbar
• slog-sentry: A slog handler for Sentry
• slog-syslog: A slog handler for Syslog
• slog-logstash: A slog handler for Logstash
• slog-fluentd: A slog handler for Fluentd
• slog-graylog: A slog handler for Graylog
• slog-loki: A slog handler for Loki
• slog-slack: A slog handler for Slack
• slog-telegram: A slog handler for Telegram
• slog-mattermost: A slog handler for Mattermost
• slog-microsoft-teams: A slog handler for Microsoft Teams
• slog-webhook: A slog handler for Webhook
• slog-kafka: A slog handler for Kafka

🚀 Install

go get github.com/samber/slog-sampling

Compatibility: go >= 1.20.3

This library is v0 and follows SemVer strictly. On slog final release (go 1.21), this library will go v1.

No breaking changes will be made to exported APIs before v1.0.0.

💡 Usage

GoDoc: https://pkg.go.dev/github.com/samber/slog-sampling

The sampling middleware can be used standalone or with slog-multi helper.

3 strategies are available:

• Uniform sampling: drop % of logs
• Treshold sampling: drop % of logs after a threshold
• Custom sampler

Uniform sampling

type UniformSamplingOption struct {
	// This will log the first `Threshold` log entries with the same level and message
	// in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0].
	Tick       time.Duration
	Threshold  uint64
	Rate       float64

    // Optional hooks
	OnAccepted func(context.Context, slog.Record)
	OnDropped  func(context.Context, slog.Record)
}

Using slog-multi:

import (
	slogmulti "github.com/samber/slog-multi"
	slogsampling "github.com/samber/slog-sampling"
	"golang.org/x/exp/slog"
)

// Will print 30% of entries.
option := slogsampling.UniformSamplingOption{
	// The sample rate for sampling traces in the range [0.0, 1.0].
    Rate:       0.33,

    OnAccepted: func(context.Context, slog.Record) {
        // ...
    },
    OnDropped: func(context.Context, slog.Record) {
        // ...
    },
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout)),
)

Treshold sampling

type ThresholdSamplingOption struct {
	// This will log the first `Threshold` log entries with the same level and message
	// in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0].
	Tick       time.Duration
	Threshold  uint64
	Rate       float64

    // Optional hooks
	OnAccepted func(context.Context, slog.Record)
	OnDropped  func(context.Context, slog.Record)
}

If Rate is zero, the middleware will drop all log entries after the first Threshold records in that interval.

Using slog-multi:

import (
	slogmulti "github.com/samber/slog-multi"
	slogsampling "github.com/samber/slog-sampling"
	"golang.org/x/exp/slog"
)

// Will print the first 10 entries having the same level+message, then every 10th messages until next interval.
option := slogsampling.ThresholdSamplingOption{
    Tick:       5 * time.Second,
    Threshold:  10,
    Rate:       10,

    OnAccepted: func(context.Context, slog.Record) {
        // ...
    },
    OnDropped: func(context.Context, slog.Record) {
        // ...
    },
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout)),
)

Custom sampler

type CustomSamplingOption struct {
	// The sample rate for sampling traces in the range [0.0, 1.0].
	Sampler func(context.Context, slog.Record) float64

    // Optional hooks
	OnAccepted func(context.Context, slog.Record)
	OnDropped  func(context.Context, slog.Record)
}

Using slog-multi:

import (
	slogmulti "github.com/samber/slog-multi"
	slogsampling "github.com/samber/slog-sampling"
	"golang.org/x/exp/slog"
)

// Will print 100% of log entries during the night, or 50% of errors, 20% of warnings and 1% of lower levels.
option := slogsampling.CustomSamplingOption{
    Sampler: func(ctx context.Context, record slog.Record) float64 {
        if record.Time.Hour() < 6 || record.Time.Hour() > 22 {
            return 1
        }

        switch record.Level {
        case slog.LevelError:
            return 0.5
        case slog.LevelWarn:
            return 0.2
        default:
            return 0.01
        }
    },

    OnAccepted: func(context.Context, slog.Record) {
        // ...
    },
    OnDropped: func(context.Context, slog.Record) {
        // ...
    },
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout)),
)

🤝 Contributing

• Ping me on twitter @samuelberthe (DMs, mentions, whatever :))
• Fork the project
• Fix open issues or request new features

Don't hesitate ;)

# Install some dev dependencies
make tools

# Run tests
make test
# or
make watch-test

👤 Contributors

💫 Show your support

Give a ⭐️ if this project helped you!

📝 License

Copyright © 2023 Samuel Berthe.

This project is MIT licensed.

Documentation

Index

• type CustomSamplingOption
  • func (o CustomSamplingOption) NewMiddleware() slogmulti.Middleware
• type ThresholdSamplingOption
  • func (o ThresholdSamplingOption) NewMiddleware() slogmulti.Middleware
• type UniformSamplingOption
  • func (o UniformSamplingOption) NewMiddleware() slogmulti.Middleware

Types

type CustomSamplingOption

type CustomSamplingOption struct {
	// The sample rate for sampling traces in the range [0.0, 1.0].
	Sampler func(context.Context, slog.Record) float64

	// Optional hooks
	OnAccepted func(context.Context, slog.Record)
	OnDropped  func(context.Context, slog.Record)
}

func (CustomSamplingOption) NewMiddleware

func (o CustomSamplingOption) NewMiddleware() slogmulti.Middleware

NewMiddleware returns a slog-multi middleware.

type ThresholdSamplingOption

type ThresholdSamplingOption struct {
	// This will log the first `Threshold` log entries with the same level and message
	// in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0].
	Tick      time.Duration
	Threshold uint64
	Rate      float64

	// Optional hooks
	OnAccepted func(context.Context, slog.Record)
	OnDropped  func(context.Context, slog.Record)
}

func (ThresholdSamplingOption) NewMiddleware

func (o ThresholdSamplingOption) NewMiddleware() slogmulti.Middleware

NewMiddleware returns a slog-multi middleware.

type UniformSamplingOption

type UniformSamplingOption struct {
	// The sample rate for sampling traces in the range [0.0, 1.0].
	Rate float64

	// Optional hooks
	OnAccepted func(context.Context, slog.Record)
	OnDropped  func(context.Context, slog.Record)
}

func (UniformSamplingOption) NewMiddleware

func (o UniformSamplingOption) NewMiddleware() slogmulti.Middleware

NewMiddleware returns a slog-multi middleware.