Documentation
¶
Overview ¶
Package dletter implements a persistent dead-letter queue for Go applications.
When a service fails to process an item (e.g. database timeout, external API unavailable), the item would normally be lost. dletter solves this by writing failed items to disk immediately and replaying them later with exponential backoff retries.
Overview ¶
A Logger manages two JSONL log files:
- A retry log (<filename>.log) for items still within the retry budget.
- A permanent-failure log (permanent-<filename>.log) for items that have exhausted all attempts.
Both files are rotated automatically by lumberjack.
Quick Start ¶
type Order struct{ ID string }
func (o Order) AppendLog(buf []byte) []byte {
return append(buf, `{"id":"`+o.ID+`"}`...)
}
// Create a logger
dlq, err := dletter.New("logs/orders.log",
dletter.WithMaxSize(50), // rotate at 50 MB
dletter.WithMaxBackups(10),
dletter.WithCompress(true),
)
if err != nil { ... }
defer dlq.Close()
// Record a failure
dletter.Log(dlq, Order{ID: "ord-42"}, processingErr, attemptNumber)
// Replay later (e.g. in a background goroutine)
err = dlq.Replay(ctx, func(payload []byte) error {
return processOrder(payload)
}, dletter.ReplayOptions{MaxAttempts: 5, InitialWait: time.Second})
Implementing Loggable ¶
Types passed to Log must implement Loggable, which serialises the value into the provided byte slice without heap allocations:
func (o Order) AppendLog(buf []byte) []byte {
buf = append(buf, `{"id":"`...)
buf = append(buf, o.ID...)
buf = append(buf, '"', '}')
return buf
}
Replay Behaviour ¶
Logger.Replay rotates the active log, discovers backup files sorted by modification time, and processes each line with exponential backoff (capped at 30 s + ≤10 % jitter). A state file is written alongside each backup so that a crash mid-replay resumes from the correct line rather than reprocessing everything from the start.
Items that have reached [ReplayOptions.MaxAttempts] are moved to the permanent-failure log instead of being retried.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Log ¶
Log writes a failed item to the retry log as a single JSON line. data can be any value: if it implements Loggable, the zero-allocation AppendLog path is used; otherwise it is serialized with json.Marshal. reason is the error that caused the failure, and attempt is the current retry count (starting at 1). Log is safe for concurrent use.
func LogPermanent ¶ added in v0.2.0
LogPermanent writes a failed item to the permanent-failure log as a single JSON line. data can be any value: if it implements Loggable, the zero-allocation AppendLog path is used; otherwise it is serialized with json.Marshal. reason is the human-readable cause of the permanent failure. Safe for concurrent use.
Types ¶
type Envelope ¶
type Envelope struct {
Timestamp int64 `json:"ts"`
Reason string `json:"reason"`
Attempt int `json:"attempt"`
Payload RawPayload `json:"payload"`
}
Envelope is the JSON structure written to disk for each failed item. It wraps the original payload with metadata (timestamp, attempt count, error reason).
type Loggable ¶
Loggable is an optional interface for zero-allocation payload serialization. If a type passed to Log implements Loggable, its AppendLog method is used instead of json.Marshal. This is only necessary for high-throughput scenarios where GC pressure from json.Marshal is measurable.
type Logger ¶
type Logger struct {
// contains filtered or unexported fields
}
Logger manages two on-disk JSONL log files: a retry log for items that can still be retried, and a permanent-failure log for items that have exhausted their retry budget. Create one with New.
func New ¶
New creates a Logger that writes to filename (retry log) and "permanent-<basename>" (permanent-failure log) in the same directory. The directory is created if it does not exist. Use functional Option values to configure log rotation (size, backups, age, compression).
func (*Logger) Close ¶
Close flushes and closes both the retry and permanent-failure log file handles. Always call this (e.g. via defer) to avoid data loss.
func (*Logger) Replay ¶
Replay reads recorded payloads out securely sequentially from the disk, blocking context and pushing items against handler. Handlers who error again are recorded automatically back to disk log, advancing attempt counts.
func (*Logger) Rotate ¶
Rotate manually triggers rotation of the active retry log. This is called automatically by Logger.Replay before processing backup files.
type Option ¶
type Option func(*lumberjack.Logger)
Option defines a functional configuration option for the lumberjack logger.
func WithCompress ¶
WithCompress determines if the rotated log files should be compressed using gzip.
func WithMaxAge ¶
WithMaxAge sets the maximum number of days to retain old log files based on the timestamp encoded in their filename.
func WithMaxBackups ¶
WithMaxBackups sets the maximum number of old log files to retain.
func WithMaxSize ¶
WithMaxSize sets the maximum size in megabytes of the log file before it gets rotated.
type RawPayload ¶
type RawPayload []byte
RawPayload is a raw JSON byte slice that implements Loggable. It is used internally during replay to re-log payloads that still fail.
func (RawPayload) AppendLog ¶
func (r RawPayload) AppendLog(buf []byte) []byte
func (*RawPayload) UnmarshalJSON ¶
func (r *RawPayload) UnmarshalJSON(data []byte) error
type ReplayOptions ¶
ReplayOptions configures the behavior of the replay mechanism.
Source Files