trail

README

Trail

Trail is a lightweight event, flow, and execution journal for Go applications. It makes structured operational events nearly as easy to emit as fmt.Println, while preserving enough identity to reconstruct a business flow, entity history, or execution later in the sink of your choice.

Live demo

Explore a deterministic one-million-event Trail dataset at trail.vestavision.io. The hosted Explorer demonstrates flow timelines, execution relationships, entity history, HTTP events, structured fields, and payload references using synthetic demo data.

trail.Init(trail.Config{
    Service:     "order-worker",
    Environment: "production",
    Sink:        sink,
})
defer trail.Close()

flowID := trail.NewFlow()

trail.Log(
    "order.fulfillment.started",
    trail.Flow(flowID),
    trail.Entity("order", orderID),
)

trail.Log(
    "order.fulfillment.inventory_reserved",
    trail.Flow(flowID),
    trail.Entity("order", orderID),
    trail.String("warehouse", "warehouse-a"),
    trail.Int("score", 97),
)

Trail is deliberately not an OpenTelemetry implementation, APM, Sentry replacement, scheduler, distributed tracing framework, database, or query service. It does not introduce traces or spans into its core model.

Design

The package-level API is intentional. Operational journaling is cross-cutting, and requiring every application function to accept a logger or context.Context would make adoption more expensive than emitting the event. Initialize Trail once, then call trail.Log wherever the business event occurs. Applications can still put a FlowID or ExecutionID in a context when that is convenient; Trail simply does not require it.

Flows are identifiers, not buffers

NewFlow generates only a compact 128-bit value. Trail never keeps a flow ID -> events map and never holds an entire flow in memory. Every call to Log creates an independent event containing its flow_id, and the configured sink publishes bounded batches. A downstream journal can reconstruct the flow by selecting events with that ID.

An entity is a durable business object, such as order/order_123. A flow is one path or attempt involving that object. One order may participate in many fulfillment flows, and one flow may touch several entities.

An execution is a particular job run or manual operation. It can contain many flows. Trail correlates the events produced by an execution but does not schedule it or keep lifecycle state for it:

execution: nightly catalog sync

├── flow A: order fulfillment attempt
│   ├── inventory loaded
│   ├── stock reserved
│   └── fulfillment confirmed
│
└── flow B: order fulfillment attempt
    ├── inventory loaded
    ├── stock unavailable
    └── fulfillment rejected

executionID := trail.NewExecution()
flowID := trail.NewFlow()

trail.Log("catalog.sync.started", trail.Execution(executionID))
trail.Log("order.fulfillment.inventory_loaded",
    trail.Execution(executionID),
    trail.Flow(flowID),
    trail.Entity("order", orderID),
)

Retries and child executions are also explicit, stateless metadata. A retry is a new execution rather than an update to an old one:

retryID := trail.NewExecution()
trail.Log("catalog.sync.started",
    trail.Execution(retryID),
    trail.RetryOfExecution(executionID),
    trail.ExecutionAttempt(2),
    trail.WithExecutionSource(trail.SourceRetry),
)

ParentExecution expresses containment or causal spawning; RetryOfExecution expresses retry history. Scheduling, attempt counters, locks, and next-run calculation remain outside Trail.

IDs

EventID, FlowID, and ExecutionID are distinct [16]byte value types. A process-random 80-bit prefix plus a shared atomic 48-bit counter makes generation concurrency-safe, allocation-free, and independent of wall-clock behavior. The counter is shared by all three ID types, so values do not collide within a process; independent process prefixes make cross-process collision negligible.

The 26-character lowercase Crockford Base32 form is only an external encoding. The IDs do not claim ULID timestamp or ordering semantics and are not security tokens.

Typed fields

The primary path avoids map[string]any, reflection, and interface-valued fields:

trail.String("warehouse", "warehouse-a")
trail.Int("score", 97)
trail.Int64("rows", rows)
trail.Bool("matched", true)
trail.Float64("confidence", 0.97)
trail.Duration("elapsed", elapsed)
trail.Error(err) // fixed key: "error"

Fields retain their type and order. Duplicate keys are allowed; an individual sink may choose its own wire representation and duplicate-key policy.

Writer behavior

Trail starts one asynchronous writer goroutine. Defaults can be overridden individually:

Setting	Default
BufferCapacity	4096 events
BatchSize	64 events
FlushInterval	100 ms
FullPolicy	trail.Drop

Drop keeps the business path non-blocking when the bounded queue is full. Block waits for queue capacity and is useful when the caller explicitly values delivery over latency. Shutdown releases blocked callers.

• Logging before Init, after Close, or while shutdown starts is a no-op and increments dropped.
• A repeated Init returns trail.ErrAlreadyInitialized and leaves the active writer untouched. A successful Close permits later initialization.
• Sink publication failures do not reach Log. The failed batch is discarded, publish_errors and dropped are incremented, and later batches continue.
• Close stops acceptance, drains accepted events, flushes the final batch, and closes the owned sink exactly once. It returns the first publishing error and any sink-close error. A sink must bound its own network calls if shutdown needs a deadline.
• After successful Init, the sink belongs to Trail and must not be reused or closed independently.

trail.Stats() returns inexpensive atomic counters. Written, Dropped, and PublishErrors are cumulative for the process lifetime. Buffered is a live gauge and returns to zero after shutdown completes. Trail does not install a metrics framework.

Sinks

The repository includes NDJSON stdout, concurrency-safe memory, and NATS / JetStream sinks:

import trstdout "github.com/vestavision/trail/sink/stdout"

sink := trstdout.New(os.Stdout)

Custom sinks implement two synchronous methods:

type Sink interface {
    WriteBatch(trail.Batch) error
    Close() error
}

Static service metadata is supplied once per batch, not copied into each event. The batch slices are borrowed for the duration of WriteBatch; a sink that keeps them must copy them. NATS support remains outside the core package.

NATS and JetStream

The NATS sink publishes one bounded, schema-versioned envelope per Trail batch:

import trailnats "github.com/vestavision/trail/sink/nats"

sink, err := trailnats.New(existingConnection, trailnats.Config{
    Mode:    trailnats.JetStream,
    Subject: "trail.events.v1",
})

New borrows the supplied connection; Connect creates a connection owned by the sink. JetStream mode waits for a publish acknowledgement. Plain NATS mode publishes and performs a timeout-bounded flush. The sink has no retry queue and does not add another unbounded buffer. The transport format is documented by the wire package and remains separate from the core API.

HTTP journaling

trailhttp wraps an outgoing transport and emits one provider.http event when the response body reaches EOF or is closed:

client := &http.Client{Transport: trailhttp.Wrap(http.DefaultTransport)}
req = trailhttp.WithFlow(req, flowID)
req = trailhttp.WithExecution(req, executionID)
req = trailhttp.WithEntity(req, "order", orderID)

Correlation attachment is explicit. The adapter uses request context only as a private carrier and does not change Trail's core semantics. Header capture, query inclusion, and bounded body previews are opt-in; common credential and cookie headers are redacted by default. Callers must close response bodies, as required by net/http, for completion to be recorded.

Payloads

The payload package keeps large data outside event messages. It defines an S3-compatible storage contract and structured references containing content type, logical and stored sizes, SHA-256, compression, and retention metadata. The filesystem and S3-compatible implementations are content-addressed. The S3 adapter supports AWS S3, MinIO, and compatible R2 configurations. payload.Ref.Fields(role) attaches only searchable reference metadata to a Trail event; it never embeds the payload itself.

trailhttp.NewPayloadSpooler enables opt-in, bounded request/response capture. Accepted captures are persisted on one bounded background worker, redaction is applied before persistence, and a full spool queue records a payload error while still emitting the HTTP event. Close the spooler before trail.Close so accepted payload jobs can attach their references.

Proving ground and Explorer

Trail includes a storage-neutral ingestor and Explorer API, ClickHouse and PostgreSQL adapters, and a deterministic business-story generator. The separate vestavision/trail-explorer React / TypeScript application is the reference UI. ClickHouse is the recommended reference store for high volume and long retention; PostgreSQL is a supported, simpler choice for moderate installations. Neither is a dependency of the core package.

Clone trail and trail-explorer as sibling directories, then start the reference stack from the trail directory:

docker compose up --build
go run ./cmd/trail-generator --events 100000 --seed 42

Open http://localhost:3000. The generated dataset travels through the same versioned JetStream envelope and durable ingestor used by applications. The Explorer UI talks only to /api/v1; it has no storage-specific code.

The ingestor selects its adapter with TRAIL_STORE=clickhouse or TRAIL_STORE=postgres. For PostgreSQL development, start the postgres Compose profile and point both Go services at TRAIL_POSTGRES_URL. See PROVING_GROUND.md for reproducible validation tiers and the measurement checklist.

Retention

Retention is an optional lifecycle component and is completely separate from trail.Init, trail.Log, and trail.Close. Trail never starts cleanup goroutines, opens retention database connections, or runs a scheduler as part of the logging library.

The one-shot trail-retention command applies explicit ordered policies to ClickHouse or PostgreSQL, archives every eligible batch, and only then deletes live rows and safely garbage-collects unreferenced payloads:

trail-retention run --config retention.json --dry-run
TRAIL_ARCHIVE_FILESYSTEM_ROOT=./trail-archives trail-retention run --config retention.json

There is no destructive default policy and no delete-without-archive mode. A failed archive write, verification, or catalog commit leaves live events untouched. Archives use versioned gzip NDJSON plus a verified manifest and support bounded, idempotent restore. PostgreSQL uses bounded transactional deletes; ClickHouse uses bounded synchronous mutations and reclaims physical space during later merges. Payload GC is eventually consistent and honors payload.Ref.RetainUntil, RetentionClass, shared content-addressed references, and configured limits. See RETENTION.md for policy precedence, configuration, and operational guidance; start from retention.example.json.

Installation and development

go get github.com/vestavision/trail

Trail requires Go 1.24.1 or newer. The core remains standard-library-only; optional NATS, database, and S3 adapters bring their respective client modules.

The ingestor creates the TRAIL_EVENTS JetStream stream and its durable consumer when either is missing. Set TRAIL_MANAGE_STREAM=false only when platform provisioning has already created both resources and the ingestor must not have JetStream management permissions.

Container startup banner

Trail container images print versioned ASCII art immediately before starting the selected binary. The art lives in startup-banner.txt and is therefore part of the released image. Update that file when changing the banner, then publish a new Trail image version.

Set TRAIL_STARTUP_BANNER_ENABLED=false only when a deployment must suppress the banner:

environment:
  TRAIL_STARTUP_BANNER_ENABLED: "false"

The banner must not contain credentials or other sensitive data.

go test ./...
go test -race ./...
go vet ./...
go test -bench=. -benchmem ./...

Benchmarks report measured ns/op, B/op, and allocs/op; the project does not encode machine-specific performance thresholds. See BENCHMARKS.md for the methodology, reference environment, and allocation analysis.

License

Trail is available under the MIT License.

Contributions are welcome. See CONTRIBUTING.md, the security policy, and the code of conduct.

Documentation

Overview

Package trail provides lightweight structured operational event journaling.

Index

• Variables
• func Close() error
• func Init(cfg Config) error
• func Log(kind string, options ...Option)
• type Batch
• type Config
• type Event
• type EventID
  • func ParseEventID(s string) (EventID, error)
  • func (id EventID) IsZero() bool
  • func (id EventID) MarshalText() ([]byte, error)
  • func (id EventID) String() string
  • func (id *EventID) UnmarshalText(text []byte) error
• type ExecutionID
  • func NewExecution() ExecutionID
  • func ParseExecutionID(s string) (ExecutionID, error)
  • func (id ExecutionID) IsZero() bool
  • func (id ExecutionID) MarshalText() ([]byte, error)
  • func (id ExecutionID) String() string
  • func (id *ExecutionID) UnmarshalText(text []byte) error
• type ExecutionSource
• type Field
  • func (f Field) Bool() bool
  • func (f Field) Duration() time.Duration
  • func (f Field) Float64() float64
  • func (f Field) Int64() int64
  • func (f Field) Key() string
  • func (f Field) Kind() FieldKind
  • func (f Field) Text() string
• type FieldKind
• type FlowID
  • func NewFlow() FlowID
  • func ParseFlowID(s string) (FlowID, error)
  • func (id FlowID) IsZero() bool
  • func (id FlowID) MarshalText() ([]byte, error)
  • func (id FlowID) String() string
  • func (id *FlowID) UnmarshalText(text []byte) error
• type FullPolicy
• type Level
  • func (l Level) String() string
• type Metadata
• type Option
  • func Bool(key string, value bool) Option
  • func Duration(key string, value time.Duration) Option
  • func Entity(entityType, entityID string) Option
  • func Error(err error) Option
  • func Execution(id ExecutionID) Option
  • func ExecutionAttempt(attempt uint32) Option
  • func Float64(key string, value float64) Option
  • func Flow(id FlowID) Option
  • func Int(key string, value int) Option
  • func Int64(key string, value int64) Option
  • func Parent(id EventID) Option
  • func ParentExecution(id ExecutionID) Option
  • func RetryOfExecution(id ExecutionID) Option
  • func String(key, value string) Option
  • func WithExecutionSource(source ExecutionSource) Option
  • func WithLevel(level Level) Option
  • func WithScope(scopeType, scopeID string) Option
• type Scope
  • func (s Scope) IsZero() bool
• type Sink
• type Statistics
  • func Stats() Statistics

Variables

var (
	ErrAlreadyInitialized = errors.New("trail: already initialized")
	ErrInvalidConfig      = errors.New("trail: invalid config")
)

var (
	ErrInvalidID = errors.New("trail: invalid ID")
)

Functions

func Close

func Close() error

Close stops the active writer, drains accepted events, and closes its Sink. Calling Close without an active writer is safe and returns nil.

func Init

func Init(cfg Config) error

Init installs the process-global Trail writer. A successful call transfers ownership of cfg.Sink to Trail.

func Log

func Log(kind string, options ...Option)

Log emits an event to the global writer. It is always fire-and-forget.

Types

type Batch

type Batch struct {
	Metadata Metadata
	Events   []Event
}

Batch is a synchronous delivery unit. A Sink must copy it before retaining any Events or Fields after WriteBatch returns.

type Config

type Config struct {
	Service        string
	Environment    string
	Version        string
	Sink           Sink
	BufferCapacity int
	BatchSize      int
	FlushInterval  time.Duration
	FullPolicy     FullPolicy
}

type Event

type Event struct {
	ID                 EventID
	Timestamp          int64
	Kind               string
	Level              Level
	FlowID             FlowID
	ExecutionID        ExecutionID
	ParentExecutionID  ExecutionID
	RetryOfExecutionID ExecutionID
	ExecutionAttempt   uint32
	ExecutionSource    ExecutionSource
	ScopeType          string
	ScopeID            string
	EntityType         string
	EntityID           string
	ParentID           EventID
	Fields             []Field
}

Event is the compact record delivered to a Sink. Timestamp is Unix nanoseconds. Sinks must treat Events and their Fields as immutable.

type EventID

type EventID [16]byte

EventID uniquely identifies an event.

func ParseEventID

func ParseEventID(s string) (EventID, error)

func (EventID) IsZero

func (id EventID) IsZero() bool

func (EventID) MarshalText

func (id EventID) MarshalText() ([]byte, error)

func (EventID) String

func (id EventID) String() string

func (*EventID) UnmarshalText

func (id *EventID) UnmarshalText(text []byte) error

type ExecutionID

type ExecutionID [16]byte

ExecutionID correlates events produced by one job, run, or manual operation.

func NewExecution

func NewExecution() ExecutionID

NewExecution returns a new stateless execution identifier.

func ParseExecutionID

func ParseExecutionID(s string) (ExecutionID, error)

func (ExecutionID) IsZero

func (id ExecutionID) IsZero() bool

func (ExecutionID) MarshalText

func (id ExecutionID) MarshalText() ([]byte, error)

func (ExecutionID) String

func (id ExecutionID) String() string

func (*ExecutionID) UnmarshalText

func (id *ExecutionID) UnmarshalText(text []byte) error

type ExecutionSource

type ExecutionSource string

ExecutionSource describes what initiated an execution. The constants are conventions, not a closed set; applications may use their own stable values.

const (
	SourceCron      ExecutionSource = "cron"
	SourceManual    ExecutionSource = "manual"
	SourceQueue     ExecutionSource = "queue"
	SourceAPI       ExecutionSource = "api"
	SourceRetry     ExecutionSource = "retry"
	SourceWebhook   ExecutionSource = "webhook"
	SourceMigration ExecutionSource = "migration"
)

type Field

type Field struct {
	// contains filtered or unexported fields
}

Field is a compact typed key/value pair. Its value is exposed through typed accessors so the hot path does not require an interface value.

func (Field) Bool

func (f Field) Bool() bool

func (Field) Duration

func (f Field) Duration() time.Duration

func (Field) Float64

func (f Field) Float64() float64

func (Field) Int64

func (f Field) Int64() int64

func (Field) Key

func (f Field) Key() string

func (Field) Kind

func (f Field) Kind() FieldKind

func (Field) Text

func (f Field) Text() string

type FieldKind

type FieldKind uint8

FieldKind identifies the value stored in a Field.

const (
	FieldString FieldKind = iota + 1
	FieldInt
	FieldInt64
	FieldBool
	FieldFloat64
	FieldDuration
	FieldError
)

type FlowID

type FlowID [16]byte

FlowID correlates events belonging to one business flow.

func NewFlow

func NewFlow() FlowID

NewFlow returns a new stateless flow identifier.

func ParseFlowID

func ParseFlowID(s string) (FlowID, error)

func (FlowID) IsZero

func (id FlowID) IsZero() bool

func (FlowID) MarshalText

func (id FlowID) MarshalText() ([]byte, error)

func (FlowID) String

func (id FlowID) String() string

func (*FlowID) UnmarshalText

func (id *FlowID) UnmarshalText(text []byte) error

type FullPolicy

type FullPolicy uint8

FullPolicy controls behavior when the bounded event queue is full.

const (
	Drop FullPolicy = iota
	Block
)

type Level

type Level uint8

Level describes the operational severity of an event.

const (
	LevelDebug Level = iota
	LevelInfo
	LevelWarn
	LevelError
)

func (Level) String

func (l Level) String() string

type Metadata

type Metadata struct {
	Service     string
	Environment string
	Version     string
}

Metadata is configured once and attached to batches rather than copied into every Event.

type Option

type Option struct {
	// contains filtered or unexported fields
}

Option adds correlation or a typed field to an event.

func Bool

func Bool(key string, value bool) Option

func Duration

func Duration(key string, value time.Duration) Option

func Entity

func Entity(entityType, entityID string) Option

func Error

func Error(err error) Option

Error records err under the conventional "error" key.

func Execution

func Execution(id ExecutionID) Option

func ExecutionAttempt

func ExecutionAttempt(attempt uint32) Option

ExecutionAttempt uses one-based attempt numbers. Zero means unspecified.

func Float64

func Float64(key string, value float64) Option

func Flow

func Flow(id FlowID) Option

func Int

func Int(key string, value int) Option

func Int64

func Int64(key string, value int64) Option

func Parent

func Parent(id EventID) Option

func ParentExecution

func ParentExecution(id ExecutionID) Option

func RetryOfExecution

func RetryOfExecution(id ExecutionID) Option

func String

func String(key, value string) Option

func WithExecutionSource

func WithExecutionSource(source ExecutionSource) Option

func WithLevel

func WithLevel(level Level) Option

func WithScope

func WithScope(scopeType, scopeID string) Option

WithScope attaches the event's tenant or security boundary. Scope is not a business entity and must not be used as an authorization decision by Trail.

type Scope

type Scope struct {
	Type string `json:"type"`
	ID   string `json:"id"`
}

Scope identifies the tenant or security boundary that owns an event. Scope is deliberately authorization-neutral: applications decide which authenticated principals may query a given scope.

func (Scope) IsZero

func (s Scope) IsZero() bool

IsZero reports whether no scope is attached.

type Sink

type Sink interface {
	WriteBatch(Batch) error
	Close() error
}

Sink consumes batches synchronously. Ownership transfers to Trail after a successful Init; Trail calls Close exactly once after draining the writer.

type Statistics

type Statistics struct {
	Written       uint64
	Dropped       uint64
	Buffered      uint64
	PublishErrors uint64
}

func Stats

func Stats() Statistics

Stats returns a lock-free snapshot. Monotonic counters cover the process lifetime; Buffered is the current active-writer gauge.