applog

Documentation

Overview

Package applog governs application log lines on the way into the store.

App logs are the highest-risk surface OpticTrace touches. A payload is structured and can be redacted by JSON path; a log line is free text written by whoever was debugging that day, and it routinely carries bearer tokens, email addresses and whole request bodies inside stack traces. So lines are scrubbed and capped BEFORE they are persisted rather than stored raw and cleaned up later — "later" is after the data is already at rest.

Package applog also collects lines the application already writes.

The alternative — every service POSTing to /api/applogs/ingest — means touching code in the services whose logs you most want, which are usually the ones nobody wants to modify. A JSON logger writing to stdout needs no change at all.

Index

• Constants
• type Collector
  • func NewCollector(gov *Governor, sink Sink, service string, logger *slog.Logger) *Collector
  • func (c *Collector) Close()
  • func (c *Collector) OnDrop(f func(reason string))
  • func (c *Collector) OnKeep(f func(n int))
  • func (c *Collector) OnStored(f func(lines []ext.AppLog))
  • func (c *Collector) Read(ctx context.Context, r io.Reader, src config.AppLogSource, echo io.Writer)
  • func (c *Collector) Start(ctx context.Context)
  • func (c *Collector) Tail(ctx context.Context, src config.AppLogSource)
• type Governor
  • func New(cfg *config.AppLogsCfg) (*Governor, error)
  • func (g *Governor) Admit(l *ext.AppLog) (bool, Reason)
  • func (g *Governor) Enabled() bool
  • func (g *Governor) WithRules(rules []config.Rule) error
• type Reason
• type Sink

Constants

const (
	DefaultMaxLinesPerSpan = 200
	DefaultMaxMessageBytes = 8 << 10 // 8 KiB — enough for a stack trace head

)

Defaults chosen so that turning the feature on without tuning it cannot swamp the store: a debug-level floor would be, and one pathological retry loop would.

Types

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector reads sources, governs each line, and batches to a Sink.

func NewCollector

func NewCollector(gov *Governor, sink Sink, service string, logger *slog.Logger) *Collector

NewCollector builds a collector. service names lines whose source does not override it.

func (*Collector) Close

func (c *Collector) Close()

Close stops collection and drains what is queued.

func (*Collector) OnDrop

func (c *Collector) OnDrop(f func(reason string))

OnDrop and OnKeep install counters, normally the metrics collector's.

func (*Collector) OnKeep

func (c *Collector) OnKeep(f func(n int))

func (*Collector) OnStored

func (c *Collector) OnStored(f func(lines []ext.AppLog))

OnStored installs a fan-out for lines that were persisted — the exporters. Called after a successful save, so an exporter never sees a line the store rejected.

func (*Collector) Read

func (c *Collector) Read(ctx context.Context, r io.Reader, src config.AppLogSource, echo io.Writer)

Read consumes lines from r until EOF or ctx is done. Used for a child process's stdout and stderr.

func (*Collector) Start

func (c *Collector) Start(ctx context.Context)

Start begins flushing. Sources are attached with Tail and Read.

func (*Collector) Tail

func (c *Collector) Tail(ctx context.Context, src config.AppLogSource)

Tail follows a file, including across rotation.

Starts at the END of an existing file: replaying a large log on every restart would flood the store with history nobody asked for, and the retention window would then silently discard the recent lines that were actually wanted.

type Governor

type Governor struct {
	// contains filtered or unexported fields
}

Governor applies the app-log policy. Safe for concurrent use: ingest is an HTTP handler and several requests land at once.

func New

func New(cfg *config.AppLogsCfg) (*Governor, error)

New builds a Governor from config. A nil or disabled config yields a Governor that rejects everything with ReasonDisabled, so callers need no special case.

func (*Governor) Admit

func (g *Governor) Admit(l *ext.AppLog) (bool, Reason)

Admit applies the policy to one line. It returns the governed line and true when it should be stored, or the reason it was dropped.

The line is mutated in place: redaction must happen before anything else can hold a reference to the raw text.

func (*Governor) Enabled

func (g *Governor) Enabled() bool

Enabled reports whether app-log ingestion is turned on.

func (*Governor) WithRules

func (g *Governor) WithRules(rules []config.Rule) error

WithRules compiles the per-rule `logs:` blocks. Call after New.

These can only TIGHTEN what New established. A per-route override able to weaken the global policy would make telemetry.app_logs a suggestion rather than a guarantee, and reviewing one file would stop telling you what is enforced.

type Reason

type Reason string

Reason explains why a line was not stored. Every drop is counted under one of these: data discarded silently is data nobody knows they are missing.

const (
	ReasonOrphan   Reason = "orphan"    // no span id — belongs to no request
	ReasonLevel    Reason = "level"     // below level_min
	ReasonSpanCap  Reason = "span_cap"  // span already at max_lines_per_span
	ReasonDisabled Reason = "disabled"  // feature off
	ReasonEmpty    Reason = "empty"     // nothing left after trimming
	ReasonRuleDrop Reason = "rule_drop" // a rule's logs.drop discarded it
)

type Sink

type Sink interface {
	SaveAppLogs(ctx context.Context, lines []ext.AppLog) error
}

Sink receives governed lines. The store implements it; tests use a fake.