ext

package
v0.15.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package ext is OpticTrace's extension surface: the contract another Go module implements to add a payload store or an output exporter.

Why this package exists

Almost all of OpticTrace lives under internal/, which Go forbids other modules from importing. That is deliberate — it keeps the implementation free to change. But it also means an out-of-tree driver had nowhere to stand: it could not even name the types an interface method returns.

ext is the narrow, deliberate exception. It holds the data types and the two plugin interfaces, and nothing else. The types here are the canonical definitions; internal/store and internal/export alias them, so there is one Record in the program, not two that happen to look alike.

Stability

Everything in this package is a public API contract and follows semantic versioning: no breaking change within a major version. Nothing under internal/ carries that promise. If you are writing an extension and find yourself wanting something from internal/, open an issue rather than vendoring it — the gap is the bug.

What an extension can add 

RegisterStore     a payload store (telemetry.store.driver: <name>)
RegisterExporter  an output plugin (telemetry.exporters[].type: <name>)

Registration happens at init time or from main before the agent starts. Both registries are keyed by the name that appears in optic.yaml, so a driver becomes configurable simply by being linked into the binary: 

package main

import (
	"github.com/dwarka-prasad/optictrace"
	"github.com/dwarka-prasad/optictrace/ext"
)

func init() {
	ext.RegisterStore("s3", func(dsn string, _ ext.Settings) (ext.Store, error) {
		return newS3Store(dsn)
	})
}

The governance contract extensions inherit

Records handed to a Store or an Exporter have ALREADY been through the rule engine: restricted fields are absent, redacted fields hold the placeholder. That is the whole design — governance sits upstream of every sink, so no extension can see raw sensitive data, and no extension has to be trusted with it.

Two obligations follow, and an extension that breaks either is a governance hole even though the core did its job:

  • Do not reconstruct what governance removed. Joining a redacted record against another source to recover the original value defeats the point.
  • Purge must actually delete. It backs erasure requests ("delete everything you hold for tenant X"), so returning before the data is gone is worse than not implementing it.

Verifying a Store

The conformance suite that the built-in drivers run is exported from ext/exttest. A new Store should pass it unmodified — that suite is what stops two drivers quietly answering the same question differently.

Index

Constants

View Source 
const DefaultAnalysisMaxRows = 20_000

DefaultAnalysisMaxRows bounds how many records an analysis pass reads when no explicit limit is given. These are FULL records including bodies, so at the default 64 KiB capture limit this is a memory ceiling, not just a row count. Override with telemetry.store.analysis_max_rows.

View Source 
const MaxAnalysisMaxRows = 500_000

MaxAnalysisMaxRows is the hard ceiling on that knob.

Variables

View Source 
var (
	// ErrNoCredentials means "this request carries nothing I recognise" —
	// the chain moves on to the next authenticator. Return this rather than a
	// real error for an absent cookie or header, or you will lock out every
	// other authentication method.
	ErrNoCredentials = errors.New("ext: no credentials for this authenticator")

	// ErrResponseWritten means the authenticator has already written the
	// response — an OIDC redirect, a device-code page — and the core must
	// stop. Nothing further is written to the ResponseWriter.
	ErrResponseWritten = errors.New("ext: authenticator wrote the response")
)
View Source 
var ErrForbidden = errors.New("ext: forbidden")

ErrForbidden is the conventional denial. Any non-nil error denies; this one exists so the common case reads clearly.

Functions

func AnalysisLimit 

func AnalysisLimit(limit int) int

AnalysisLimit resolves a requested limit against the defaults.

func LevelRank added in v0.9.0 

func LevelRank(level string) int

LevelRank exposes the severity ordering so drivers and filters agree on what "at least warn" means.

func NormaliseLevel added in v0.9.0 

func NormaliseLevel(level string) string

NormaliseLevel maps a logger's spelling onto the canonical set. It returns an unrecognised level unchanged rather than forcing it into a bucket.

func NoteAccess 

func NoteAccess(ctx context.Context, a Accessed)

NoteAccess records what a handler actually reached, for the audit trail. Safe to call more than once — counts accumulate, which is what a paged export needs. A no-op when nothing is auditing.

func RegisterAdminRoutes 

func RegisterAdminRoutes(routes ...AdminRoute)

RegisterAdminRoutes adds handlers to the admin server. Call before the server is built — from init or from main.

func RegisterAuditor 

func RegisterAuditor(a Auditor)

RegisterAuditor adds an audit sink. Every registered auditor sees every decision.

func RegisterAuthenticator 

func RegisterAuthenticator(a Authenticator)

RegisterAuthenticator adds an authentication method. Registered authenticators are tried in registration order, BEFORE the built-in bearer token, so an extension can take precedence over it.

func RegisterAuthorizer 

func RegisterAuthorizer(a Authorizer)

RegisterAuthorizer adds an authorization policy. ALL registered authorizers must allow, so adding one can only narrow access, never widen it.

func RegisterExporter 

func RegisterExporter(name string, build ExporterBuilder)

RegisterExporter makes a plugin available as `telemetry.exporters[].type: <name>`.

Panics on a duplicate name, for the same reason RegisterStore does.

func RegisterStore 

func RegisterStore(name string, open StoreOpener)

RegisterStore makes a driver available as `telemetry.store.driver: <name>`.

Call it from an init function or from main before starting the agent. Registering is what makes the name pass config validation, so a driver becomes configurable purely by being linked into the binary.

Panics on a duplicate or reserved name: a silently ignored registration would show up much later as "unknown driver", pointing at the config rather than at the collision.

func RegisteredExporters 

func RegisteredExporters() []string

RegisteredExporters lists registered exporter types, sorted.

func RegisteredStores 

func RegisteredStores() []string

RegisteredStores lists registered driver names, sorted — used to build the "not supported (…)" message when a config names an unknown driver.

func ResetRegistriesForTest 

func ResetRegistriesForTest()

ResetRegistriesForTest clears every extension registry.

Registration is process-wide by design — extensions register from init — so tests that register need a way back to a clean state. Exported because the core's own tests live in another package, and because an extension's tests need it too. Not for use outside tests.

func WithAccessRecorder 

func WithAccessRecorder(ctx context.Context) (context.Context, func() Accessed)

WithAccessRecorder prepares a context to collect NoteAccess calls, returning the context and a getter for what accumulated. Called by the core.

func WithIdentity 

func WithIdentity(ctx context.Context, id *Identity) context.Context

WithIdentity attaches an identity to a context. The core calls this; an extension normally only needs IdentityFrom.

Types

type Accessed 

type Accessed struct {
	// Count is how many stored records the response covered.
	Count int
	// RecordIDs identifies specific records when the set is small enough to
	// be worth naming — a single-record fetch, not a 20,000-row export.
	RecordIDs []int64
	// Filter is a human-readable summary of the query used.
	Filter string
	// Consumer is the tenant/consumer label value when the request was scoped
	// to one, so "who looked at THIS customer's data" is answerable.
	Consumer string
}

Accessed describes WHAT a request touched. This is the difference between an audit trail that answers an auditor's question and one that does not: "alice listed logs" is close to useless, "alice exported 12,043 records filtered to tenant=acme" is the answer.

type Action 

type Action struct {
	Capability Capability
	Method     string
	Path       string
}

Action is one authorization decision point.

type AdminRoute 

type AdminRoute struct {
	// Pattern is an http.ServeMux pattern, e.g. "GET /auth/callback".
	// It must not collide with a core route; registration panics if it does.
	Pattern string
	Handler http.Handler
	// Capability gates the route like any core route. A login callback needs
	// CapPublic, since it runs before the caller has a session.
	Capability Capability
}

AdminRoute is a handler an extension adds to the admin server — an OIDC callback, a role-management API, an audit-log viewer.

func AdminRoutes 

func AdminRoutes() []AdminRoute

type AppLog added in v0.9.0 

type AppLog struct {
	ID   int64     `json:"id"`
	Time time.Time `json:"time"`
	// Service is the application that emitted the line, which need not be the
	// service that recorded the span — a downstream hop logs under the same
	// trace.
	Service string `json:"service"`
	// TraceID and SpanID tie the line to one hop of one request. A line
	// without a SpanID belongs to no request; what happens to it is
	// telemetry.app_logs.drop_orphans.
	TraceID string `json:"trace_id"`
	SpanID  string `json:"span_id"`
	// Level is normalised lowercase: debug, info, warn, error, fatal.
	// Anything unrecognised is kept verbatim and sorts above fatal, so a
	// custom level is never silently dropped by a level filter.
	Level   string `json:"level,omitempty"`
	Message string `json:"message"`
	// Fields carries structured logger key/values, stringified. Values are
	// redacted under the same policy as the message — a token pasted into a
	// field is exactly as leaked as one in the message text.
	Fields map[string]string `json:"fields,omitempty"`
	// Route is the rule pattern the request matched, when the producer knows
	// it. It lets a per-rule `logs:` block apply to this line.
	//
	// Client-supplied and therefore untrusted — which is safe here only
	// because a per-rule log policy can exclusively TIGHTEN the global one. A
	// producer that lies about its route, or omits it, gets the global policy:
	// the floor, never less. If per-rule blocks could ever loosen, this field
	// would have to be verified instead of trusted.
	Route string `json:"route,omitempty"`
	// Source names the producer: an SDK name, or "ingest" for a direct POST.
	Source string `json:"source,omitempty"`
	// Truncated reports that Message was cut to the configured byte cap.
	Truncated bool `json:"truncated,omitempty"`
}

AppLog is one line your application logged while serving a request.

The name is deliberately not "log": in this codebase a "log" is an HTTP exchange (Record, the `logs` table, Store). An AppLog is the other thing — what the application itself wrote to its logger during that exchange.

Correlation is by SpanID, never by timing. The proxy already hands the application a traceparent carrying this hop's span, so a line the app logs under that context belongs to that span as a fact, not as an inference. Guessing from timestamps would file one tenant's line under another tenant's request whenever two are served concurrently.

type AppLogExporter added in v0.11.0 

type AppLogExporter interface {
	Exporter
	// ExportAppLogs delivers one batch of governed lines. Same contract as
	// Export: an error marks the whole batch failed, and ctx is cancelled on
	// shutdown.
	ExportAppLogs(ctx context.Context, batch []*AppLog) error
}

AppLogExporter is an OPTIONAL companion to Exporter: an exporter may also accept application log lines.

Separate interface rather than a second method on Exporter, for the same reason ext.AppLogStore is separate from ext.Store — Exporter is published and implemented outside this module, so adding a method to it would break every third-party exporter at compile time. Detect support with a type assertion: 

if ale, ok := myExporter.(ext.AppLogExporter); ok { ... }

An exporter without it still receives records; log lines simply do not fan out to it. That is reported at startup rather than left to be discovered, because an audit trail quietly missing the highest-risk surface is worse than one that says it is records-only.

type AppLogFilter added in v0.9.0 

type AppLogFilter struct {
	TraceID string
	SpanID  string
	Service string
	// LevelMin drops anything less severe. Empty means no level filter.
	LevelMin string
	Since    time.Time
	Until    time.Time
	// Search is a substring match over the message.
	Search string
	Limit  int
	Offset int
}

AppLogFilter selects stored lines. The zero value matches everything within the store's own limits.

type AppLogStore added in v0.9.0 

type AppLogStore interface {
	// SaveAppLogs persists a batch. Called from the async writer, never from
	// the request hot path. Implementations must be safe for concurrent use.
	SaveAppLogs(ctx context.Context, lines []AppLog) error
	// QueryAppLogs returns matching lines oldest-first — reading a request's
	// logs means reading them in the order they happened — plus the total
	// matching count, ignoring Limit/Offset.
	QueryAppLogs(ctx context.Context, f AppLogFilter) (lines []AppLog, total int64, err error)
	// CountAppLogs returns the total stored.
	CountAppLogs(ctx context.Context) (int64, error)
	// AppLogStats aggregates the lines stored since a time.
	AppLogStats(ctx context.Context, since time.Time) (*AppLogSummary, error)
	// PruneAppLogsBefore enforces age-based retention. App logs run orders of
	// magnitude above request volume, so they usually want a shorter horizon
	// than records do.
	PruneAppLogsBefore(ctx context.Context, cutoff time.Time) (removed int64, err error)
}

AppLogStore is an OPTIONAL companion to Store: a driver may implement it to persist application log lines, and is a perfectly good driver if it does not. It is deliberately a separate interface rather than more methods on Store, because Store is published and implemented outside this module — adding a method to it would break every third-party driver at compile time, and drivers here have twice been broken by far smaller changes.

Detect support with a type assertion: 

if als, ok := myStore.(ext.AppLogStore); ok { ... }

CONTRACT — if you implement both Store and AppLogStore, Store.Purge MUST also delete the app logs belonging to the records it purges. Erasure that removes a tenant's requests but leaves the log lines those requests wrote is not erasure, and app logs are the likelier place for the personal data to be sitting. ext/exttest asserts this.

type AppLogSummary added in v0.9.0 

type AppLogSummary struct {
	Total int64 `json:"total"`
	// ByLevel and ByService are counts keyed by level and by emitting service.
	ByLevel   map[string]int64 `json:"by_level"`
	ByService map[string]int64 `json:"by_service"`
	// SpansWithLogs is how many distinct requests contributed lines — the
	// denominator for "how much of my traffic is actually explainable".
	SpansWithLogs int64 `json:"spans_with_logs"`
}

AppLogSummary aggregates stored lines for a window. Computed in the store rather than by counting a page in the browser: a dashboard that summarises the first 200 rows and calls it a total is quietly lying at exactly the volumes where the summary starts to matter.

type AuditEvent 

type AuditEvent struct {
	Time     time.Time
	Identity *Identity // nil when unauthenticated
	Action   Action
	Outcome  Outcome
	// Reason carries the denial cause. Never sent to the caller.
	Reason   string
	Status   int    // HTTP status actually written
	Remote   string // client address
	Accessed Accessed
}

AuditEvent is one access-control decision plus what it reached.

type Auditor 

type Auditor interface {
	Name() string
	Record(ctx context.Context, e AuditEvent)
}

Auditor receives every access-control decision on the admin surface.

Record must not block: it runs in the request path. It returns no error on purpose. The core will not fail a request because an audit backend is unavailable — that would make the audit system an availability dependency of the dashboard, precisely when an incident is underway and someone needs it.

If your compliance posture genuinely requires "no audit, no access", put that check in an Authorizer, not here: auditing a read happens after the read, so refusing at this point would record the access and deny the response, which is the worst of both.

func Auditors 

func Auditors() []Auditor

type Authenticator 

type Authenticator interface {
	Name() string
	// Authenticate returns the caller's identity, ErrNoCredentials to defer
	// to the next authenticator, ErrResponseWritten if it has responded, or
	// any other error to reject the request outright.
	//
	// The ResponseWriter is provided for flows that must respond (a redirect)
	// and for setting a session cookie on success. Do not write to it in the
	// ErrNoCredentials case.
	Authenticate(w http.ResponseWriter, r *http.Request) (*Identity, error)
}

Authenticator resolves the caller's identity.

Implementations must be safe for concurrent use and must not block: this runs on every admin request.

func Authenticators 

func Authenticators() []Authenticator

Authenticators, Authorizers, Auditors and AdminRoutes return the registered extensions. The core calls these when building the admin server.

type Authorizer 

type Authorizer interface {
	Name() string
	Authorize(ctx context.Context, id *Identity, a Action) error
}

Authorizer decides whether an identity may perform an action.

Every registered Authorizer must allow, so composing two never widens access. Returning an error denies; the message is logged and audited but never returned to the caller, since a denial reason is itself information.

func Authorizers 

func Authorizers() []Authorizer

type Capability 

type Capability string

Capability classifies a request by what it exposes, not by its URL.

The split matters: the CORE owns the route→capability mapping, because only the core knows which handlers return captured payloads. An extension writes policy against capabilities. That means an RBAC plugin never has to track OpticTrace's URL structure, and — more importantly — a route added later cannot silently escape a policy that was written against URLs. 

const (
	// CapPublic is reachable without authentication: /healthz, and any route
	// an extension registers for a login callback.
	CapPublic Capability = "public"
	// CapMetrics is the Prometheus endpoint. Separate because scrapers
	// authenticate differently from people, and it exposes no payloads.
	CapMetrics Capability = "metrics"
	// CapReadStats covers aggregates only — counts, latencies, route and
	// service summaries, usage totals. No captured payloads.
	CapReadStats Capability = "read:stats"
	// CapReadPayload returns captured request/response bodies to the caller.
	// This is the one that matters: it is the capability that lets someone
	// read customer data.
	CapReadPayload Capability = "read:payload"
	// CapExport is bulk payload egress — the whole store, streamed to a file.
	// Separated from CapReadPayload deliberately: "can look at one request
	// while debugging" and "can download everything" are different grants.
	CapExport Capability = "export"
	// CapAnalyse reads payloads server-side but returns only derived output
	// (leak findings with masked samples, an inferred spec). Lower risk than
	// CapReadPayload, higher than CapReadStats.
	CapAnalyse Capability = "analyse"
	// CapReadConfig returns the governance policy.
	CapReadConfig Capability = "read:config"
	// CapIngest accepts telemetry from SDKs — a machine-to-machine write.
	CapIngest Capability = "ingest"
	// CapAdmin changes agent state: reload.
	CapAdmin Capability = "admin"
	// CapUI serves the dashboard's static assets.
	CapUI Capability = "ui"
)

func Capabilities 

func Capabilities() []Capability

Capabilities lists every capability the core defines, sorted — useful for an extension building a role editor, and for asserting a policy covers them all.

type Challenger 

type Challenger interface {
	// Challenge reports whether it handled the response.
	Challenge(w http.ResponseWriter, r *http.Request) bool
}

Challenger is an optional Authenticator extension. When no authenticator could identify the caller, each Challenger is offered the request before the core gives up with 401 — this is where an interactive login redirects to its identity provider.

Kept separate from Authenticate so that a browser with no session still reaches the redirect even though a token authenticator ran first and simply found no bearer header.

type Exporter 

type Exporter interface {
	// Name is the configured exporter name; it becomes the `exporter` label
	// on optictrace_exported_total and friends.
	Name() string
	// Type is the registered kind, e.g. "file" or "s3".
	Type() string
	// Export delivers one batch. Returning an error marks the WHOLE batch
	// failed — the core counts it and moves on rather than retrying forever,
	// because blocking here would eventually apply backpressure to the
	// request path.
	//
	// Respect ctx: it is cancelled on shutdown.
	Export(ctx context.Context, batch []*Record) error
	Close() error
}

Exporter delivers batches of governed records to one destination.

internal/export.Exporter is an alias of this type. The batching, retry and backpressure machinery stays in the core: an Exporter only has to deliver a batch and report whether it worked.

type ExporterBuilder 

type ExporterBuilder func(opts ExporterOptions) (Exporter, error)

ExporterBuilder constructs an Exporter from its configuration.

func LookupExporter 

func LookupExporter(name string) (ExporterBuilder, bool)

LookupExporter returns the builder registered for name.

type ExporterOptions 

type ExporterOptions struct {
	Name string
	Type string
	// Settings holds `telemetry.exporters[].settings` — the keys the core
	// knows nothing about. optic.yaml rejects unknown top-level keys, so this
	// map is where an extension's own configuration lives.
	Settings Settings

	BatchSize     int
	FlushInterval time.Duration
	QueueSize     int
	// ServiceName is service.name, for exporters that tag their output.
	ServiceName string
}

ExporterOptions is one entry from `telemetry.exporters` as an extension sees it. The core owns batching, so BatchSize/FlushInterval/QueueSize are informational — useful for sizing an internal buffer, not something the exporter must implement.

type Filter 

type Filter struct {
	Method     string
	PathPrefix string
	Search     string // substring across path + bodies
	StatusMin  int
	StatusMax  int
	Since      time.Time
	Until      time.Time
	// TraceID selects every hop of one request — the question "what did this
	// call actually do" once several services report into one store.
	TraceID string
	// Labels selects records carrying ALL of these label values exactly —
	// the multi-tenant question, "show me only this tenant's calls".
	//
	// Matched literally, never as a pattern. A tenant named "acme_1" must not
	// select "acmeX1": the same mistake that once made `purge` destroy a
	// neighbour's data, and just as wrong when it silently widens what someone
	// is shown.
	Labels map[string]string
	Limit  int
	Offset int
}

Filter narrows a log query. Zero values mean "no constraint".

type Identity 

type Identity struct {
	// Subject is the stable unique id — an OIDC `sub`, a service-account
	// name, a token id. Prefer something that survives a rename: an audit
	// trail keyed on an email address becomes ambiguous the day someone
	// changes theirs.
	Subject string
	Name    string
	Email   string
	// Groups carry the caller's roles, for an Authorizer to key on.
	Groups []string
	// Method names the authenticator that produced this, e.g. "token" or
	// "oidc". Recorded in the audit trail: "who" is incomplete without "how".
	Method string
	// Attrs is free-form, for claims an extension wants to keep.
	Attrs map[string]string
}

Identity is who made a request. Built by an Authenticator, carried on the request context, and recorded in every audit event.

func IdentityFrom 

func IdentityFrom(ctx context.Context) *Identity

IdentityFrom returns the authenticated caller, or nil when the request was not authenticated (which is the normal state with auth disabled).

func (*Identity) InGroup 

func (i *Identity) InGroup(name string) bool

InGroup reports whether the identity belongs to a group.

type Outcome 

type Outcome string

Outcome is how a request ended, from the access-control point of view. 

const (
	OutcomeAllowed      Outcome = "allowed"
	OutcomeDenied       Outcome = "denied"       // authorization refused
	OutcomeUnauthorized Outcome = "unauthorized" // no usable credentials
)

type Record 

type Record struct {
	ID int64 `json:"id"`
	// Time is when the exchange COMPLETED — the record is written once the
	// response is known, and every implementation stamps it there.
	//
	// Anything that needs the start must subtract DurationMS. A trace
	// waterfall built on Time directly draws the parent starting after the
	// children it called, because the parent is the last hop to finish.
	// Stamping the start instead would have been the friendlier choice, but
	// changing it now would make old and new rows indistinguishable and
	// silently corrupt every timeline that spans the change.
	Time    time.Time `json:"time"`
	Service string    `json:"service"`
	Method  string    `json:"method"`
	Path    string    `json:"path"`
	// Query is the sanitized query string (policy-masked, stable ordering).
	Query  string `json:"query,omitempty"`
	Route  string `json:"route"` // low-cardinality route pattern
	Status int    `json:"status"`
	// DurationMS is milliseconds (float) — the natural unit for SDKs in
	// every language to produce and for the dashboard to consume.
	DurationMS float64 `json:"duration_ms"`
	Remote     string  `json:"remote"`
	Source     string  `json:"source"` // "proxy" or an SDK name

	// TraceID ties every hop of one request together across services, so a
	// flat log becomes a tree. Taken from the inbound W3C traceparent when
	// the caller sent one, generated when it did not.
	TraceID string `json:"trace_id,omitempty"`
	// SpanID identifies this hop; ParentSpanID is the caller's span, empty at
	// the root. Together these are what makes the tree reconstructible rather
	// than just a filtered list.
	SpanID       string `json:"span_id,omitempty"`
	ParentSpanID string `json:"parent_span_id,omitempty"`
	// Stream marks a long-lived streaming response (SSE or chunked). Its
	// DurationMS is a connection lifetime, not a latency, so percentile
	// aggregations exclude it — one 10-minute stream would otherwise define
	// a route's p95 for the whole window.
	Stream bool `json:"stream,omitempty"`

	RequestHeaders  map[string]string `json:"request_headers,omitempty"`
	ResponseHeaders map[string]string `json:"response_headers,omitempty"`
	RequestBody     string            `json:"request_body,omitempty"`
	ResponseBody    string            `json:"response_body,omitempty"`
	ReqTruncated    bool              `json:"req_truncated,omitempty"`
	RespTruncated   bool              `json:"resp_truncated,omitempty"`

	ReqBytes     int64             `json:"req_bytes"`
	RespBytes    int64             `json:"resp_bytes"`
	Labels       map[string]string `json:"labels,omitempty"`
	MatchedRules []string          `json:"matched_rules,omitempty"`
	// Meters holds numeric usage extracted from the response by rule-level
	// meter paths (e.g. LLM token counts) — the raw material for billing.
	Meters map[string]float64 `json:"meters,omitempty"`
}

Record is one captured HTTP exchange, post-governance.

type RouteDetail 

type RouteDetail struct {
	RouteStat
	P50Latency float64 `json:"p50_latency_ms"`
	P99Latency float64 `json:"p99_latency_ms"`
	ReqBytes   int64   `json:"req_bytes"`
	RespBytes  int64   `json:"resp_bytes"`
}

RouteDetail extends RouteStat with the percentiles the Routes dashboard shows for every route (not just the top 10).

type RouteStat 

type RouteStat struct {
	Route      string  `json:"route"`
	Method     string  `json:"method"`
	Count      int64   `json:"count"`
	Errors     int64   `json:"errors"`
	AvgLatency float64 `json:"avg_latency_ms"`
	P95Latency float64 `json:"p95_latency_ms"`
}

RouteStat aggregates one route's traffic.

type RuleMatch 

type RuleMatch struct {
	Rule  string `json:"rule"`
	Count int64  `json:"count"`
}

RuleMatch is how often one governance rule fired in a window.

type ServiceStat 

type ServiceStat struct {
	Service    string    `json:"service"`
	Requests   int64     `json:"requests"`
	Errors     int64     `json:"errors"`
	ErrorRate  float64   `json:"error_rate"`
	AvgLatency float64   `json:"avg_latency_ms"`
	P95Latency float64   `json:"p95_latency_ms"`
	Routes     int64     `json:"routes"`
	Sources    string    `json:"sources"` // e.g. "proxy, express"
	LastSeen   time.Time `json:"last_seen"`
}

ServiceStat summarizes one service in a multi-service deployment. One agent proxies one service, but many SDKs can report into a single agent, so the store may hold several.

type Settings 

type Settings map[string]any

Settings carries driver-specific configuration from optic.yaml's `telemetry.store.settings` map — the keys the core knows nothing about. Named keys (driver, dsn, retention) are validated by the core and passed separately; everything here is the extension's own business.

func (Settings) Bool 

func (s Settings) Bool(key string, def bool) bool

Bool reads a boolean setting.

func (Settings) Int 

func (s Settings) Int(key string, def int) int

Int reads an integer setting. YAML decodes whole numbers as int, but a value that arrived through JSON is a float64, so both are accepted.

func (Settings) String 

func (s Settings) String(key, def string) string

String reads a string setting, returning def when absent or of another type.

type Span added in v0.15.0 

type Span struct {
	ID int64 `json:"id"`
	// Start is when the operation BEGAN.
	//
	// Deliberately named differently from [Record.Time], which is when an
	// exchange COMPLETED. The two cannot be reconciled — Record.Time cannot
	// change without making old and new rows indistinguishable — so the field
	// names differ to stop anyone assuming they mean the same thing.
	Start time.Time `json:"start"`
	// Service is the application that ran the operation.
	Service string `json:"service"`
	// TraceID and SpanID place this operation in a trace. SpanID is this
	// operation's own id; ParentSpanID is the span it happened inside, which
	// is normally the HTTP span the SDK recorded, or another internal span
	// when operations nest.
	TraceID      string `json:"trace_id"`
	SpanID       string `json:"span_id"`
	ParentSpanID string `json:"parent_span_id"`
	// Name is what ran: "db.query", "redis.get", "GET /rates", "score.model".
	// Kept low-cardinality by convention — the varying part belongs in an
	// attribute, not the name, or every chart built on this becomes useless.
	Name string `json:"name"`
	// Kind classifies the operation so a waterfall can colour it and a
	// breakdown can group by it: db, cache, http, queue, rpc, internal.
	// Anything unrecognised is kept verbatim rather than coerced.
	Kind string `json:"kind,omitempty"`
	// DurationMS is how long it took.
	DurationMS float64 `json:"duration_ms"`
	// Error is non-empty when the operation failed, and is governed like any
	// other free text: a driver error routinely quotes the statement that
	// failed, parameters included.
	Error string `json:"error,omitempty"`
	// Attrs describe the operation. Conventional keys, so the dashboard and a
	// breakdown can rely on them:
	//
	//	db.system     postgres · mysql · sqlite · mongodb
	//	db.statement  the statement — pass the TEMPLATE, not the interpolated one
	//	db.rows       rows returned or affected
	//	cache.key     the key, or better its shape
	//	cache.hit     true · false
	//	http.method   the outbound method
	//	http.url      the outbound URL
	//	http.status   the outbound status
	//	queue.topic   the destination
	Attrs map[string]string `json:"attrs,omitempty"`
	// Route is the rule pattern the enclosing request matched, when the
	// producer knows it. It lets a per-rule `spans:` block apply.
	//
	// Client-supplied and therefore untrusted — safe only because a per-rule
	// block can exclusively TIGHTEN the global policy. A producer that lies
	// about its route, or omits it, gets the global floor and never less.
	Route string `json:"route,omitempty"`
	// Source names the producer: an SDK name, or "ingest" for a direct POST.
	Source string `json:"source,omitempty"`
	// Truncated reports that an attribute or the error was cut to the
	// configured byte cap.
	Truncated bool `json:"truncated,omitempty"`
}

Span is one operation inside a request: a database query, a cache lookup, an outbound call, a stretch of computation worth naming.

A Record covers a whole HTTP exchange as OpticTrace saw it from outside. A Span is what happened while that exchange was being served, which is the difference between "this request took 300ms" and "this request took 300ms, 280 of them in one query".

Correlation is by ParentSpanID, never by timing, for the same reason AppLog correlates that way: under concurrent traffic, matching on timestamps files one tenant's work inside another tenant's request.

ATTRIBUTES ARE THE RISK HERE. A statement reads `SELECT * FROM users WHERE email = 'a@b.com'`, a cache key embeds an account id, a URL carries a token in its query string. Attributes are therefore governed — redacted, capped and filtered — before they are stored, exactly like an app log line, and for the same reason: "clean it up later" is after the data is already at rest.

func (Span) End added in v0.15.0 

func (s Span) End() time.Time

End returns when the operation finished.

type SpanBreakdown added in v0.15.0 

type SpanBreakdown struct {
	Name string `json:"name"`
	Kind string `json:"kind,omitempty"`
	// Count is how many times it ran across the window, which for a query
	// inside a loop is the number worth seeing.
	Count int64 `json:"count"`
	// Requests is how many distinct requests ran it, so Count/Requests is the
	// per-request multiplier — the shape an N+1 query makes.
	Requests int64   `json:"requests"`
	Errors   int64   `json:"errors"`
	TotalMS  float64 `json:"total_ms"`
	AvgMS    float64 `json:"avg_ms"`
	P95MS    float64 `json:"p95_ms"`
	MaxMS    float64 `json:"max_ms"`
}

SpanBreakdown aggregates where a route's time actually goes.

Computed in the store rather than by summing a page in the browser: a summary of the first 200 rows is wrong at exactly the volumes where a summary starts to matter.

type SpanFilter added in v0.15.0 

type SpanFilter struct {
	TraceID string
	// SpanID matches a span's OWN id.
	SpanID string
	// ParentSpanID selects the operations that ran inside one HTTP hop — the
	// query a trace waterfall actually makes.
	ParentSpanID string
	Service      string
	Kind         string
	// MinDurationMS keeps only spans at least this slow. The point of a
	// breakdown is usually the slow thing.
	MinDurationMS float64
	// ErrorsOnly keeps only failed operations.
	ErrorsOnly bool
	Since      time.Time
	Until      time.Time
	// Search is a substring match over the name.
	Search string
	Limit  int
	Offset int
}

SpanFilter selects stored spans. The zero value matches everything within the store's own limits.

type SpanStore added in v0.15.0 

type SpanStore interface {
	// SaveSpans persists a batch. Called from the async writer, never from a
	// request hot path. Implementations must be safe for concurrent use.
	SaveSpans(ctx context.Context, spans []Span) error
	// QuerySpans returns matching spans oldest-first — reading a request's
	// work means reading it in the order it happened — plus the total
	// matching count, ignoring Limit/Offset.
	QuerySpans(ctx context.Context, f SpanFilter) (spans []Span, total int64, err error)
	// SpanBreakdown groups spans by name for a window, optionally narrowed to
	// one route. This is the "where did the time go" query.
	SpanBreakdown(ctx context.Context, since time.Time, route string, limit int) ([]SpanBreakdown, error)
	// SpanStats aggregates stored spans since a time.
	SpanStats(ctx context.Context, since time.Time) (*SpanSummary, error)
	// CountSpans returns the total stored.
	CountSpans(ctx context.Context) (int64, error)
	// PruneSpansBefore enforces age-based retention. Spans run well above
	// request volume, so they carry their own horizon.
	PruneSpansBefore(ctx context.Context, cutoff time.Time) (int64, error)
}

SpanStore is an OPTIONAL companion to Store: a driver may implement it to persist inner spans, and is a perfectly good driver if it does not. It is a separate interface rather than more methods on Store for the same reason AppLogStore and TraceStore are — Store is published and implemented outside this module, so adding a method to it breaks every third-party driver at compile time.

Detect support with a type assertion: 

if ss, ok := myStore.(ext.SpanStore); ok { ... }

CONTRACT — if you implement both Store and SpanStore, Store.Purge MUST also delete the spans belonging to the records it purges. Erasure that removes a tenant's requests but leaves the statements those requests ran is not erasure. ext/exttest asserts this.

type SpanSummary added in v0.15.0 

type SpanSummary struct {
	Total int64 `json:"total"`
	// ByKind and ByService are counts keyed by kind and by emitting service.
	ByKind    map[string]int64 `json:"by_kind"`
	ByService map[string]int64 `json:"by_service"`
	// RequestsWithSpans is how many HTTP spans have any inner detail — the
	// denominator for "how much of my traffic can I actually break down".
	RequestsWithSpans int64 `json:"requests_with_spans"`
	Errors            int64 `json:"errors"`
}

SpanSummary aggregates stored spans for a window.

type Stats 

type Stats struct {
	Total        int64            `json:"total"`
	Errors       int64            `json:"errors"`
	ErrorRate    float64          `json:"error_rate"`
	P50LatencyMS float64          `json:"p50_latency_ms"`
	P95LatencyMS float64          `json:"p95_latency_ms"`
	P99LatencyMS float64          `json:"p99_latency_ms"`
	StatusCounts map[string]int64 `json:"status_counts"` // by class: 2xx, 4xx...
	Series       []TimeBucket     `json:"series"`
	TopRoutes    []RouteStat      `json:"top_routes"`

	// BodiesKept is how many records in the window stored a request or
	// response body. Against Total it is the only honest read on what
	// sampling is actually doing — a `sample: 0.05` that quietly matches
	// nothing and a `sample: 1.0` look identical in every other number on the
	// page. Zero from a driver that does not compute it, so treat 0 as
	// "unknown" rather than "nothing was kept".
	BodiesKept int64 `json:"bodies_kept"`
	// BytesSeen is the request and response bytes that passed through, whether
	// or not their bodies were stored. Paired with BodiesKept it answers "what
	// is this costing me to keep".
	BytesSeen int64 `json:"bytes_seen"`
}

Stats is the dashboard's aggregate view over a time window.

type Store 

type Store interface {
	Save(ctx context.Context, rec *Record) error
	Query(ctx context.Context, f Filter) (records []Record, total int64, err error)
	Get(ctx context.Context, id int64) (*Record, error)
	Stats(ctx context.Context, since time.Time, bucket time.Duration) (*Stats, error)
	// RouteStats aggregates every route seen since the given time.
	RouteStats(ctx context.Context, since time.Time) ([]RouteDetail, error)
	// RuleMatchCounts reports how often each named rule fired since the given
	// time. Report a zero for a requested rule that never fired — that is the
	// interesting answer for a rule someone expected to be matching.
	RuleMatchCounts(ctx context.Context, since time.Time, ruleNames []string) ([]RuleMatch, error)
	// Count returns total stored records.
	Count(ctx context.Context) (int64, error)
	// Recent returns up to limit full records since a time (newest first).
	// limit <= 0 means DefaultAnalysisMaxRows. Prefer RecentFunc for anything
	// that folds over the result.
	Recent(ctx context.Context, since time.Time, limit int) ([]Record, error)
	// RecentFunc streams the same records one at a time, so memory stays at
	// one record regardless of how many there are. A non-nil error from fn
	// stops the walk and is returned to the caller.
	RecentFunc(ctx context.Context, since time.Time, limit int, fn func(*Record) error) error
	// ServiceStats aggregates per service — the fleet view.
	ServiceStats(ctx context.Context, since time.Time) ([]ServiceStat, error)
	// UsageByLabel aggregates traffic per consumer (a label value, e.g.
	// tenant) — the cost-attribution view.
	UsageByLabel(ctx context.Context, since time.Time, label string) ([]Usage, error)
	Prune(ctx context.Context, maxRows int64) (removed int64, err error)
	// PruneBefore deletes everything older than cutoff — age-based retention,
	// which is how data-retention policies are actually written.
	PruneBefore(ctx context.Context, cutoff time.Time) (removed int64, err error)
	// Purge deletes records matching a consumer label (and optionally a time
	// bound): "delete everything you hold for tenant X".
	//
	// Match the label value LITERALLY. The built-in SQLite driver once matched
	// it as a LIKE pattern, so purging a tenant named "acme_1" also destroyed
	// "acmeX1" — deleting a bystander's data is the one mistake an erasure
	// tool must never make. ext/exttest has the regression test.
	Purge(ctx context.Context, label, value string, before time.Time) (removed int64, err error)
	Close() error
}

Store is the persistence contract. Implementations must be safe for concurrent use; Save is called from the async writer, never from the request hot path.

Records arriving here are already governed — see the package doc. The two methods worth extra care are Purge, which backs erasure requests and must actually delete before it returns, and RecentFunc, which exists so an analysis pass costs one record of memory rather than the whole window.

internal/store.LogStore is an alias of this type, so a driver that satisfies Store is usable everywhere the built-in drivers are.

type StoreOpener 

type StoreOpener func(dsn string, settings Settings) (Store, error)

StoreOpener constructs a Store from the configured DSN and settings.

func LookupStore 

func LookupStore(name string) (StoreOpener, bool)

LookupStore returns the opener registered for name.

type TimeBucket 

type TimeBucket struct {
	Time       time.Time `json:"time"`
	Count      int64     `json:"count"`
	Errors     int64     `json:"errors"` // status >= 500
	AvgLatency float64   `json:"avg_latency_ms"`
	// ClientErrors counts 4xx separately. Folding them into Errors would say
	// the service is failing when a caller is sending bad requests, and the
	// two need opposite responses.
	ClientErrors int64 `json:"client_errors"`
	// P95Latency is the tail within this bucket. An average over a bucket
	// hides exactly the requests worth looking at: a handful of 3s responses
	// inside a minute of 5ms ones move the mean by almost nothing.
	P95Latency float64 `json:"p95_latency_ms"`
}

TimeBucket is one point in an aggregated traffic series.

type TraceFilter added in v0.14.0 

type TraceFilter struct {
	Since time.Time
	// ErrorsOnly keeps traces with at least one 5xx hop.
	ErrorsOnly bool
	// Service keeps traces any hop of which was served by this service.
	Service string
	// Search matches the root path or route.
	Search string
	// Labels must all match on the root hop.
	Labels map[string]string
	Limit  int
	Offset int
}

TraceFilter narrows a trace listing.

type TraceStore added in v0.14.0 

type TraceStore interface {
	// Traces returns matching traces newest-first, plus the total matching
	// count ignoring Limit/Offset.
	Traces(ctx context.Context, f TraceFilter) (traces []TraceSummary, total int64, err error)
}

TraceStore is an OPTIONAL companion to Store: a driver may implement it to list traces, and is a perfectly good driver if it does not. It is a separate interface rather than more methods on Store for the same reason AppLogStore is — Store is published and implemented outside this module, so adding a method to it breaks every third-party driver at compile time.

Detect support with a type assertion: 

if ts, ok := myStore.(ext.TraceStore); ok { ... }

A driver that does not implement it loses the trace list, not correlation: every record still carries its trace and span ids, and fetching one trace by ID is an ordinary Query.

type TraceSummary added in v0.14.0 

type TraceSummary struct {
	TraceID string `json:"trace_id"`
	// Root describes the entry hop — the span with no parent, or failing that
	// the earliest one. A trace whose root was never recorded (the entry
	// service is not instrumented) still lists, named by what WAS seen,
	// because a partial trace is evidence and hiding it helps nobody.
	Method  string `json:"method"`
	Route   string `json:"route"`
	Path    string `json:"path"`
	Service string `json:"service"`
	// Status of the root hop: what the caller was actually told.
	Status int `json:"status"`
	// Spans is the number of recorded hops.
	Spans int `json:"spans"`
	// Services is how many distinct services took part.
	Services int `json:"services"`
	// Errors counts hops that returned 5xx, including inner ones. An inner
	// failure a retry rescued never shows in the root status, and it is
	// exactly what someone opening a trace list is hunting for.
	Errors int `json:"errors"`
	// DurationMS is the root hop's duration when there is a root, which is
	// what the caller waited. Concurrent inner hops make the sum of spans
	// meaningless as a wall-clock figure.
	DurationMS float64 `json:"duration_ms"`
	// Start is the earliest hop's time, End the latest hop's completion.
	Start time.Time `json:"start"`
	End   time.Time `json:"end"`
	// Labels from the root hop, so a trace list can be read per tenant.
	Labels map[string]string `json:"labels,omitempty"`
	// LogLines is how many application log lines the whole trace produced,
	// or -1 when the store cannot answer cheaply.
	LogLines int64 `json:"log_lines"`
}

TraceSummary is one distributed trace rolled up to a single row: what was called, how many hops it took, and whether it went wrong.

The fields are deliberately the ones a person scans a list by. Everything else about a trace is a query away by ID, and putting it here would mean loading every hop's payload to render a table.

type Usage 

type Usage struct {
	Consumer   string             `json:"consumer"` // label value; "" -> "(unattributed)"
	Requests   int64              `json:"requests"`
	Errors     int64              `json:"errors"`
	ReqBytes   int64              `json:"req_bytes"`
	RespBytes  int64              `json:"resp_bytes"`
	DurationMS float64            `json:"duration_ms_total"`
	Meters     map[string]float64 `json:"meters,omitempty"`
}

Usage aggregates one consumer's traffic for cost attribution.

Source Files

View all Source files

Directories

Path Synopsis
Package exttest is the acceptance suite for an ext.Store implementation.
 Package exttest is the acceptance suite for an ext.Store implementation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.