sdk

Documentation

Overview

Package sdk defines the contract between the BanyanDB post-trace pipeline engine and user-supplied native Go plugins (each a .so loaded in-process via the standard Go plugin package). It is the single pinned surface plugin authors build against; see docs/design/post-trace-pipeline.md §2.5.

A plugin is one kind of the generic Plugin interface. Every kind shares Kind, Project, and Close; each kind adds its own processing method (the sampler kind adds Decide — see Sampler). Today the only kind is the sampler. A plugin is a package main built with `-buildmode=plugin` that exports exactly two symbols (the constructor name is the kind's convention — NewSampler for the sampler kind):

var  ABIVersion int                          // == sdk.ABIVersion
func NewSampler(config []byte) (sdk.Sampler, error)

The engine refuses to load a plugin whose ABIVersion differs from its own compiled sdk.ABIVersion, turning a silent miscompile into a fail-fast error. The config bytes are the canonical JSON serialization of the google.protobuf.Struct set in the plugin's proto payload (e.g. SamplerPlugin.config); the plugin unmarshals them into its own typed config.

In a TracePipelineConfig or StageRule, plugins are wired as an ordered chain (a sequential pipe): links run in declared order, each processing what the previous link kept, and a link that fails is bypassed. For an all-sampler chain this is the conjunction of the links' keep/drop verdicts.

The boundary deliberately crosses only stdlib slices plus the engine's stable, byte-sized valuetype.ValueType enum (a dependency-free leaf package, re-exported by pkg/pb/v1 as pbv1.ValueType), so no banyand/trace-internal struct — and no logging/meter dependency — is shared across the .so boundary.

Index

• Constants
• func DecodeTagValueInto(dst *Value, valueType valuetype.ValueType, raw []byte) error
• func EncodeTagValue(vt valuetype.ValueType, v any) ([]byte, error)
• type BypassInfo
• type Counter
• type Gauge
• type Histogram
• type Host
• type HostAware
• type Kind
• type Logger
• type Meter
• type Plugin
• type Projection
• type Sampler
  • func OpenSampler(path, symbol string, cfg []byte) (Sampler, error)
• type TagColumn
  • func (c *TagColumn) At(row int) (Value, error)
  • func (c *TagColumn) AtInto(row int, dst *Value) error
• type TraceBatch
• type TraceBlock
  • func (b *TraceBlock) Len() int
  • func (b *TraceBlock) Tag(name string) *TagColumn
• type Value
  • func DecodeTagValue(valueType valuetype.ValueType, raw []byte) (Value, error)
  • func (v Value) Bytes() []byte
  • func (v Value) Float64() float64
  • func (v Value) Int64() int64
  • func (v Value) Int64Arr() []int64
  • func (v Value) IsNull() bool
  • func (v Value) Str() string
  • func (v Value) StrArr() []string
  • func (v Value) ValueType() valuetype.ValueType
• type Verdict
  • func EvaluateChain(samplers []Sampler, batch *TraceBatch, onBypass func(idx int, info BypassInfo)) Verdict
  • func EvaluateChainInto(samplers []Sampler, batch *TraceBatch, mask []bool, ...) Verdict

Constants

const (
	// BypassReasonDecideError means Sampler.Decide returned a non-nil error.
	BypassReasonDecideError = "decide_error"
	// BypassReasonLengthMismatch means Verdict.Keep's length did not match
	// len(batch.Traces).
	BypassReasonLengthMismatch = "length_mismatch"
	// BypassReasonPanic means Sampler.Decide panicked; the panic was recovered.
	// Distinguished from BypassReasonDecideError so callers can tell "the
	// plugin returned an error" apart from "the plugin crashed" even though
	// both fail open identically.
	BypassReasonPanic = "panic"
)

Bypass reasons reported via BypassInfo.Reason. A link is bypassed (pass-through: the running keep-mask is left unchanged for that link) for exactly one of these three causes.

const ABIVersion = 1

ABIVersion is compiled into the host and must be re-exported, unchanged, by every plugin. The engine refuses to load a plugin whose ABIVersion differs.

Functions

func DecodeTagValueInto

func DecodeTagValueInto(dst *Value, valueType valuetype.ValueType, raw []byte) error

DecodeTagValueInto decodes one marshaled tag value into dst. Existing array capacity in dst is reused; all fields from a preceding value are reset.

func EncodeTagValue

func EncodeTagValue(vt valuetype.ValueType, v any) ([]byte, error)

EncodeTagValue marshals a raw Go value into the native tag-value byte layout that DecodeTagValue decodes — the inverse of DecodeTagValue. It deliberately takes a plain Go value (any), not a Value: Value's fields are unexported and this package exports no constructor for it, so an encoder built around Value would be unconstructible from outside this package (e.g. from pkg/pipeline/sdk/sdktest, a different package, or a plugin author's own test) — defeating the purpose of a shared, reusable encoder.

Only a genuinely nil v (an untyped nil `any`) encodes to a nil raw value — the native "tag absent on this row" representation. A present-but-empty collection ([]byte{}, []string{}, []int64{}) encodes to a NON-nil empty raw value, so it decodes back as an empty value rather than as null, consistently across all three collection types.

Supported (vt, Go type) pairs:

• ValueTypeStr: string
• ValueTypeInt64: int64
• ValueTypeTimestamp: int64 (unix nanoseconds; same 8-byte wire layout as Int64)
• ValueTypeFloat64: float64
• ValueTypeBinaryData: []byte
• ValueTypeStrArr: []string
• ValueTypeInt64Arr: []int64

A mismatch between vt and v's concrete Go type, or an unsupported vt, returns an error.

Types

type BypassInfo

type BypassInfo struct {
	// Err is the Decide error (BypassReasonDecideError) or the recovered
	// panic wrapped as an error (BypassReasonPanic). Nil for
	// BypassReasonLengthMismatch.
	Err error
	// Reason classifies why the link was bypassed: one of
	// BypassReasonDecideError, BypassReasonLengthMismatch, or
	// BypassReasonPanic.
	Reason string
	// Got is len(Verdict.Keep) as returned by the link (BypassReasonLengthMismatch only).
	Got int
	// Want is len(batch.Traces) (BypassReasonLengthMismatch only).
	Want int
}

BypassInfo describes why EvaluateChain bypassed one chain link.

type Counter

type Counter interface {
	// Inc adds delta to the counter for the given label values. Label values
	// must align positionally with the label names declared in Meter.Counter.
	// Delta must be non-negative.
	Inc(delta float64, labelValues ...string)
}

Counter is a monotonically increasing metric. It must not be decremented.

type Gauge

type Gauge interface {
	// Set replaces the gauge value for the given label values.
	Set(value float64, labelValues ...string)
	// Add adds delta (positive or negative) to the gauge for the given label values.
	Add(delta float64, labelValues ...string)
}

Gauge is a metric whose value can go up or down freely.

type Histogram

type Histogram interface {
	// Observe records a single observation for the given label values.
	Observe(value float64, labelValues ...string)
}

Histogram records observations into pre-declared buckets. Use it to track distributions such as latency or payload size.

type Host

type Host interface {
	// Meter returns the metric surface scoped to this plugin's group.
	Meter() Meter
	// Logger returns the logging surface scoped to this plugin's group.
	Logger() Logger
}

Host is the telemetry context the engine injects into a HostAware plugin. It provides scoped metric and logging surfaces that the engine owns; the plugin must not cache a Host beyond the lifetime of the pipeline config that created it. Instrument handles obtained from Meter() are valid for the lifetime of that Host: do not cache a Counter, Gauge, or Histogram across distinct UseHost calls, because the engine may reclaim and rebuild a group's metric series on reconcile or teardown — a handle from a prior Host would emit into a reclaimed series.

type HostAware

type HostAware interface {
	// UseHost delivers the engine-owned telemetry context to the plugin. It is
	// called once, before the first Decide, with a Host scoped to the plugin's
	// group.
	UseHost(Host)
}

HostAware is the optional interface a plugin implements to receive a Host. The engine type-asserts the constructed sampler to HostAware and, if present, calls UseHost exactly once before the plugin's first Decide call. Plugins that do not implement HostAware are unaffected — there is no ABI change. A plugin may cache the Host and use it for the duration of Decide; the engine guarantees a distinct Host per group so metrics and logs are automatically namespaced.

type Kind

type Kind uint8

Kind identifies a Plugin's role in a chain. It mirrors the set arm of the proto Plugin.kind oneof and lets the engine label and cross-check a plugin before type-asserting it to the matching kind interface. New kinds are added here in lockstep with new oneof arms.

const (
	// KindUnspecified is the zero value; a real plugin never reports it.
	KindUnspecified Kind = iota
	// KindSampler is a plugin that owns a keep/drop verdict (see Sampler).
	KindSampler
)

type Logger

type Logger interface {
	// Debug logs a debug-level message with optional structured key/value pairs.
	Debug(msg string, keysAndValues ...any)
	// Info logs an info-level message with optional structured key/value pairs.
	Info(msg string, keysAndValues ...any)
	// Warn logs a warning-level message with optional structured key/value pairs.
	Warn(msg string, keysAndValues ...any)
	// Error logs an error-level message with optional structured key/value pairs.
	Error(msg string, keysAndValues ...any)
}

Logger is the bounded logging surface handed to a plugin via Host. It exposes no third-party types (e.g. zerolog, zap) — only a message plus alternating key/value pairs so no external logger dependency crosses the .so boundary. Implementations are expected to be safe for concurrent use.

type Meter

type Meter interface {
	// Counter returns or creates a counter with the given name and label names.
	Counter(name string, labelNames ...string) Counter
	// Gauge returns or creates a gauge with the given name and label names.
	Gauge(name string, labelNames ...string) Gauge
	// Histogram returns or creates a histogram with the given name, explicit
	// bucket boundaries, and label names.
	Histogram(name string, buckets []float64, labelNames ...string) Histogram
}

Meter is the bounded metric surface handed to a plugin via Host. The host enforces naming, labels, cardinality, and lifecycle; the plugin only requests. Methods are safe to call from multiple goroutines concurrently.

type Plugin

type Plugin interface {
	// Kind reports which plugin kind this is, for engine bookkeeping and as a
	// cross-check against the proto oneof arm. It must be constant for the
	// plugin's lifetime.
	Kind() Kind

	// Project is the column-selection handshake called when a merge chain is
	// built (once per merge, not necessarily once per process load). The engine
	// unions the projections from all samplers in the chain and honors the
	// result for that chain's lifetime: Tags drives the native tag projection
	// (only those tag columns are decoded); SpanIDs and Spans gate the spans
	// stream. Intrinsic columns (TraceID, MinTS, MaxTS) are always present
	// regardless of the projection.
	Project() Projection

	// Close releases any resources the plugin holds. It is called once when the
	// pipeline config is removed; because Go plugins cannot be unloaded, the .so
	// itself stays mapped until the process restarts.
	Close() error
}

Plugin is the common interface every plugin kind satisfies — the generic link type the engine handles uniformly. The engine constructs a Plugin, checks Kind against the proto oneof arm that named it, then type-asserts to that kind's interface (e.g. Sampler) for the kind-specific call. Project and Close are shared by every kind; the per-kind processing method lives on the kind interface. Each kind keeps its own constructor symbol convention (the sampler kind defaults to NewSampler; a future kind would use its own, e.g. NewTransformer).

type Projection

type Projection struct {
	// Tags names the tag columns to decode into TraceBlock.Tags. Empty means no
	// tag columns are decoded.
	Tags []string
	// SpanIDs opts in to the span-id column. Default false. Span ids and span
	// bodies share one encoded data block in the native layout, so requesting
	// span ids forces a read of the spans stream — it is not free metadata.
	SpanIDs bool
	// Spans opts in to the heavy span-body column. Default false: the engine
	// leaves TraceBlock.Spans nil and, on the merge raw fast path, never decodes
	// span bodies. Set true only when the verdict reads them.
	Spans bool
}

Projection is the plugin's up-front column request — one handshake covering every optional column. Intrinsic columns (TraceID, MinTS, MaxTS) are always materialized and are not listed here.

type Sampler

type Sampler interface {
	Plugin

	// Decide receives a vectorized batch of assembled per-trace blocks and
	// returns a keep-mask aligned to batch.Traces. The batch is READ-ONLY:
	// Decide must not mutate any slice it receives. The keep-mask is the only
	// output channel; the engine writes retained traces from its own untouched
	// block data, so a returned error or a length-mismatched verdict makes the
	// engine fail open (retain the whole batch).
	Decide(batch *TraceBatch) (Verdict, error)
}

Sampler is the keep/drop kind of Plugin (Kind reports KindSampler). The engine calls Project once at load, then Decide once per batch, then Close at unload. In a chain it is a conjunction link: each Sampler narrows the traces the next link sees.

func OpenSampler

func OpenSampler(path, symbol string, cfg []byte) (Sampler, error)

OpenSampler opens the .so at path, verifies its exported ABIVersion symbol matches this package's ABIVersion, looks up the constructor symbol, and calls it with cfg (the canonical JSON encoding of the plugin's config).

This is the ONE loader-contract implementation: banyand/trace's host loader delegates here, pkg/pipeline/sdk/sdktest.LoadSO calls it directly to drive a real .so offline, and any future toolchain-authoritative CLI validate/decide command would call it too — so every caller runs the exact same code path, never a hand-rolled mirror that can silently drift from the host.

A panic while opening the plugin or while running its constructor is recovered and returned as an error; OpenSampler itself never panics.

type TagColumn

type TagColumn struct {
	// Name is the tag key.
	Name string
	// Values holds one marshaled value per span row; a nil element means the
	// tag is absent on that row.
	Values [][]byte
	// ValueType is the engine's stable, byte-sized type tag for every value in
	// the column.
	ValueType valuetype.ValueType
}

TagColumn mirrors the native tag: a row-aligned column of marshaled values plus the value type needed to decode them via At.

func (*TagColumn) At

func (c *TagColumn) At(row int) (Value, error)

At decodes the value at the given span row. A nil element decodes to a null Value. It returns an error if row is out of range.

func (*TagColumn) AtInto

func (c *TagColumn) AtInto(row int, dst *Value) error

AtInto decodes the value at the given span row into dst. Callers may reuse dst across rows to avoid copying the Value and reallocating array storage.

type TraceBatch

type TraceBatch struct {
	// Traces holds one block per trace_id. The verdict's keep-mask is aligned
	// to this slice.
	Traces []TraceBlock
}

TraceBatch is a vectorized batch of assembled per-trace blocks. It is the engine's native columnar trace layout, shared read-only with the plugin.

type TraceBlock

type TraceBlock struct {
	// TraceID identifies the trace; the keep/drop verdict is per trace_id.
	TraceID string
	// Tags holds the projected tag columns, one per Projection.Tags entry that
	// the trace actually carries.
	Tags []TagColumn
	// SpanIDs is the row-aligned span-id column; nil unless Projection.SpanIDs.
	SpanIDs []string
	// Spans is the row-aligned span-body column (opaque marshaled bytes); nil
	// unless Projection.Spans.
	Spans [][]byte
	// MinTS is the earliest span start in unix nanoseconds.
	MinTS int64
	// MaxTS is the latest span end in unix nanoseconds; trace duration is
	// MaxTS - MinTS.
	MaxTS int64
}

TraceBlock mirrors the native trace block: every populated column is indexed in lockstep by span row i in [0, Len). Intrinsic columns are always set; Tags, SpanIDs, and Spans appear only as requested by Project. Slices are shared with the engine and must be treated as read-only.

func (*TraceBlock) Len

func (b *TraceBlock) Len() int

Len reports the number of span rows in the block. It is available only when a row-indexed column (a projected tag, SpanIDs, or Spans) was materialized; with a metadata-only projection it returns 0.

func (*TraceBlock) Tag

func (b *TraceBlock) Tag(name string) *TagColumn

Tag returns the projected tag column with the given name, or nil if the trace did not carry it (or the plugin did not project it).

type Value

type Value struct {
	// contains filtered or unexported fields
}

Value is a single decoded tag value. The accessor matching ValueType returns the decoded datum; the others return their zero value. A nil raw value decodes to a null Value (IsNull reports true).

func DecodeTagValue

func DecodeTagValue(valueType valuetype.ValueType, raw []byte) (Value, error)

DecodeTagValue decodes one marshaled tag value, as stored in the native trace block, into a typed Value. It mirrors the engine's own per-row decode so a plugin never needs to import banyand/trace internals. A nil raw value yields a null Value.

func (Value) Bytes

func (v Value) Bytes() []byte

Bytes returns the raw value (valid for ValueTypeBinaryData).

func (Value) Float64

func (v Value) Float64() float64

Float64 returns the float value (valid for ValueTypeFloat64).

func (Value) Int64

func (v Value) Int64() int64

Int64 returns the integer value (valid for ValueTypeInt64 and, as unix nanoseconds, ValueTypeTimestamp).

func (Value) Int64Arr

func (v Value) Int64Arr() []int64

Int64Arr returns the integer-array value (valid for ValueTypeInt64Arr).

func (Value) IsNull

func (v Value) IsNull() bool

IsNull reports whether the tag was absent on this row.

func (Value) Str

func (v Value) Str() string

Str returns the string value (valid for ValueTypeStr).

func (Value) StrArr

func (v Value) StrArr() []string

StrArr returns the string-array value (valid for ValueTypeStrArr).

func (Value) ValueType

func (v Value) ValueType() valuetype.ValueType

ValueType returns the type tag of the decoded value.

type Verdict

type Verdict struct {
	// Keep must have the same length as the batch; Keep[i] true retains
	// Traces[i]. A length mismatch makes the engine fail open.
	Keep []bool
}

Verdict is the per-trace decision, aligned to TraceBatch.Traces.

func EvaluateChain

func EvaluateChain(samplers []Sampler, batch *TraceBatch, onBypass func(idx int, info BypassInfo)) Verdict

EvaluateChain runs an ordered sampler chain over batch and returns the conjunction (AND) keep-mask. It is the single shared implementation of chain-evaluation semantics — the host engine's merge chain (banyand/trace's mergeChain.runChain) and the offline pkg/pipeline/sdk/sdktest.RunChain harness both call this function, so how a chain of samplers combines into one verdict is never duplicated or allowed to drift between the two.

Every sampler runs, in declared order, over the SAME batch — a chain link narrows the running mask, it does not filter rows out of what the next link sees. A link that panics, errors, or returns a keep-mask of the wrong length is bypassed: the running mask is left unchanged for that link (fail-open, per link), and onBypass — when non-nil — is invoked with the link's index and a BypassInfo describing why.

onBypass is the sole observability seam: it lets a caller log and/or count a bypass without EvaluateChain importing a logger or meter (this package must stay dependency-pure — see importgraph_test.go). The host passes an onBypass that reproduces its pre-existing WARN log (and, going forward, increments a metric); sdktest.RunChain passes a recorder that captures every BypassInfo into its Report for offline inspection.

func EvaluateChainInto

func EvaluateChainInto(samplers []Sampler, batch *TraceBatch, mask []bool, onBypass func(idx int, info BypassInfo)) Verdict

EvaluateChainInto runs EvaluateChain using mask as reusable verdict storage. The returned verdict aliases mask when its capacity is sufficient. Callers must keep the storage alive and immutable for the duration of every sampler call, including a sampler still running after a host-side timeout.