storage

package module
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: Apache-2.0 Imports: 45 Imported by: 0

README

storage Go Reference codecov experimental

Low-level, distributed, OpenTelemetry-centric storage library (Go) for signals:

  • Metrics
  • Logs
  • Traces
  • Profiles

Built as the storage tier for oteldb, an OpenTelemetry observability backend — embeddable as a native Go storage engine for all signals.

Currently in WIP and PoC. See ARCHITECTURE.md for current design.

Documentation

Overview

Package storage is the embeddable entry point for the oteldb storage library.

It is a low-level, distributed, OpenTelemetry-centric storage engine for signals (metrics, logs, traces, profiles), exposed as a small Storage facade. Everything else is an implementation detail behind this package. See the package layout and milestone plan in DESIGN.md at the repository root.

The in-memory engine (InMemory) is the default in tests: a full ingest+query path with WAL and durable flush disabled.

Index

Constants

View Source 
const (
	// KeyScopeResource marks a resource attribute (stream identity).
	KeyScopeResource = KeyScope(recordengine.KeyScopeResource)
	// KeyScopeScope marks an instrumentation-scope attribute (stream identity).
	KeyScopeScope = KeyScope(recordengine.KeyScopeScope)
	// KeyScopeRecord marks a per-record attribute (the attrs column).
	KeyScopeRecord = KeyScope(recordengine.KeyScopeRecord)
)

Variables

View Source 
var ErrClosed = errors.New("storage: closed")

ErrClosed is returned by Storage methods after Storage.Close.

View Source 
var ErrNotEphemeral = errors.New("storage: reset requires an ephemeral backend")

ErrNotEphemeral is returned by Storage.Reset when the backend is durable: Reset wipes all ingested data and is only permitted on an ephemeral (in-memory) store.

View Source 
var ErrNotOwner = errors.New("storage: this node is not the compaction owner of the tenant/shard")

ErrNotOwner is returned by an Admin flush/compact when this node is not the cluster compaction-owner (ring primary) of the tenant/shard, so it must not write that shard's parts.

Functions

This section is empty.

Types

type Accepted 

type Accepted struct {
	// Accepted is the number of accepted data points (metric points, log records,
	// span, profile samples depending on signal).
	Accepted int64
	// Rejected is the number of rejected data points (e.g. over a tenant limit or
	// out of the OOO window). Rejections are reported back to the producer via OTLP
	// partial_success so it can retry.
	Rejected int64
	// RejectedReason is a machine-readable reason for rejections (empty if none).
	RejectedReason string
}

Accepted carries per-OTLP partial-success counts (DESIGN.md §5). It implements the OTLP partial-success semantics: accepted/rejected data points + a reason string.

type Admin added in v0.10.0 

type Admin struct {
	// contains filtered or unexported fields
}

Admin is the imperative operator-control surface, complementing the background maintenance loop: force a flush, compaction, retention sweep, or ownership reconciliation on demand. It is the surface a CLI/UI drives. Obtain it from Storage.Admin; it holds no state of its own.

The key argument is the engine key — the tenant id in the default layout, or a metric shard key ({tenant}/_s{idx}) when Options.Cluster sets ShardsPerTenant > 1. In cluster mode flush/compact act only on shards this node is the ring-primary of (else ErrNotOwner), so a shard's parts are still written by exactly one node — the same invariant the maintenance loop preserves.

func (Admin) Compact added in v0.10.0 

func (a Admin) Compact(ctx context.Context, key signal.TenantID, sig signal.Signal) error

Compact merges a tenant/shard's parts for one signal now, applying the tenant's policy (retention cutoff, plus downsampling/recompression/precision for metrics) — the same merge the background loop runs, so there is no parallel code path. No-op when nothing is ingested; returns ErrNotOwner in cluster mode unless this node is the shard's ring-primary.

func (Admin) Flush added in v0.10.0 

func (a Admin) Flush(ctx context.Context, key signal.TenantID, sig signal.Signal) error

Flush drains a tenant/shard's in-memory head for one signal to an immutable part now. It is a no-op (nil) when nothing has been ingested for that key+signal. In cluster mode it returns ErrNotOwner unless this node is the shard's ring-primary.

func (Admin) MaintainNow added in v0.10.0 

func (a Admin) MaintainNow(ctx context.Context) error

MaintainNow runs one full maintenance cycle immediately — flush + merge + retention across every owned tenant and signal (the background loop's body). Best-effort: per-engine errors are logged, not returned, matching the loop.

func (Admin) Rebalance added in v0.10.0 

func (a Admin) Rebalance(ctx context.Context) error

Rebalance triggers an immediate cluster ownership reconciliation (the maintenance loop otherwise does it on its tick), so a freshly-changed ring takes effect without waiting. It is a no-op in single-node mode.

func (Admin) Retention added in v0.10.0 

func (a Admin) Retention(ctx context.Context, key signal.TenantID) error

Retention forces a retention sweep across all of a tenant/shard's signals by compacting each (a merge drops parts older than the policy's cutoff). Signals this node does not own are skipped.

type AdmissionStats added in v0.4.0 

type AdmissionStats struct {
	Accepted            int64
	RejectedOOO         int64 // out of the out-of-order window
	RejectedRate        int64 // over IngestBytesPerSecond
	RejectedCardinality int64 // over MaxSeries
	RejectedInFlight    int64 // over MaxInFlightBytes
	// SampledDropped counts points dropped by budgeted (lossy) sampling. Unlike the Rejected
	// counters these are NOT failures — the kept samples carry a scale factor that represents
	// them — so they count toward Accepted, not Rejected.
	SampledDropped int64
	// Overflowed counts points for new series past the soft cardinality budget that were routed to
	// an overflow series instead of being shed. Like SampledDropped these are accepted (the data is
	// retained, relabeled), not rejected.
	Overflowed int64
}

AdmissionStats are the per-tenant admission counters (the overload-control meta-metrics): how many samples were accepted and how many were shed, by reason. They make it observable which valve tripped under load. Counts are cumulative since the store opened.

func (AdmissionStats) Rejected added in v0.4.0 

func (s AdmissionStats) Rejected() int64

Rejected returns the total shed across all reasons.

type CacheStats added in v0.10.0 

type CacheStats struct {
	// Decode is the decoded-column cache, summed across metric engines (zero when unconfigured).
	Decode engine.DecodeCacheStats
}

CacheStats aggregates the read-path caches.

type CardinalityStats added in v0.12.0 

type CardinalityStats struct {
	// TotalSeries is the distinct series/streams the engine indexes (head ∪ flushed).
	TotalSeries int64
	// DistinctLabelNames is the number of distinct indexed label names.
	DistinctLabelNames int
	// SymbolCount is the size of the engine's interned-symbol table (names + values).
	SymbolCount int
	// TopLabelNames is the highest-cardinality label names, sorted by series count descending then
	// name; bounded by the topN argument.
	TopLabelNames []LabelCardinality
}

CardinalityStats summarizes a (tenant, signal) engine's label cardinality — the operator's first stop for a cardinality-explosion incident. It is computed from the in-memory inverted index (which spans head ∪ flushed series), so it does no backend I/O and is safe to poll. See Storage.Cardinality.

type ClusterStats added in v0.10.0 

type ClusterStats struct {
	// Self is this node's address.
	Self string
	// Members is the live ring membership (sorted by id).
	Members []MemberStats
	// Owned is the shards this node currently holds a compaction claim on (sorted).
	Owned []string
	// LastRebalance is the primary handoffs enacted at the most recent ring change (empty if none).
	LastRebalance []RebalanceMove
}

ClusterStats is the cluster-mode view of this node.

type ColumnInfo added in v0.12.0 

type ColumnInfo struct {
	Name     string
	Kind     string // physical type: int64 / float64 / bytes / int128
	Codec    string // value codec
	Compress string // block-compression algorithm
}

ColumnInfo is one part column's physical description.

type Durability 

type Durability uint8

Durability is the WAL + flush policy (DESIGN.md §5, §8). 

const (
	// DurabilityDefault is the default durability for a durable backend (WAL on,
	// flush to backend). For the in-memory engine use [DurabilityEphemeral].
	DurabilityDefault Durability = iota
	// DurabilityEphemeral disables the WAL and durable flush: the head answers all
	// queries and "flushed" parts live in RAM, dropped on [Storage.Close]. This is
	// the in-memory engine and the default in tests (DESIGN.md §5).
	DurabilityEphemeral
)

type KeyInfo added in v0.5.0 

type KeyInfo struct {
	Key   []byte
	Scope KeyScope
}

KeyInfo is a distinct attribute key and the scope(s) it was observed in. Key aliases interned, low-cardinality metadata owned by the engine; copy it to retain past the call.

type KeyScope added in v0.5.0 

type KeyScope uint8

KeyScope is a bitset of the scopes an attribute key appears in. A key can appear in more than one (e.g. as a resource attribute on one stream and a record attribute on another).

type LabelCardinality added in v0.12.0 

type LabelCardinality struct {
	Name           string
	Series         int64
	DistinctValues int
}

LabelCardinality is one label name's cardinality: how many series carry it and how many distinct values it takes across them.

type MemberStats added in v0.10.0 

type MemberStats struct {
	ID   string
	Zone string
	Addr string
}

MemberStats is one cluster member's identity.

type Option 

type Option func(*Options)

Option is a functional option applied to Options by Open/InMemory.

func WithAggregateStats added in v0.6.0 

func WithAggregateStats() Option

WithAggregateStats writes the per-series aggregate sidecar that lets Storage.AggregateMetrics answer range-covering aggregates without decoding. See Options.AggregateStats.

func WithBackend 

func WithBackend(b backend.Backend) Option

WithBackend sets the storage backend.

func WithCluster 

func WithCluster(c *cluster.Config) Option

WithCluster sets the cluster configuration. nil ⇒ single-node.

func WithDecodeCache added in v0.6.0 

func WithDecodeCache(maxBytes int64) Option

WithDecodeCache enables a per-tenant LRU cache of decoded part columns, sized to maxBytes, plus concurrent prefetch of a fetch's parts. See Options.DecodeCacheBytes.

func WithDecodeMemory added in v0.25.0 

func WithDecodeMemory(maxBytes int64) Option

WithDecodeMemory caps the process-wide in-flight decoded column bytes across concurrent queries (decode admission control), shared by every tenant engine. Zero disables it. See Options.DecodeMemoryBytes.

func WithDurability 

func WithDurability(d Durability) Option

WithDurability sets the WAL + flush policy.

func WithEncoding 

func WithEncoding(p encoding.Profile) Option

WithEncoding sets the default encoding profile.

func WithFlushInterval 

func WithFlushInterval(ns int64) Option

WithFlushInterval sets the head flush time interval in nanoseconds. Zero keeps the durable-store default ([defaultFlushInterval]); a negative value disables the background flush/compaction loop entirely (manual/Admin-driven maintenance only). See Options.FlushInterval.

func WithFlushThresholdBytes 

func WithFlushThresholdBytes(n int64) Option

WithFlushThresholdBytes sets the head flush size threshold.

func WithLogger added in v0.5.0 

func WithLogger(l *zap.Logger) Option

WithLogger sets the zap logger the library logs through (DESIGN.md §16). nil ⇒ a no-op logger.

func WithMaintenanceConcurrency added in v0.6.0 

func WithMaintenanceConcurrency(n int) Option

WithMaintenanceConcurrency caps the background maintenance loop's parallel flush/merge fan-out across engines. Zero ⇒ a CPU-derived default.

func WithMeterProvider added in v0.5.0 

func WithMeterProvider(mp metric.MeterProvider) Option

WithMeterProvider sets the OTel meter provider for the library's metrics (including the admission meta-metrics). nil ⇒ the noop meter.

func WithOOOWindow 

func WithOOOWindow(ns int64) Option

WithOOOWindow sets the out-of-order ingestion window in nanoseconds.

func WithQueryCache added in v0.2.0 

func WithQueryCache(maxEntries int) Option

WithQueryCache enables a shared bounded-LRU results cache on Storage.Fetcher holding at most maxEntries results. maxEntries ≤ 0 disables the cache. Only fully-pushable equality requests over an explicit tenant set are cached, and entries do not auto-invalidate.

func WithQueryCacheFreshness added in v0.2.0 

func WithQueryCacheFreshness(ns int64) Option

WithQueryCacheFreshness sets the results-cache recent-window guard in nanoseconds: a query whose window ends within ns of now is not cached, so only settled windows are memoized. ns ≤ 0 disables the guard. Has effect only with the cache enabled (see WithQueryCache).

func WithQuerySplitInterval added in v0.2.0 

func WithQuerySplitInterval(ns int64) Option

WithQuerySplitInterval enables split-by-interval on Storage.Fetcher: a query window is divided into aligned sub-windows of ns nanoseconds, fetched concurrently and merged. ns ≤ 0 disables splitting.

func WithReadCache added in v0.6.0 

func WithReadCache(maxBytes int64) Option

WithReadCache enables an in-memory LRU object cache over the backend, sized to maxBytes (the object-store read cache for the cold tier). Skipped for an ephemeral backend. See Options.ReadCacheBytes.

func WithRetry added in v0.5.0 

func WithRetry(c reliability.RetryConfig) Option

WithRetry sets the transport reliability profile (per-attempt timeouts, bounded retries, hedged reads). Use reliability.LossyEnvironment in noisy networks; the zero config keeps the mild reliability.Default.

func WithTenancy 

func WithTenancy(r tenant.Resolver) Option

WithTenancy sets the tenant policy resolver.

func WithTenant 

func WithTenant(fn func(signal.Resource, signal.Scope) signal.TenantID) Option

WithTenant sets the record→tenant routing callback (resource/scope → tenant id).

func WithTracerProvider added in v0.5.0 

func WithTracerProvider(tp trace.TracerProvider) Option

WithTracerProvider sets the OTel tracer provider for the library's spans. nil ⇒ the noop tracer.

func WithWALDir 

func WithWALDir(dir string) Option

WithWALDir sets the WAL directory (file backend + durability).

func WithWALSync added in v0.3.0 

func WithWALSync(m WALSyncMode) Option

WithWALSync sets the WAL fsync policy (WALSyncNone/WALSyncAlways/WALSyncInterval).

func WithWALSyncInterval added in v0.3.0 

func WithWALSyncInterval(d time.Duration) Option

WithWALSyncInterval selects background fsync (WALSyncInterval) every d.

type Options 

type Options struct {
	// Backend is the storage backend (memory/file/s3). If nil, [Open] defaults to
	// [backend.Memory] — the in-memory, ephemeral engine (DESIGN.md §5).
	Backend backend.Backend

	// Cluster is the cluster configuration. nil ⇒ single-node (the cluster layer
	// is absent, DESIGN.md §3, §11). Single-node users ignore L0 entirely.
	Cluster *cluster.Config

	// Tenant derives a record's tenant id from its Resource and Scope (so one OTLP
	// batch may fan out to many tenants). If nil, every record routes to "default".
	Tenant func(signal.Resource, signal.Scope) signal.TenantID

	// Tenancy resolves a tenant id to limits, retention, downsampling, and routing.
	// If nil, a permissive default resolver is used.
	Tenancy tenant.Resolver

	// Encoding is the default codec-chain profile per column kind (DESIGN.md §6).
	// If nil, a sensible default is used.
	Encoding encoding.Profile

	// Durability is the WAL + flush policy. [DurabilityEphemeral] disables both —
	// the in-memory engine: the head answers all queries and "flushed" parts are
	// kept in RAM and dropped on [Storage.Close] (DESIGN.md §5).
	Durability Durability

	// WALDir is the WAL directory for the file backend with durability enabled.
	// Ignored when [Durability] is [DurabilityEphemeral].
	WALDir string

	// WALSync is the WAL fsync policy (default [WALSyncNone]). Ignored without a WAL.
	WALSync WALSyncMode

	// WALSyncInterval is the background fsync period when [WALSync] is [WALSyncInterval]. Zero ⇒
	// default. (Resolved into a duration internally.)
	WALSyncInterval int64 // nanoseconds

	// FlushThresholdBytes is the head size at which a flush to an immutable part is
	// triggered. Zero ⇒ default.
	FlushThresholdBytes int64

	// ReadCacheBytes enables an in-memory LRU cache over backend read objects (immutable part
	// columns/manifests/marks/index), sized to this many bytes. It targets the cold tier
	// (file/S3), where a part is otherwise re-read on every query; recommended for those
	// deployments. Zero disables it, and it is always skipped for an ephemeral (in-memory)
	// backend, which is already RAM. See [backend.Cached].
	ReadCacheBytes int64

	// DecodeCacheBytes enables a per-tenant LRU cache of *decoded* part columns, sized to this many
	// bytes. It eliminates the column re-decode that the read cache cannot — a hit returns the
	// already-decoded arrays — so it helps every backend (a decode is CPU even when the read is
	// RAM-fast). With it, a fetch also prefetches its parts' decodes concurrently. Zero disables it.
	DecodeCacheBytes int64

	// DecodeMemoryBytes caps the total in-flight decoded column bytes across concurrent queries,
	// process-wide (one budget shared by every tenant engine): a query reserves its estimated
	// decode footprint before reading any part, blocking while the budget is exhausted, so query
	// concurrency cannot drive the resident heap past the cap — N heavy queries serialize through
	// it instead of each materializing whole columns at once. It trades tail latency under
	// saturation for a memory ceiling; size it to the process memory budget minus the caches and
	// baseline. Zero disables it (no admission control). A query whose own footprint exceeds the
	// whole budget is admitted alone (it cannot be bounded below its own footprint).
	DecodeMemoryBytes int64

	// AggregateStats writes a per-series aggregate sidecar (count/sum/min/max) alongside each metric
	// part, so [Storage.AggregateMetrics] answers a range-covering aggregate without decoding the
	// value column — returning one number per series instead of every sample. It costs a little
	// storage per series; off by default. AggregateMetrics works without it (by decoding).
	AggregateStats bool

	// FlushInterval is the max age of unflushed head data: the cadence of the background loop that
	// flushes each head to a part and runs compaction/retention. Zero ⇒ a sane default
	// ([defaultFlushInterval]) for a durable store — a non-ephemeral engine MUST flush periodically
	// or its head and WAL grow unbounded in RAM until the process OOMs (and every restart replays the
	// whole WAL). A negative value explicitly disables the background loop (manual/[Admin]-driven
	// maintenance only) — the opt-in escape hatch for that unbounded-growth mode. The ephemeral
	// in-memory engine never flushes, so the default does not apply to it.
	// (Resolved into a duration internally to keep the API import-light.)
	FlushInterval int64 // nanoseconds

	// OOOWindow is the out-of-order ingestion window in nanoseconds (DESIGN.md §8).
	// Samples older than (newest - OOOWindow) are rejected (counted). Zero ⇒ default.
	OOOWindow int64 // nanoseconds

	// MaintenanceConcurrency caps how many engines the background maintenance loop flushes/merges
	// (and fsyncs) concurrently. Engines are independent per-tenant/per-signal shards, so this
	// bounds the parallel compaction fan-out against the backend. Zero ⇒ a CPU-derived default.
	MaintenanceConcurrency int

	// QuerySplitInterval, when > 0, makes [Storage.Fetcher] split a query's time window
	// into aligned sub-windows of this width (nanoseconds), fetched concurrently and merged
	// (the query-frontend "split by interval" technique). Zero ⇒ no splitting.
	QuerySplitInterval int64 // nanoseconds

	// QueryCacheEntries, when > 0, gives [Storage.Fetcher] a shared bounded-LRU results
	// cache of this many entries. Only fully-pushable (serializable-equality) requests over
	// an explicit tenant set are cached. Zero ⇒ no cache.
	QueryCacheEntries int

	// QueryCacheFreshness is the recent-window guard for the results cache, in nanoseconds: a
	// query whose window ends within this of now is not cached (new samples may still land in
	// it), so only settled/historical windows are memoized. 0 ⇒ no guard (cache every eligible
	// request). Ignored when QueryCacheEntries is 0.
	QueryCacheFreshness int64 // nanoseconds

	// Observability is injected, never owned (DESIGN.md §16): the library logs, traces, and meters
	// through the embedder's handles and is a zero-overhead no-op when they are unset. The library
	// imports only the OTel API; the embedder owns the SDK, sampling, and exporters.
	//
	// Logger is the zap logger (nil ⇒ a no-op logger). TracerProvider supplies OTel spans (nil ⇒
	// the OTel noop tracer). MeterProvider supplies OTel metrics, including the mandatory admission
	// meta-metrics (nil ⇒ the OTel noop meter).
	Logger         *zap.Logger
	TracerProvider trace.TracerProvider
	MeterProvider  metric.MeterProvider

	// Retry tunes how the library survives unreliable transports: per-attempt timeouts, bounded
	// retries, and hedged (opportunistic concurrent) reads across cluster replicas. The zero value
	// selects [reliability.Default] (a mild, LAN-safe profile); set [reliability.LossyEnvironment]
	// (or a custom config) for networks that drop/stall/30s-timeout requests. It governs the cluster
	// RPCs; the S3 backend is configured independently where it is constructed.
	Retry reliability.RetryConfig
}

Options configures a Storage. Use [WithX] functional options to override defaults; Open applies them. The zero value is not valid — call Open (or InMemory) to construct a Storage.

type PartDetail added in v0.12.0 

type PartDetail struct {
	PartInfo

	Bytes   int64        // sum of the part's backend object sizes
	Chunks  int          // sparse-index granules
	Columns []ColumnInfo // per-column physical layout
}

PartDetail augments PartInfo with fields that require a backend read: the part's on-backend byte size and its column/codec layout and chunk (granule) count.

type PartInfo added in v0.12.0 

type PartInfo struct {
	ID      string // the part's backend key prefix
	MinTime int64  // inclusive unix-ns bounds of the part's data
	MaxTime int64
	Series  int   // distinct series/streams in the part
	Rows    int64 // total samples/records in the part
}

PartInfo is one flushed part's in-memory shape for a drill-down dashboard: identity, time bounds, and the series/row counts. It does no backend I/O — see Storage.Parts. For byte sizes, codecs, and chunk counts use Storage.PartsDetailed.

type RebalanceMove added in v0.10.0 

type RebalanceMove struct {
	Shard   string
	Added   []string
	Removed []string
}

RebalanceMove is one shard's primary handoff at a ring change: the node ids that gained it and those that lost it.

type SeriesAggregate added in v0.9.0 

type SeriesAggregate struct {
	engine.SeriesAgg

	Series signal.Series
}

SeriesAggregate pairs a series identity with its whole-range engine.SeriesAgg: the labeled form of Storage.AggregateMetrics. It is computed from the per-part stats sidecar where the plan is pushdown-safe, so a range-covering aggregate costs ~one row per series rather than a decode of every sample — and because the identity rides along, an embedder can render the result as a PromQL vector (labels + value) without a second, value-decoding fetch.

type SignalStats added in v0.10.0 

type SignalStats struct {
	Signal signal.Signal
	// Series is the distinct series/streams ever seen (index span: head ∪ flushed parts).
	Series int64
	// HeadItems is the samples (metrics) or records (logs/traces/profiles) buffered unflushed.
	HeadItems int64
	// HeadBytes is the head's buffered bytes — the in-flight memory measure a flush drains.
	HeadBytes int64
	// Parts is the number of flushed immutable parts (the compaction-backlog proxy: many small
	// parts means merge is behind).
	Parts int
	// MinTimeUnixNano / MaxTimeUnixNano bound the data this engine holds (0 when empty); MinTime is
	// over flushed parts, MaxTime includes the head's newest.
	MinTimeUnixNano int64
	MaxTimeUnixNano int64
	// MergeRunning is true while a compaction/merge is executing on this engine.
	MergeRunning bool
	// MergeBacklog is the count of flushed parts pending compaction — the backlog proxy (currently
	// equal to Parts; a separate field so a dashboard can label it as the merge backlog).
	MergeBacklog int
	// WAL is true when this engine has a write-ahead log (false for the ephemeral in-memory engine);
	// the WAL* fields below are meaningful only when it is true.
	WAL bool
	// WALSegments is the WAL's current segment sequence number (segments opened so far).
	WALSegments int
	// WALBytes is the byte size of the WAL's currently-open segment.
	WALBytes int64
	// WALEpoch is the WAL's active flush generation (the epoch stamped onto new segments). The same
	// generation across both engine families; not the recovery watermark.
	WALEpoch uint64
}

SignalStats is one (tenant, signal) engine's in-memory shape.

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage is the embeddable entry point (DESIGN.md §5). It is safe for concurrent use. Construct with Open or InMemory; never with a literal.

All ingestion is push-based and OTLP-shaped: methods accept the library's internal, []byte-based, zero-alloc ingest batches (e.g. [metric.Metrics]) and return an Accepted carrying per-OTLP partial-success counts. OTel-Go users convert pdata to these via the optional otlp/pdataconv bridge, which keeps pdata off this hot path. Reads go through the language-agnostic Storage.Fetcher seam; query languages live in the embedder.

All four signals are wired end-to-end: metrics (Storage.WriteMetrics/Storage.Fetcher) on the float-sample engine, and logs, traces, and profiles (Storage.WriteLogs/Storage.WriteTraces/ Storage.WriteProfiles and their fetchers) on the shared record engine.

func InMemory 

func InMemory(opts ...Option) (*Storage, error)

InMemory returns a fully in-memory, ephemeral Storage (DESIGN.md §5): equivalent to Open with backend.Memory and DurabilityEphemeral. It is the default in tests — a full ingest+query path with no disk or object store.

func Open 

func Open(ctx context.Context, o Options, opts ...Option) (*Storage, error)

Open constructs a Storage from Options (DESIGN.md §5). If Options.Backend is nil it defaults to backend.Memory; if the backend is ephemeral and no durability is chosen, it defaults to DurabilityEphemeral (the in-memory engine).

func (*Storage) Admin added in v0.10.0 

func (s *Storage) Admin() Admin

Admin returns the operator-control surface for on-demand maintenance.

func (*Storage) AdmissionStats added in v0.4.0 

func (s *Storage) AdmissionStats(tid signal.TenantID) AdmissionStats

AdmissionStats returns the cumulative admission counters for a tenant (the overload-control meta-metrics). It is safe to call concurrently; an unknown tenant returns the zero value.

func (*Storage) AggregateMetrics added in v0.6.0 

func (s *Storage) AggregateMetrics(ctx context.Context, t signal.TenantID, r fetch.Request) (map[signal.SeriesID]engine.SeriesAgg, error)

AggregateMetrics returns a per-series aggregate (count/sum/min/max — enough for avg) of one tenant's metric series over the request window. An embedder's PromQL engine can use it to evaluate `*_over_time` cheaply.

Single-node it is the storage-side **pushdown**: with WithAggregateStats a range-covering aggregate is answered from precomputed per-part stats without decoding the value column (one row per series instead of every sample), else by decoding. In **cluster** mode it gathers across the tenant's shards through the owner-aware read fan-out and folds at the coordinator — correct across all shards, though it transfers raw samples rather than pushing the aggregate to each node (a compact per-node aggregate RPC is a planned optimization).

func (*Storage) AggregateMetricsNamed added in v0.9.0 

func (s *Storage) AggregateMetricsNamed(ctx context.Context, t signal.TenantID, r fetch.Request) ([]SeriesAggregate, error)

AggregateMetricsNamed is the labeled form of Storage.AggregateMetrics: it returns each matching series' identity alongside its whole-range aggregate (count/sum/min/max — enough for avg). Use it when the caller needs to render the result with labels (e.g. an embedder's PromQL `*_over_time` pushdown); use Storage.AggregateMetrics when only the aggregate is needed. Series with no sample in the window are omitted.

func (*Storage) Cardinality added in v0.12.0 

func (s *Storage) Cardinality(tenant signal.TenantID, sig signal.Signal, topN int) CardinalityStats

Cardinality summarizes a (tenant, signal) engine's label cardinality. topN bounds the returned TopLabelNames slice (≤0 returns every label name). It does no backend I/O and returns a zero value when the tenant has no engine for the signal.

func (*Storage) Close 

func (s *Storage) Close(ctx context.Context) error

Close releases all resources. It is idempotent. After [Close], every method on s returns ErrClosed.

func (*Storage) Fetcher 

func (s *Storage) Fetcher(tenants ...signal.TenantID) fetch.Fetcher

Fetcher returns the read seam — a fetch.Fetcher over the named tenants' data (head ∪ flushed parts). It is the storage library's query surface: this is a language-agnostic columnar store, so the embedder drives its own query engines (PromQL/LogQL/TraceQL) over the fetch contract. The optional query/promql package bridges this to the Prometheus storage.Queryable for embedders using the Prometheus engine.

Tenant scoping:

  • Fetcher(t) — one tenant.
  • Fetcher(a, b, …) — a fan-out over several tenants (multi-tenant query): results are merged by series id, so a series with equal labels in more than one tenant is federated into one (see fetch.Merge). Add a tenant label upstream if per-tenant separation is wanted.
  • Fetcher() — all tenants that have ingested data (a cross-tenant query).

The returned Fetcher is always usable: with no matching tenant (or after [Close]) it is an empty fetcher that yields no series, so callers need not special-case "no data yet".

Scale-out: when Options.QuerySplitInterval and/or Options.QueryCacheEntries are set, the returned fetcher is wrapped with split-by-interval and/or a results cache (the query/scale decorators). The cache keys on the explicit tenant set, so it applies only to named-tenant queries — a no-arg cross-tenant query is never cached (its tenant membership is dynamic).

func (*Storage) Inspect added in v0.10.0 

func (s *Storage) Inspect() StoreStats

Inspect returns an in-memory snapshot of store state for a dashboard/CLI. It does no backend I/O and decodes nothing; it takes a brief read lock per engine to copy counters. Poll it at dashboard cadence (seconds), not per request.

func (*Storage) LogFetcher added in v0.2.0 

func (s *Storage) LogFetcher(tenants ...signal.TenantID) fetch.Fetcher

LogFetcher returns the read seam for logs — a fetch.Fetcher over the named tenants' log data (head ∪ flushed parts). Like Storage.Fetcher it scopes by tenant: one, several (concatenated), or none ⇒ all tenants with log data. Always usable: an empty fetcher when no tenant matches or after [Close]. Label matchers resolve streams; column Conditions filter records.

func (*Storage) LogKeys added in v0.5.0 

func (s *Storage) LogKeys(ctx context.Context, tenant signal.TenantID, start, end int64) ([]KeyInfo, error)

LogKeys returns the distinct attribute keys present in a tenant's log records within [start, end], each tagged with the scope(s) it appears in (resource, scope, record). A zero start AND end disables the time filter. It is the counterpart to Storage.LogSeries (which enumerates only stream identities): the per-record KeyScopeRecord keys let an embedder list and push down record-attribute labels its Storage.LogSeries-based resolution cannot see, and the scope bitset authoritatively distinguishes a stream label from a record attribute (or both). Keys are low-cardinality metadata. In cluster mode it serves locally when this node owns the tenant, else it fans out to an owner (hedged); each owner is a complete replica, so one response is authoritative.

func (*Storage) LogSeries added in v0.3.0 

func (s *Storage) LogSeries(
	ctx context.Context, tenant signal.TenantID, matchers []fetch.Matcher, start, end int64,
) ([]signal.Series, error)

LogSeries returns the identities of a tenant's log streams matching the label matchers within [start, end] (zero start AND end disables the time filter). It mirrors Storage.ProfileSeries for the logs vertical, so an embedder can build LabelNames/LabelValues/Series responses without materializing records. In cluster mode it serves locally when this node owns the tenant, else it fans out to an owner (hedged), re-applying the non-equality matchers to the superset.

func (*Storage) Parts added in v0.12.0 

func (s *Storage) Parts(tenant signal.TenantID, sig signal.Signal) []PartInfo

Parts returns an in-memory snapshot of a (tenant, signal) engine's flushed parts. It does no backend I/O and decodes nothing — safe to poll at dashboard cadence — and returns nil when the tenant has no engine for the signal. For byte sizes, codecs, and chunk counts, use Storage.PartsDetailed.

func (*Storage) PartsDetailed added in v0.12.0 

func (s *Storage) PartsDetailed(ctx context.Context, tenant signal.TenantID, sig signal.Signal) ([]PartDetail, error)

PartsDetailed augments Storage.Parts with each part's on-backend byte size, column/codec layout, and chunk count. It reads object sizes from the backend, so unlike Parts it is not hot-path-free — call it for a drill-down view, not a high-frequency poll. It returns nil (no error) when the tenant has no engine for the signal.

func (*Storage) ProfileFetcher added in v0.2.0 

func (s *Storage) ProfileFetcher(tenants ...signal.TenantID) fetch.Fetcher

ProfileFetcher returns the read seam for profiles — a fetch.Fetcher over the named tenants' sample data. Label matchers resolve streams (service plus the profile type, carried in reserved `otel.profile.*` labels); column Conditions filter samples (value, profile id, attributes). Returned rows carry the global content-addressed `stack_id`; resolve it with Storage.ProfileResolver. Same tenant scoping as Storage.TraceFetcher.

func (*Storage) ProfileResolver added in v0.2.0 

func (s *Storage) ProfileResolver(ctx context.Context, tenant signal.TenantID) (*profile.Resolver, error)

ProfileResolver returns a symbol resolver over a tenant's profile symbol store (the unioned head + part sidecars), so an embedder resolves the content-addressed `stack_id` column of a sample fetch to function frames and builds a flamegraph. An unknown tenant yields an empty resolver (every stack resolves to no frames), so callers need not special-case "no data". Resolution is correct across the cluster — `stack_id`s are global content ids — but reads the local node's symbol store; replicating the store itself is deferred (see `signal/profile`).

func (*Storage) ProfileSeries added in v0.2.0 

func (s *Storage) ProfileSeries(
	ctx context.Context, tenant signal.TenantID, matchers []fetch.Matcher, start, end int64,
) ([]signal.Series, error)

ProfileSeries returns the identities of a tenant's profile streams matching the label matchers within [start, end] (zero start AND end disables the time filter). It is the enumeration primitive an embedder uses to build the Pyroscope-style ProfileTypes / LabelNames / LabelValues responses: the profile type is carried in each series' reserved `otel.profile.*` labels (see `signal/profile`), and the user labels are the resource/scope attributes. Local to this node — cluster fan-out for enumeration is not yet wired (the sample read path already fans out).

func (*Storage) Reset 

func (s *Storage) Reset(ctx context.Context) error

Reset discards all ingested data, returning every tenant engine to empty (head and flushed parts) without reconstructing the Storage. It is intended for tests and benchmarks that reuse one store across runs instead of rebuilding it. Reset is only valid on an ephemeral (in-memory) backend; on a durable backend it returns ErrNotEphemeral rather than wipe persisted data. The tenant engines themselves are retained (emptied, not dropped), so subsequent writes reuse them.

func (*Storage) Trace added in v0.2.0 

func (s *Storage) Trace(ctx context.Context, tenant signal.TenantID, traceID []byte) ([]*fetch.Batch, error)

Trace fetches every span of one trace from a tenant by trace id: an equality condition on the trace_id column, pruned by its per-part equality bloom (trace-by-id). It returns one batch per stream (service) carrying the trace's spans — including the nested-set columns the embedder's TraceQL uses for structural operators.

func (*Storage) TraceFetcher added in v0.2.0 

func (s *Storage) TraceFetcher(tenants ...signal.TenantID) fetch.Fetcher

TraceFetcher returns the read seam for traces — a fetch.Fetcher over the named tenants' span data. Label matchers resolve streams (services); column Conditions filter spans (name, kind, status, duration, attributes). Same tenant scoping as Storage.LogFetcher.

func (*Storage) TraceSeries added in v0.3.0 

func (s *Storage) TraceSeries(
	ctx context.Context, tenant signal.TenantID, matchers []fetch.Matcher, start, end int64,
) ([]signal.Series, error)

TraceSeries returns the identities of a tenant's span streams matching the label matchers within [start, end] (zero start AND end disables the time filter). It mirrors Storage.ProfileSeries for the traces vertical, so an embedder can build tag-name/tag-value responses without materializing spans. In cluster mode it serves locally when this node owns the tenant, else it fans out to an owner (hedged), re-applying the non-equality matchers to the superset.

func (*Storage) WriteLogs 

func (s *Storage) WriteLogs(ctx context.Context, ld log.Logs) (acc Accepted, err error)

WriteLogs ingests a logs batch. It projects the internal model, derives each record's tenant from its Resource+Scope, and appends to that tenant's logs engine (indexing stream labels + buffering records). Returns per-OTLP partial-success counts: rejected counts out-of-order drops.

func (*Storage) WriteMetrics 

func (s *Storage) WriteMetrics(ctx context.Context, md metric.Metrics) (acc Accepted, err error)

WriteMetrics ingests a metrics batch. It projects the internal model, derives each point's tenant from its Resource+Scope, and appends to that tenant's engine (indexing labels + buffering samples). Returns per-OTLP partial-success counts: rejected counts out-of-order drops. (Unsupported point kinds and value-less points never reach here: they are filtered and counted by the producer — e.g. the otlp/pdataconv bridge.)

func (*Storage) WriteProfiles 

func (s *Storage) WriteProfiles(ctx context.Context, pd profile.Profiles) (acc Accepted, err error)

WriteProfiles ingests a profiles batch. It projects each sample into a record row (flattening timestamped samples and denormalizing profile fields) and a content-addressed symbol delta, derives each sample's tenant from its Resource+Scope, and appends to that tenant's profiles engine — which persists the symbol store as part sidecars. Returns OTLP partial-success counts.

func (*Storage) WriteTraces 

func (s *Storage) WriteTraces(ctx context.Context, td trace.Traces) (acc Accepted, err error)

WriteTraces ingests a traces batch. It projects the span model (computing nested-set ids and serializing events/links), derives each span's tenant from its Resource+Scope, and appends to that tenant's traces engine. Returns per-OTLP partial-success counts (out-of-order drops).

type StoreStats added in v0.10.0 

type StoreStats struct {
	// Tenants is one entry per tenant that has any engine, sorted by tenant id.
	Tenants []TenantStats
	// Cluster is the cluster-mode view (membership, owned shards, last rebalance); nil single-node.
	Cluster *ClusterStats
	// Caches aggregates the read-path caches.
	Caches CacheStats
}

StoreStats is a point-in-time, in-memory snapshot of store state for an embedder's CLI/UI dashboard (and for debugging). It is pull-based and does **no backend I/O and no column decode**, so it is safe to poll at dashboard cadence — each call takes only a brief per-engine read lock to copy counters, never touching the ingest/query hot path. On-disk part byte sizes are deliberately omitted (they would require backend stat calls); this is an in-memory view of head, part counts, time spans, admission, and cluster state.

type TenantStats added in v0.10.0 

type TenantStats struct {
	Tenant signal.TenantID
	// Admission is the tenant's cumulative admission tally (shared across signals — the valves are
	// keyed by tenant, not signal).
	Admission AdmissionStats
	// Signals is one entry per signal the tenant has ingested, sorted by signal.
	Signals []SignalStats
}

TenantStats is one tenant's per-signal breakdown plus its (cross-signal) admission counters.

type WALSyncMode added in v0.3.0 

type WALSyncMode uint8

WALSyncMode is the WAL fsync policy (the durability/throughput trade-off). It applies only when a WAL is configured (Options.WALDir on a durable backend). 

const (
	// WALSyncNone (default) lets WAL writes settle in the OS page cache: they survive a process
	// crash (recovery replays them) but not necessarily a power loss. Fastest.
	WALSyncNone WALSyncMode = iota
	// WALSyncAlways fsyncs after every WAL record — power-loss durable, slowest.
	WALSyncAlways
	// WALSyncInterval fsyncs every engine's WAL in the background every [Options.WALSyncInterval]:
	// a bounded power-loss window at a fraction of the cost of WALSyncAlways.
	WALSyncInterval
)

Source Files

View all Source files

Directories

Path Synopsis
Package backend defines the L1 storage seam (DESIGN.md §3, §5): a common Backend interface (Read/Write/List/Delete over whole-object keys) with interchangeable implementations.
 Package backend defines the L1 storage seam (DESIGN.md §3, §5): a common Backend interface (Read/Write/List/Delete over whole-object keys) with interchangeable implementations.
backendtest
Package backendtest provides a shared conformance suite that every backend.Backend implementation must pass, proving the implementations are interchangeable (DESIGN.md §2: "backends are interchangeable behind backend.Backend").
 Package backendtest provides a shared conformance suite that every backend.Backend implementation must pass, proving the implementations are interchangeable (DESIGN.md §2: "backends are interchangeable behind backend.Backend").
bucketindex
Package bucketindex maintains a compact, incremental index of the immutable parts under a key prefix, so a stateless reader enumerates a tenant's parts (and prunes them by time) from a single object instead of a full, expensive bucket LIST (DESIGN.md §11, the object-store-native read path).
 Package bucketindex maintains a compact, incremental index of the immutable parts under a key prefix, so a stateless reader enumerates a tenant's parts (and prunes them by time) from a single object instead of a full, expensive bucket LIST (DESIGN.md §11, the object-store-native read path).
file
Package file implements a backend.Backend over a local directory tree.
 Package file implements a backend.Backend over a local directory tree.
s3
Package s3 implements a backend.Backend over an S3-compatible object store.
 Package s3 implements a backend.Backend over an S3-compatible object store.
Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last.
 Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last.
Package cluster implements the L0 distribution layer (DESIGN.md §3, §11, §14 M6–M7): rendezvous (HRW) hashing with spread-minimizing tokens, etcd-backed ring state and leases, RF=3 quorum replication, and rebalancing.
 Package cluster implements the L0 distribution layer (DESIGN.md §3, §11, §14 M6–M7): rendezvous (HRW) hashing with spread-minimizing tokens, etcd-backed ring state and leases, RF=3 quorum replication, and rebalancing.
etcd
Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration.
 Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration.
rebalance
Package rebalance computes the minimal ownership changes to apply a cluster membership change (DESIGN.md §11).
 Package rebalance computes the minimal ownership changes to apply a cluster membership change (DESIGN.md §11).
replica
Package replica is the L0 write-replication layer: it fans an opaque write payload out to the ring-owners of a key and returns once a write **quorum** has durably applied it, so the unflushed head survives the loss of a minority of replicas (DESIGN.md §11; RF=3, quorum (RF/2)+1=2).
 Package replica is the L0 write-replication layer: it fans an opaque write payload out to the ring-owners of a key and returns once a write **quorum** has durably applied it, so the unflushed head survives the loss of a minority of replicas (DESIGN.md §11; RF=3, quorum (RF/2)+1=2).
ring
Package ring implements rendezvous (highest-random-weight, HRW) hashing — the L0 sharding primitive (DESIGN.md §11).
 Package ring implements rendezvous (highest-random-weight, HRW) hashing — the L0 sharding primitive (DESIGN.md §11).
Package encoding holds the L2 codec layers (DESIGN.md §3, §14 M0):
 Package encoding holds the L2 codec layers (DESIGN.md §3, §14 M0):
bitstream
Package bitstream implements a zero-alloc, MSB-first bit stream reader and writer.
 Package bitstream implements a zero-alloc, MSB-first bit stream reader and writer.
chunk
Package chunk implements the zero-alloc value-column codecs (DESIGN.md §14 M0): delta-of-delta timestamps, Gorilla XOR float gauges, T64, dictionary, and scaled-decimal+nearest-delta.
 Package chunk implements the zero-alloc value-column codecs (DESIGN.md §14 M0): delta-of-delta timestamps, Gorilla XOR float gauges, T64, dictionary, and scaled-decimal+nearest-delta.
compress
Package compress wraps zstd/lz4 block compression and defines the composable codec-chain type (preprocessor → general compressor) used to reach 0.4–0.8 bytes/point (DESIGN.md §3.3, §14 M0).
 Package compress wraps zstd/lz4 block compression and defines the composable codec-chain type (preprocessor → general compressor) used to reach 0.4–0.8 bytes/point (DESIGN.md §3.3, §14 M0).
Package engine is the L3 single-node storage engine: an in-memory head that absorbs writes (indexing labels and buffering samples) with a bounded out-of-order window, optionally backed by a write-ahead log for crash recovery.
 Package engine is the L3 single-node storage engine: an in-memory head that absorbs writes (indexing labels and buffering samples) with a bounded out-of-order window, optionally backed by a write-ahead log for crash recovery.
Package index implements the L3 indexing layer (DESIGN.md §3, §14 M2):
 Package index implements the L3 indexing layer (DESIGN.md §3, §14 M2):
bloom
Package bloom is a token bloom filter for full-text pruning: a compact, approximate set of the terms present in a column block (e.g.
 Package bloom is a token bloom filter for full-text pruning: a compact, approximate set of the terms present in a column block (e.g.
postings
Package postings is the inverted index: for each (name, value) attribute — identified by **interned symbol ids**, not strings — it keeps the sorted list of [signal.SeriesID]s that carry it, and composes those lists with lazy set-op iterators (Intersect/Merge/Without) to resolve label matchers to series.
 Package postings is the inverted index: for each (name, value) attribute — identified by **interned symbol ids**, not strings — it keeps the sorted list of [signal.SeriesID]s that carry it, and composes those lists with lazy set-op iterators (Intersect/Merge/Without) to resolve label matchers to series.
series
Package series is the series-identity index: it maps a content-addressed signal.SeriesID to its full identity (signal.Series — Resource + Scope + data-point attributes) and back.
 Package series is the series-identity index: it maps a content-addressed signal.SeriesID to its full identity (signal.Series — Resource + Scope + data-point attributes) and back.
symbols
Package symbols is the string-interning symbol table.
 Package symbols is the string-interning symbol table.
internal
cmd/benchreport command
Command benchreport turns `benchstat -format csv` (read from stdin) into a rich, GitHub-flavored Markdown report: a generalized verdict alert, per-metric tables with direction-aware deltas and status emoji, and an optional collapsible raw table.
 Command benchreport turns `benchstat -format csv` (read from stdin) into a rich, GitHub-flavored Markdown report: a generalized verdict alert, per-metric tables with direction-aware deltas and status emoji, and an optional collapsible raw table.
cmd/fuzz command
Command fuzz starts a fuzzing campaign for a randomly chosen fuzz target using gosentry (https://github.com/trailofbits/gosentry), Trail of Bits' Go-toolchain fork that runs `go test -fuzz` on top of LibAFL.
 Command fuzz starts a fuzzing campaign for a randomly chosen fuzz target using gosentry (https://github.com/trailofbits/gosentry), Trail of Bits' Go-toolchain fork that runs `go test -fuzz` on top of LibAFL.
cmd/gensimd command
Command gensimd generates the AVX2 assembly kernels in internal/simd via avo.
 Command gensimd generates the AVX2 assembly kernels in internal/simd via avo.
obs
Package obs bundles the library's injected observability — a zap logger, an OTel tracer, and the metric instruments — built once from the embedder's configuration and handed to each subsystem.
 Package obs bundles the library's injected observability — a zap logger, an OTel tracer, and the metric instruments — built once from the embedder's configuration and handed to each subsystem.
parallel
Package parallel provides bounded fan-out over an index range — the project's house pattern for running independent work (per-tenant maintenance, per-shard scatter-gather, WAL fsyncs) concurrently under a cap, without pulling in an errgroup dependency.
 Package parallel provides bounded fan-out over an index range — the project's house pattern for running independent work (per-tenant maintenance, per-shard scatter-gather, WAL fsyncs) concurrently under a cap, without pulling in an errgroup dependency.
retry
Package retry adds durability and tail-latency control to calls over unreliable transports (node-to-node RPCs, S3).
 Package retry adds durability and tail-latency control to calls over unreliable transports (node-to-node RPCs, S3).
simd
Package simd provides vectorized kernels for hot columnar loops, each with a portable pure-Go fallback selected at build time by architecture and, on amd64, at runtime by CPU feature.
 Package simd provides vectorized kernels for hot columnar loops, each with a portable pure-Go fallback selected at build time by architecture and, on amd64, at runtime by CPU feature.
otlp
pdataconv
Package pdataconv is the optional OTel-Go bridge: it converts the collector pdata metrics type (pmetric.Metrics) into the storage library's internal, []byte-based metric.Metrics ingest batch.
 Package pdataconv is the optional OTel-Go bridge: it converts the collector pdata metrics type (pmetric.Metrics) into the storage library's internal, []byte-based metric.Metrics ingest batch.
Package pool provides cross-cutting sync.Pool helpers, capacity-bucketed byte pools, and arena allocation for same-lifetime batches (DESIGN.md §10).
 Package pool provides cross-cutting sync.Pool helpers, capacity-bucketed byte pools, and arena allocation for same-lifetime batches (DESIGN.md §10).
Package query groups the storage library's read seam and its language adapters.
 Package query groups the storage library's read seam and its language adapters.
fetch
Package fetch is the storage seam: the contract every query language compiles to and every data source (head, parts, cluster fan-out) implements.
 Package fetch is the storage seam: the contract every query language compiles to and every data source (head, parts, cluster fan-out) implements.
profile
Package profile is the storage tier's EXPLAIN ANALYZE: a profiled tree of the fetch operators with per-node timing and I/O counters, showing where and how much time a query spent (which parts were scanned vs pruned, rows in/out, bytes decoded).
 Package profile is the storage tier's EXPLAIN ANALYZE: a profiled tree of the fetch operators with per-node timing and I/O counters, showing where and how much time a query spent (which parts were scanned vs pruned, rows in/out, bytes decoded).
promql
Package promql is an optional adapter that bridges the storage fetch contract (query/fetch) to the Prometheus storage.Queryable interface.
 Package promql is an optional adapter that bridges the storage fetch contract (query/fetch) to the Prometheus storage.Queryable interface.
scale
Package scale provides fetch-seam scale-out primitives: [Fetcher → Fetcher] decorators that any embedder's query engine composes over fetch.Fetcher (a per-tenant engine, a cluster fan-out, a cross-tenant fetch.Merge) without the library owning a query language.
 Package scale provides fetch-seam scale-out primitives: [Fetcher → Fetcher] decorators that any embedder's query engine composes over fetch.Fetcher (a per-tenant engine, a cluster fan-out, a cross-tenant fetch.Merge) without the library owning a query language.
Package recordengine is the shared storage engine for **record-shaped** signals — logs and traces (later, profiles' sample table).
 Package recordengine is the shared storage engine for **record-shaped** signals — logs and traces (later, profiles' sample table).
Package reliability holds the embedder-facing knobs for surviving unreliable transports — the node-to-node cluster RPCs and the S3 backend — in lossy, noisy environments where a request can fail immediately, hang and fail after a long timeout, or simply run slow.
 Package reliability holds the embedder-facing knobs for surviving unreliable transports — the node-to-node cluster RPCs and the S3 backend — in lossy, noisy environments where a request can fail immediately, hang and fail after a long timeout, or simply run slow.
Package signal holds the OTel data model, signal-neutral types shared across metrics, logs, traces, and profiles: the Signal enum, TenantID, and the typed attribute identity primitives (Value, KeyValue, Attributes, SeriesID).
 Package signal holds the OTel data model, signal-neutral types shared across metrics, logs, traces, and profiles: the Signal enum, TenantID, and the typed attribute identity primitives (Value, KeyValue, Attributes, SeriesID).
log
Package log holds the logs signal's ingest model: the []byte-based, OTLP-shaped, zero-alloc batch accepted at the storage boundary (in place of OTel-Go plog.Logs), and its projection into the columnar log-record model the logs engine ingests.
 Package log holds the logs signal's ingest model: the []byte-based, OTLP-shaped, zero-alloc batch accepted at the storage boundary (in place of OTel-Go plog.Logs), and its projection into the columnar log-record model the logs engine ingests.
metric
Package metric implements the metrics signal: OTLP metric point types, temporality, series identity, and the OTLP → internal projection.
 Package metric implements the metrics signal: OTLP metric point types, temporality, series identity, and the OTLP → internal projection.
profile
Package profile holds the profiles signal's ingest model: the []byte-based, OTLP-shaped batch accepted at the storage boundary (in place of OTel-Go pprofile.Profiles), and its projection into the columnar sample model the record engine ingests plus the content-addressed symbol store.
 Package profile holds the profiles signal's ingest model: the []byte-based, OTLP-shaped batch accepted at the storage boundary (in place of OTel-Go pprofile.Profiles), and its projection into the columnar sample model the record engine ingests plus the content-addressed symbol store.
trace
Package trace holds the traces signal's ingest model: the []byte-based, OTLP-shaped, zero-alloc batch accepted at the storage boundary (in place of OTel-Go ptrace.Traces), and its projection into the columnar span model the record engine ingests.
 Package trace holds the traces signal's ingest model: the []byte-based, OTLP-shaped, zero-alloc batch accepted at the storage boundary (in place of OTel-Go ptrace.Traces), and its projection into the columnar span model the record engine ingests.
Package tenant provides the cross-cutting policy-callback layer (DESIGN.md §2, §8): a tenant.Resolver maps a tenant id to limits, retention, downsampling, and routing, hot-reloadable by the consumer.
 Package tenant provides the cross-cutting policy-callback layer (DESIGN.md §2, §8): a tenant.Resolver maps a tenant id to limits, retention, downsampling, and routing, hot-reloadable by the consumer.
Package wal is the write-ahead log: CRC-framed records appended to numbered segment files, replayed in order to reconstruct the in-memory index after a crash.
 Package wal is the write-ahead log: CRC-framed records appended to numbered segment files, replayed in order to reconstruct the in-memory index after a crash.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.