storage

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

README

storage Go Reference codecov experimental

Low-level, distributed, OpenTelemetry-centric storage library (Go) for signals:

  • Metrics
  • Logs
  • Traces
  • Profiles

Built as the storage tier for oteldb, an OpenTelemetry observability backend — embeddable as a native Go storage engine for all signals.

Currently in WIP and PoC. See ARCHITECTURE.md for current design.

Documentation

Overview

Package storage is the embeddable entry point for the oteldb storage library.

It is a low-level, distributed, OpenTelemetry-centric storage engine for signals (metrics, logs, traces, profiles), exposed as a small Storage facade. Everything else is an implementation detail behind this package. See the package layout and milestone plan in DESIGN.md at the repository root.

The in-memory engine (InMemory) is the default in tests: a full ingest+query path with WAL and durable flush disabled.

Index

Constants

This section is empty.

Variables

View Source 
var ErrClosed = errors.New("storage: closed")

ErrClosed is returned by Storage methods after Storage.Close.

View Source 
var ErrNotEphemeral = errors.New("storage: reset requires an ephemeral backend")

ErrNotEphemeral is returned by Storage.Reset when the backend is durable: Reset wipes all ingested data and is only permitted on an ephemeral (in-memory) store.

View Source 
var ErrNotImplemented = errors.New("storage: not implemented in this milestone")

ErrNotImplemented is returned by scaffold-stub methods whose end-to-end wiring lands in a later milestone. It is not a fatal error: embedders may compile against the surface and gate on errors.Is(err, ErrNotImplemented).

Functions

This section is empty.

Types

type Accepted 

type Accepted struct {
	// Accepted is the number of accepted data points (metric points, log records,
	// span, profile samples depending on signal).
	Accepted int64
	// Rejected is the number of rejected data points (e.g. over a tenant limit or
	// out of the OOO window). Rejections are reported back to the producer via OTLP
	// partial_success so it can retry.
	Rejected int64
	// RejectedReason is a machine-readable reason for rejections (empty if none).
	RejectedReason string
}

Accepted carries per-OTLP partial-success counts (DESIGN.md §5). It implements the OTLP partial-success semantics: accepted/rejected data points + a reason string.

type Durability 

type Durability uint8

Durability is the WAL + flush policy (DESIGN.md §5, §8). 

const (
	// DurabilityDefault is the default durability for a durable backend (WAL on,
	// flush to backend). For the in-memory engine use [DurabilityEphemeral].
	DurabilityDefault Durability = iota
	// DurabilityEphemeral disables the WAL and durable flush: the head answers all
	// queries and "flushed" parts live in RAM, dropped on [Storage.Close]. This is
	// the in-memory engine and the default in tests (DESIGN.md §5).
	DurabilityEphemeral
)

type Option 

type Option func(*Options)

Option is a functional option applied to Options by Open/InMemory.

func WithBackend 

func WithBackend(b backend.Backend) Option

WithBackend sets the storage backend.

func WithCluster 

func WithCluster(c *cluster.Config) Option

WithCluster sets the cluster configuration. nil ⇒ single-node.

func WithDurability 

func WithDurability(d Durability) Option

WithDurability sets the WAL + flush policy.

func WithEncoding 

func WithEncoding(p encoding.Profile) Option

WithEncoding sets the default encoding profile.

func WithFlushInterval 

func WithFlushInterval(ns int64) Option

WithFlushInterval sets the head flush time interval in nanoseconds.

func WithFlushThresholdBytes 

func WithFlushThresholdBytes(n int64) Option

WithFlushThresholdBytes sets the head flush size threshold.

func WithOOOWindow 

func WithOOOWindow(ns int64) Option

WithOOOWindow sets the out-of-order ingestion window in nanoseconds.

func WithTenancy 

func WithTenancy(r tenant.Resolver) Option

WithTenancy sets the tenant policy resolver.

func WithTenant 

func WithTenant(fn func(signal.Resource, signal.Scope) signal.TenantID) Option

WithTenant sets the record→tenant routing callback (resource/scope → tenant id).

func WithWALDir 

func WithWALDir(dir string) Option

WithWALDir sets the WAL directory (file backend + durability).

type Options 

type Options struct {
	// Backend is the storage backend (memory/file/s3). If nil, [Open] defaults to
	// [backend.Memory] — the in-memory, ephemeral engine (DESIGN.md §5).
	Backend backend.Backend

	// Cluster is the cluster configuration. nil ⇒ single-node (the cluster layer
	// is absent, DESIGN.md §3, §11). Single-node users ignore L0 entirely.
	Cluster *cluster.Config

	// Tenant derives a record's tenant id from its Resource and Scope (so one OTLP
	// batch may fan out to many tenants). If nil, every record routes to "default".
	Tenant func(signal.Resource, signal.Scope) signal.TenantID

	// Tenancy resolves a tenant id to limits, retention, downsampling, and routing.
	// If nil, a permissive default resolver is used.
	Tenancy tenant.Resolver

	// Encoding is the default codec-chain profile per column kind (DESIGN.md §6).
	// If nil, a sensible default is used.
	Encoding encoding.Profile

	// Durability is the WAL + flush policy. [DurabilityEphemeral] disables both —
	// the in-memory engine: the head answers all queries and "flushed" parts are
	// kept in RAM and dropped on [Storage.Close] (DESIGN.md §5).
	Durability Durability

	// WALDir is the WAL directory for the file backend with durability enabled.
	// Ignored when [Durability] is [DurabilityEphemeral].
	WALDir string

	// FlushThresholdBytes is the head size at which a flush to an immutable part is
	// triggered. Zero ⇒ default.
	FlushThresholdBytes int64

	// FlushInterval is the max age of unflushed head data. Zero ⇒ default.
	// (Resolved into a duration internally to keep the API import-light.)
	FlushInterval int64 // nanoseconds

	// OOOWindow is the out-of-order ingestion window in nanoseconds (DESIGN.md §8).
	// Samples older than (newest - OOOWindow) are rejected (counted). Zero ⇒ default.
	OOOWindow int64 // nanoseconds
}

Options configures a Storage. Use [WithX] functional options to override defaults; Open applies them. The zero value is not valid — call Open (or InMemory) to construct a Storage.

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage is the embeddable entry point (DESIGN.md §5). It is safe for concurrent use. Construct with Open or InMemory; never with a literal.

All ingestion is push-based and OTLP-shaped: methods accept the library's internal, []byte-based, zero-alloc ingest batches (e.g. metric.Metrics) and return an Accepted carrying per-OTLP partial-success counts. OTel-Go users convert pdata to these via the optional otlp/pdataconv bridge, which keeps pdata off this hot path. Reads go through the language-agnostic Storage.Fetcher seam; query languages live in the embedder.

This is a scaffold stub: ingestion and query are wired end-to-end at M3 (metrics vertical). The methods currently validate arguments and return ErrNotImplemented so the surface is stable for embedders to compile against.

func InMemory 

func InMemory(opts ...Option) (*Storage, error)

InMemory returns a fully in-memory, ephemeral Storage (DESIGN.md §5): equivalent to Open with backend.Memory and DurabilityEphemeral. It is the default in tests — a full ingest+query path with no disk or object store.

func Open 

func Open(ctx context.Context, o Options, opts ...Option) (*Storage, error)

Open constructs a Storage from Options (DESIGN.md §5). If Options.Backend is nil it defaults to backend.Memory; if the backend is ephemeral and no durability is chosen, it defaults to DurabilityEphemeral (the in-memory engine).

func (*Storage) Close 

func (s *Storage) Close(ctx context.Context) error

Close releases all resources. It is idempotent. After [Close], every method on s returns ErrClosed.

func (*Storage) Fetcher 

func (s *Storage) Fetcher(tenants ...signal.TenantID) fetch.Fetcher

Fetcher returns the read seam — a fetch.Fetcher over the named tenants' data (head ∪ flushed parts). It is the storage library's query surface: this is a language-agnostic columnar store, so the embedder drives its own query engines (PromQL/LogQL/TraceQL) over the fetch contract. The optional query/promql package bridges this to the Prometheus storage.Queryable for embedders using the Prometheus engine.

Tenant scoping:

  • Fetcher(t) — one tenant.
  • Fetcher(a, b, …) — a fan-out over several tenants (multi-tenant query): results are merged by series id, so a series with equal labels in more than one tenant is federated into one (see fetch.Merge). Add a tenant label upstream if per-tenant separation is wanted.
  • Fetcher() — all tenants that have ingested data (a cross-tenant query).

The returned Fetcher is always usable: with no matching tenant (or after [Close]) it is an empty fetcher that yields no series, so callers need not special-case "no data yet".

func (*Storage) Reset 

func (s *Storage) Reset(ctx context.Context) error

Reset discards all ingested data, returning every tenant engine to empty (head and flushed parts) without reconstructing the Storage. It is intended for tests and benchmarks that reuse one store across runs instead of rebuilding it. Reset is only valid on an ephemeral (in-memory) backend; on a durable backend it returns ErrNotEphemeral rather than wipe persisted data. The tenant engines themselves are retained (emptied, not dropped), so subsequent writes reuse them.

func (*Storage) WriteLogs 

func (s *Storage) WriteLogs(_ context.Context, _ log.Logs) (Accepted, error)

WriteLogs ingests a logs batch. Later vertical.

func (*Storage) WriteMetrics 

func (s *Storage) WriteMetrics(ctx context.Context, md metric.Metrics) (Accepted, error)

WriteMetrics ingests a metrics batch. It projects the internal model, derives each point's tenant from its Resource+Scope, and appends to that tenant's engine (indexing labels + buffering samples). Returns per-OTLP partial-success counts: rejected counts out-of-order drops. (Unsupported point kinds and value-less points never reach here: they are filtered and counted by the producer — e.g. the otlp/pdataconv bridge.)

func (*Storage) WriteProfiles 

func (s *Storage) WriteProfiles(_ context.Context, _ profile.Profiles) (Accepted, error)

WriteProfiles ingests a profiles batch. Later vertical.

func (*Storage) WriteTraces 

func (s *Storage) WriteTraces(_ context.Context, _ trace.Traces) (Accepted, error)

WriteTraces ingests a traces batch. Later vertical.

Source Files

View all Source files

Directories

Path Synopsis
Package backend defines the L1 storage seam (DESIGN.md §3, §5): a common Backend interface (Read/Write/List/Delete over whole-object keys) with interchangeable implementations.
 Package backend defines the L1 storage seam (DESIGN.md §3, §5): a common Backend interface (Read/Write/List/Delete over whole-object keys) with interchangeable implementations.
backendtest
Package backendtest provides a shared conformance suite that every backend.Backend implementation must pass, proving the implementations are interchangeable (DESIGN.md §2: "backends are interchangeable behind backend.Backend").
 Package backendtest provides a shared conformance suite that every backend.Backend implementation must pass, proving the implementations are interchangeable (DESIGN.md §2: "backends are interchangeable behind backend.Backend").
bucketindex
Package bucketindex maintains a compact, incremental index of the immutable parts under a key prefix, so a stateless reader enumerates a tenant's parts (and prunes them by time) from a single object instead of a full, expensive bucket LIST (DESIGN.md §11, the object-store-native read path).
 Package bucketindex maintains a compact, incremental index of the immutable parts under a key prefix, so a stateless reader enumerates a tenant's parts (and prunes them by time) from a single object instead of a full, expensive bucket LIST (DESIGN.md §11, the object-store-native read path).
file
Package file implements a backend.Backend over a local directory tree.
 Package file implements a backend.Backend over a local directory tree.
s3
Package s3 implements a backend.Backend over an S3-compatible object store.
 Package s3 implements a backend.Backend over an S3-compatible object store.
Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last.
 Package block implements the L2 immutable columnar part format (DESIGN.md §3, §14 M1): per-column streams with min/max stats and constant-column collapse, a sparse granule mark index, and an atomic manifest written last.
Package cluster implements the L0 distribution layer (DESIGN.md §3, §11, §14 M6–M7): rendezvous (HRW) hashing with spread-minimizing tokens, etcd-backed ring state and leases, RF=3 quorum replication, and rebalancing.
 Package cluster implements the L0 distribution layer (DESIGN.md §3, §11, §14 M6–M7): rendezvous (HRW) hashing with spread-minimizing tokens, etcd-backed ring state and leases, RF=3 quorum replication, and rebalancing.
etcd
Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration.
 Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration.
rebalance
Package rebalance computes the minimal ownership changes to apply a cluster membership change (DESIGN.md §11).
 Package rebalance computes the minimal ownership changes to apply a cluster membership change (DESIGN.md §11).
replica
Package replica is the L0 write-replication layer: it fans an opaque write payload out to the ring-owners of a key and returns once a write **quorum** has durably applied it, so the unflushed head survives the loss of a minority of replicas (DESIGN.md §11; RF=3, quorum (RF/2)+1=2).
 Package replica is the L0 write-replication layer: it fans an opaque write payload out to the ring-owners of a key and returns once a write **quorum** has durably applied it, so the unflushed head survives the loss of a minority of replicas (DESIGN.md §11; RF=3, quorum (RF/2)+1=2).
ring
Package ring implements rendezvous (highest-random-weight, HRW) hashing — the L0 sharding primitive (DESIGN.md §11).
 Package ring implements rendezvous (highest-random-weight, HRW) hashing — the L0 sharding primitive (DESIGN.md §11).
Package encoding holds the L2 codec layers (DESIGN.md §3, §14 M0):
 Package encoding holds the L2 codec layers (DESIGN.md §3, §14 M0):
bitstream
Package bitstream implements a zero-alloc, MSB-first bit stream reader and writer.
 Package bitstream implements a zero-alloc, MSB-first bit stream reader and writer.
chunk
Package chunk implements the zero-alloc value-column codecs (DESIGN.md §14 M0): delta-of-delta timestamps, Gorilla XOR float gauges, T64, dictionary, and scaled-decimal+nearest-delta.
 Package chunk implements the zero-alloc value-column codecs (DESIGN.md §14 M0): delta-of-delta timestamps, Gorilla XOR float gauges, T64, dictionary, and scaled-decimal+nearest-delta.
compress
Package compress wraps zstd/lz4 block compression and defines the composable codec-chain type (preprocessor → general compressor) used to reach 0.4–0.8 bytes/point (DESIGN.md §3.3, §14 M0).
 Package compress wraps zstd/lz4 block compression and defines the composable codec-chain type (preprocessor → general compressor) used to reach 0.4–0.8 bytes/point (DESIGN.md §3.3, §14 M0).
Package engine is the L3 single-node storage engine: an in-memory head that absorbs writes (indexing labels and buffering samples) with a bounded out-of-order window, optionally backed by a write-ahead log for crash recovery.
 Package engine is the L3 single-node storage engine: an in-memory head that absorbs writes (indexing labels and buffering samples) with a bounded out-of-order window, optionally backed by a write-ahead log for crash recovery.
Package index implements the L3 indexing layer (DESIGN.md §3, §14 M2):
 Package index implements the L3 indexing layer (DESIGN.md §3, §14 M2):
postings
Package postings is the inverted index: for each (name, value) attribute — identified by **interned symbol ids**, not strings — it keeps the sorted list of [signal.SeriesID]s that carry it, and composes those lists with lazy set-op iterators (Intersect/Merge/Without) to resolve label matchers to series.
 Package postings is the inverted index: for each (name, value) attribute — identified by **interned symbol ids**, not strings — it keeps the sorted list of [signal.SeriesID]s that carry it, and composes those lists with lazy set-op iterators (Intersect/Merge/Without) to resolve label matchers to series.
series
Package series is the series-identity index: it maps a content-addressed signal.SeriesID to its full identity (signal.Series — Resource + Scope + data-point attributes) and back.
 Package series is the series-identity index: it maps a content-addressed signal.SeriesID to its full identity (signal.Series — Resource + Scope + data-point attributes) and back.
symbols
Package symbols is the string-interning symbol table.
 Package symbols is the string-interning symbol table.
internal
cmd/fuzz command
Command fuzz starts a fuzzing campaign for a randomly chosen fuzz target using gosentry (https://github.com/trailofbits/gosentry), Trail of Bits' Go-toolchain fork that runs `go test -fuzz` on top of LibAFL.
 Command fuzz starts a fuzzing campaign for a randomly chosen fuzz target using gosentry (https://github.com/trailofbits/gosentry), Trail of Bits' Go-toolchain fork that runs `go test -fuzz` on top of LibAFL.
otlp
pdataconv
Package pdataconv is the optional OTel-Go bridge: it converts the collector pdata metrics type (pmetric.Metrics) into the storage library's internal, []byte-based metric.Metrics ingest batch.
 Package pdataconv is the optional OTel-Go bridge: it converts the collector pdata metrics type (pmetric.Metrics) into the storage library's internal, []byte-based metric.Metrics ingest batch.
Package pool provides cross-cutting sync.Pool helpers, capacity-bucketed byte pools, and arena allocation for same-lifetime batches (DESIGN.md §10).
 Package pool provides cross-cutting sync.Pool helpers, capacity-bucketed byte pools, and arena allocation for same-lifetime batches (DESIGN.md §10).
Package query groups the storage library's read seam and its language adapters.
 Package query groups the storage library's read seam and its language adapters.
fetch
Package fetch is the storage seam: the contract every query language compiles to and every data source (head, parts, cluster fan-out) implements.
 Package fetch is the storage seam: the contract every query language compiles to and every data source (head, parts, cluster fan-out) implements.
promql
Package promql is an optional adapter that bridges the storage fetch contract (query/fetch) to the Prometheus storage.Queryable interface.
 Package promql is an optional adapter that bridges the storage fetch contract (query/fetch) to the Prometheus storage.Queryable interface.
Package signal holds the OTel data model, signal-neutral types shared across metrics, logs, traces, and profiles: the Signal enum, TenantID, and the typed attribute identity primitives (Value, KeyValue, Attributes, SeriesID).
 Package signal holds the OTel data model, signal-neutral types shared across metrics, logs, traces, and profiles: the Signal enum, TenantID, and the typed attribute identity primitives (Value, KeyValue, Attributes, SeriesID).
log
Package log holds the logs signal's ingest model.
 Package log holds the logs signal's ingest model.
metric
Package metric implements the metrics signal: OTLP metric point types, temporality, series identity, and the OTLP → internal projection.
 Package metric implements the metrics signal: OTLP metric point types, temporality, series identity, and the OTLP → internal projection.
profile
Package profile holds the profiles signal's ingest model.
 Package profile holds the profiles signal's ingest model.
trace
Package trace holds the traces signal's ingest model.
 Package trace holds the traces signal's ingest model.
Package tenant provides the cross-cutting policy-callback layer (DESIGN.md §2, §8): a tenant.Resolver maps a tenant id to limits, retention, downsampling, and routing, hot-reloadable by the consumer.
 Package tenant provides the cross-cutting policy-callback layer (DESIGN.md §2, §8): a tenant.Resolver maps a tenant id to limits, retention, downsampling, and routing, hot-reloadable by the consumer.
Package wal is the write-ahead log: CRC-framed records appended to numbered segment files, replayed in order to reconstruct the in-memory index after a crash.
 Package wal is the write-ahead log: CRC-framed records appended to numbered segment files, replayed in order to reconstruct the in-memory index after a crash.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.