imbhgo

package module
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 9, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

README

imbh-go

A Go binding for IMBH — an embeddable, Rust observability database (logs, traces, metrics on Apache DataFusion) — with zero-copy Apache Arrow query results, fused onto Go's scheduler via sable.

You open an IMBH database from Go, feed it OpenTelemetry (OTLP) data, and query it with SQL or typed observability queries. Query results cross the language boundary as Arrow record batches without a copy: Go holds IMBH's Arc-refcounted Arrow buffers by pointer and releases them when done.

db, _ := imbhgo.OpenInMemory()
defer db.Close()
ctx := context.Background()

db.IngestOTLPLogs(otlpBytes)                 // what any OTLP/HTTP exporter sends

logs, _ := db.QueryLogsTyped(ctx, imbhgo.LogQuery{Service: "checkout", Match: "error"})
for _, e := range logs {
    fmt.Println(e.Time, e.Service, e.Body)
}

Status: early but working. The full path — open → ingest → SQL, typed queries, and the LGTM query languages (PromQL / LogQL / TraceQL), all zero-copy → errors → cancellation → backpressure — is implemented and tested (83 tests under -race, including on-disk durability, concurrency-under-load, and the admin / lifecycle / cursor-paging surface), and the memory-ownership protocol is leak-verified two independent ways (an in-process counter and a Valgrind gate). Dependencies are now external — imbh from crates.io and sable git-pinned — so it builds without local checkouts, and CI (.github/workflows/ci.yml) runs the whole gate on linux/amd64 and linux/arm64 for every push and pull request. See Feature coverage for what is and isn't exposed, and Limitations.

How it works

flowchart LR
    Go["Go program<br/>(imbhgo package)"]

    subgraph lib["libimbhgo.a — one static library"]
        direction LR
        Sable["sable<br/>Rust ⇄ Go runtime fusion"]
        Imbh["imbh::Db<br/>DataFusion + Tantivy"]
    end

    Go -->|"OTLP ingest (bytes)"| Sable
    Go -->|"SQL / typed queries"| Sable
    Sable --> Imbh
    Imbh -.->|"zero-copy Arrow batches<br/>(C Data Interface)"| Go
  • Everything links into one combined static library (libimbhgo.a) containing IMBH, sable's runtime, and the binding's handlers.
  • Queries stream one Arrow batch per await: the goroutine parks between batches while IMBH's engine runs, so no OS thread is blocked and the query executes lazily (bounded memory).
  • Each result batch crosses via the Arrow C Data Interface (FFI_ArrowArray), imported by arrow-go/cdata — no serialization, no GoBytes copy.

The full design is in ARCHITECTURE.md.

Requirements

  • Go 1.26.4 (pinned — sable reaches Go's internal ABI via //go:linkname). go.mod enforces this toolchain, so Go downloads it automatically.
  • cgo (a C toolchain) to link the static library into your binary.
  • The github.com/apache/arrow-go/v18 module (for reading result batches).
  • Rust (stable, 1.96+) is needed only if you build the library from source. Consumers using a prebuilt archive (below) do not need Rust.

Prebuilt libimbhgo.a archives are published per platform (see Using a prebuilt library).

Using a prebuilt library (no Rust build)

You can depend on the binding without ever building the Rust side. cmd/imbhgo-fetch downloads the prebuilt libimbhgo.a for your platform from the matching GitHub release, verifies its checksum, and prints the CGO_LDFLAGS to link against it:

go get github.com/moriyoshi/imbh-go@v0.8.0
eval "$(go run github.com/moriyoshi/imbh-go/cmd/imbhgo-fetch@v0.8.0 -print-env)"
go build -tags sable_extern_lib ./...

  • The sable_extern_lib build tag is required — it makes sable's Go package link the combined archive rather than its own libsable.a.

  • -print-env emits POSIX shell syntax on every platform, including Windows — the shell that consumes the line is the one you invoked the tool from, and on Windows CI that is normally git-bash / MSYS2. From a native Windows shell, ask for its dialect explicitly:

    for /f "delims=" %i in ('go run github.com/moriyoshi/imbh-go/cmd/imbhgo-fetch@v0.8.0 -print-env -shell cmd') do @%i

    go run github.com/moriyoshi/imbh-go/cmd/imbhgo-fetch@v0.8.0 -print-env -shell powershell | Invoke-Expression

  • The fetch tool caches the archive under your user cache dir (override with -dest) and re-detects glibc vs musl on Linux automatically (override with -libc).

  • Prebuilt cells: Linux amd64/arm64 (glibc and musl), macOS amd64/arm64, and Windows amd64 (best-effort). Each archive is pinned to the Go toolchain in go.mod.

Building from source

To build the static library yourself (co-development, or a platform with no prebuilt), build the Rust side first; the Go package links it.

make            # cargo build --release in rust/  → rust/target/release/libimbhgo.a
make test       # go test -tags sable_extern_lib -race ./...

Then compile your program with the sable_extern_lib build tag:

go build -tags sable_extern_lib ./...

(The per-platform cgo linker flags live in link_linux.go / link_darwin.go / link_windows.go; they default to rust/target/release for a local build, and defer to CGO_LDFLAGS for a prebuilt.)

Cutting a release (maintainers)
make release VERSION=v0.1.2   # rewrite the version everywhere, then build + vet + race-test

That rewrites internal/release.Version — which the release workflow requires the tag to match — along with the version references in this README and in cmd/imbhgo-fetch, then runs the local gate. It stops there and prints the remaining steps: commit, merge to main, then, from an up-to-date main:

make tag                 # creates the signed tag named by internal/release.Version
git push origin v0.1.2   # this publishes

Pushing the tag is what triggers .github/workflows/release.yml, which cross-builds every prebuilt cell, attaches the archives plus SHA256SUMS to the release, and smoke-tests the result as a consumer would.

A pushed tag is permanent. proxy.golang.org caches the tag → commit mapping immutably as soon as anyone resolves it, whether or not the release workflow succeeded, so a bad tag cannot be moved — only abandoned for the next number. Two guards keep the tag from ever disagreeing with the tree: make check-version (run by CI on every push and PR) rejects a half-applied bump or a version that has gone backwards past a published tag, and make tag takes the tag name from the constant rather than from an argument, so there is nothing to mistype.

Quick start

A complete runnable tour lives in examples/quickstart — it ingests OTLP logs and metrics and queries them three ways:

make            # build the static library
make example    # go run -tags sable_extern_lib ./examples/quickstart

The package initializes the runtime automatically on first use — just import and go.

package main

import (
	"context"
	"fmt"

	imbhgo "github.com/moriyoshi/imbh-go"
)

func main() {
	db, err := imbhgo.OpenInMemory() // or imbhgo.Open("/path/to/data")
	if err != nil {
		panic(err)
	}
	defer db.Close()
	ctx := context.Background()

	// Ingest OTLP/HTTP export-request protobuf bytes (from any OTel SDK exporter).
	if _, err := db.IngestOTLPLogs(otlpLogBytes); err != nil {
		panic(err)
	}

	// Query with SQL — results stream as zero-copy Arrow batches.
	rows, err := db.Query(ctx, "SELECT service, count(*) AS n FROM logs GROUP BY service")
	if err != nil {
		panic(err)
	}
	defer rows.Close()
	for {
		rec, ok, err := rows.Next()
		if err != nil {
			panic(err)
		}
		if !ok {
			break
		}
		fmt.Println(rec) // an arrow.RecordBatch
		rec.Release()    // return the buffers when done
	}
}

Querying

SQL (lazy, zero-copy)

Query streams the result batch-by-batch through IMBH's lazy scan. Tables: logs, spans, metrics_gauge, metrics_sum, metrics_histogram, and more. IMBH's UDFs are available in SQL — including histogram_quantile, matches (full-text), and json_get_str:

rows, _ := db.Query(ctx,
    "SELECT histogram_quantile(0.95, explicit_bounds, bucket_counts) AS p95 " +
        "FROM metrics_histogram WHERE metric = 'http.server.duration'")

Query returns *Rows; iterate with Next() (arrow.RecordBatch, ok bool, err error) until ok is false, then check Err(). Every query takes a context.Context as its first argument — cancel a running query by cancelling that ctx, and a parked Next is interrupted and reports context.Canceled from Err().

Typed queries

Endpoint-shaped queries as native Go structs. Each has a raw form (returns *Rows of Arrow batches) and a decoded form (returns Go structs).

// Logs → []LogEntry
entries, _ := db.QueryLogsTyped(ctx, imbhgo.LogQuery{
    Service: "checkout",
    Match:   "timeout",       // full-text on the body
    Limit:   100,
})

// Metric range → a Matrix (one series per group-by label set)
m, _ := db.QueryMetricsTyped(ctx, imbhgo.MetricQuery{
    Metric:  "cpu.utilization",
    Step:    int64(time.Second),
    Start:   start.UnixNano(),
    End:     end.UnixNano(),
    GroupBy: []string{"host"},
})
for _, s := range m.Series {
    fmt.Println(s.Labels, len(s.Points))
}

// Span RED metrics → []SpanMetricPoint (calls / errors / p50 / p95 / p99)
red, _ := db.QuerySpanMetricsTyped(ctx, imbhgo.SpanMetricsQuery{
    Service: "checkout",
    Step:    int64(time.Minute),
    Start:   start.UnixNano(),
    End:     end.UnixNano(),
})

Typed queries collect eagerly (they materialize the result), so prefer them for bounded results (a limited page, a fixed-step range) and use Query(ctx, sql) for large, unbounded scans.

Every GroupBy / attribute predicate (LogVolumeBy, MetricQuery.GroupBy, SpanMetricsQuery.GroupBy, LogQuery.AttrEq and friends) also accepts "service.name" — or its column spelling "service" — even though service is a resource attribute lifted into a built-in column rather than a record attribute. Requires imbh 0.3.0: earlier versions resolved the key against the record attributes, silently collapsing a per-service breakdown into one empty-labelled series with the counts merged.

Tailing logs on the arrival clock

Every record carries two instants: Time (when the event happened) and ObservedTime (when ingest received it). A follow loop must watermark on the second. Event time is not monotone in arrival — a record can be emitted before one already delivered and still land after it, whether because ingest batches or because the line's own timestamp was trusted — so a loop that advances a Start bound by the newest event time it has seen silently drops those records. Ordering by arrival cannot be overtaken that way:

var watermark int64 // last arrival instant delivered
for {
    entries, err := db.QueryLogsTyped(ctx, imbhgo.LogQuery{
        Service:       "checkout",
        ObservedAfter: watermark,                 // strict: observed_time > watermark
        OrderBy:       imbhgo.LogOrderObservedTime,
        Limit:         500,
    })
    if err != nil {
        return err
    }
    for _, e := range entries {
        handle(e)
        if e.ObservedTime > watermark {
            watermark = e.ObservedTime
        }
    }
    time.Sleep(pollInterval)
}

ObservedTime is nullable — an OTLP producer need not send one, and LogEntry.ObservedTime is then 0. Such records sort last in either direction and never match ObservedAfter (SQL NULL > t is unknown), so a record that cannot be placed against a watermark is left out of the loop rather than replayed on every poll. OrderBy is orthogonal to Backward, which picks the direction along the chosen axis; the default is LogOrderTime, and an unrecognized value is rejected rather than silently falling back. Requires imbh 0.6.0.

LGTM query languages (PromQL / LogQL / TraceQL)

IMBH implements the Grafana-stack query languages as explicitly-versioned compatibility profiles, and the binding exposes them over the same zero-copy Arrow path. Constructs outside a profile are rejected with a clear diagnostic rather than silently approximated.

// PromQL → labeled series. Metric names resolve against the stored catalog
// (OTel dots map to Prometheus underscores: cpu.util → cpu_util).
series, _ := db.QueryPromQLSeries(ctx, "rate(http_requests_total[5m])", start, end, step)
for _, s := range series {
    fmt.Println(s.Labels, len(s.Points))
}

// LogQL has two result shapes, as in Loki:
lines,  _ := db.QueryLogQLLines(ctx, `{service="checkout"} |= "error"`, start, end, 100) // streams
counts, _ := db.QueryLogQLSeries(ctx, `count_over_time({service="checkout"}[5m])`, start, end, step) // matrix

// TraceQL → matching traces + the span ids its spanset selected.
// Note attribute scope: service.name is a *resource* attribute.
matches, _ := db.QueryTraceQLMatches(ctx, `{ resource.service.name = "checkout" }`, start, end)

// …and follow a match through to that trace's spans (zero-copy).
spans, _ := db.GetTraceSpans(ctx, matches[0].TraceID)

Each also has a raw *Rows form (QueryPromQL, QueryLogQL, QueryTraceQL, GetTrace). Like every query method, all take a context.Context first argument for cancellation.

Ingesting

IngestOTLPLogs/IngestOTLPTraces/IngestOTLPMetrics take the protobuf bytes an OTLP/HTTP exporter sends (an ExportLogsServiceRequest, etc.) and return a Receipt:

r, _ := db.IngestOTLPLogs(otlpBytes)
fmt.Println(r.Accepted, r.Rejected, r.Durable)

Data is queryable immediately, before any Flush(). Flush() seals the in-memory buffer into an on-disk segment (to bound memory).

Close() runs IMBH's clean shutdown — drain the async-ingest worker, join the background maintenance thread, a final seal, commit_pending — and returns the error that shutdown reported. The handle is released either way, so Close() is never retryable and a second call is a no-op returning nil. An error means the on-disk state is not the tidy post-shutdown one; it does not mean rows were lost. Whatever the final seal could not write stays in the WAL and replays on the next Open — the same state a crash would have left. defer db.Close() stays valid where you do not need to check.

Backpressure

By default ingest is unbounded and never refused. To cap concurrency, set a global in-flight limit and use the Try* variants, which return ErrBackpressure instead of piling on:

imbhgo.SetMaxInFlight(64)
if _, err := db.TryIngestOTLPLogs(otlpBytes); errors.Is(err, imbhgo.ErrBackpressure) {
    // shed load or retry with backoff
}
stats := imbhgo.RuntimeStats() // InFlight, Rejected, MaxInFlight, …

The cap is process-global and also bounds concurrent open result streams (an open Rows holds a slot until Close).

Zero-copy: lifetime rules (important)

Result batches wrap IMBH-owned Arrow buffers that are freed when you Release() the batch:

  • Always Close() a Rows and Release() each RecordBatch you take from Next.
  • Values read from a batch alias its buffers. In particular, arrow-go's String.Value(i) and binary accessors return data that points into the Arrow buffer without copying. Anything you keep past that batch's Release() must be copied out (e.g. strings.Clone for strings). The typed decoders (QueryLogsTyped, etc.) do this for you.

Feature coverage

What this binding exposes today, against IMBH's own surface. "Reachable via SQL" means the capability works but through db.Query(ctx, sql) rather than a dedicated method.

Ingest & lifecycle
IMBH capability Go binding Notes
OTLP ingest (logs / traces / metrics) ✅ IngestOTLPLogs / …Traces / …Metrics returns a Receipt
Backpressure-aware ingest ✅ TryIngestOTLP*, SetMaxInFlight, RuntimeStats ErrBackpressure at the cap
Open in-memory / on-disk ✅ OpenInMemory, Open(path)
flush (seal buffer → segment) ✅ Flush
close (drain, join maintenance, final seal, commit_pending) ✅ Close() error the handle is released even when the shutdown errors; un-sealed rows stay in the WAL for the next open to replay
Read-only open (open_read_only) ✅ OpenReadOnly many-reader / single-writer; writes on the handle are rejected
Builder options (memory budget, WAL mode, retention, compression, maintenance, promote) ✅ OpenWith(DbOptions) host-runtime-Handle variants (async ingest, runtime-driven maintenance) deferred. Since imbh 0.8.0 the promote list and the retention policy are durable database state, so omitting PromoteKeys / RetentionDays inherits what the database carries — ResetPromote / ResetRetention are the explicit "clear it" spellings, and a read-only open always adopts the durable promote set
Flush policy (DbBuilder::flush, imbh 0.2.0) ✅ DbOptions.Flush imbh's spec string ("interval=5s,wal=64MiB", or "manual"); needs MaintenanceBackgroundNs set, since that is what runs the scheduler
Duplicate-timestamp policy (DbBuilder::duplicates, imbh 0.5.0) ✅ DbOptions.Duplicates imbh's spec string: "error_on_read" (default, fails the PromQL read), "last_wins" (collapse at read), "reject[,recent=N]" (drop at ingest → Receipt.Rejected, DbStats.IngestRejected). Since imbh 0.6.0 "last_wins" also collapses the typed metric reads (QueryMetricsTyped / QueryMetricInstant), which previously aggregated both points
Ops: stats, compact, maintain, snapshot, segments, segment_files, durable_through, export (Arrow IPC) ✅ Stats / Compact / Maintain / Snapshot / Segments / SegmentFiles / DurableThrough / Export (+ExportRecords) writer-only ops error on a read-only handle. SegmentFiles answers correctly on a reader since imbh 0.8.0 — it used to report an empty list, indistinguishable from "no segments"
Bounded compaction (compact_bounded, imbh 0.8.0) ✅ CompactBounded(maxPartitions) a pass costs the corpus, not the request; the bound makes it incremental. Partitions past it are untouched rather than half-done, so there is nothing to resume — call it again. CompactionReport.SegmentsConverged counts the 1→1 rewrites a promote change leaves behind
Pending housekeeping (commit_pending, imbh 0.8.0) ✅ CommitPending → PendingReport the embedded half of out-of-process segment rewriting: validate an external preparer's records, one manifest delta. Maintain now does it on every pass too (MaintenanceReport.Pending*)
Durable promote set (promote / set_promote, imbh 0.8.0) ✅ Promoted / SetPromoted(keys) → []string the database's set, not the handle's. SetPromoted seals first, persists, and returns what is in effect; segments sealed before a key was promoted answer through the attributes JSON until Compact converges them
Durable retention policy (retention, imbh 0.8.0) ✅ Retention → RetentionPolicy both bounds optional; an external housekeeper applies the host's rules instead of inventing its own
Attribute statistics (attribute_stats, imbh 0.8.0) ✅ AttributeStats(AttrStatsOptions) → AttrStatsReport cardinality, sigma (per-segment selectivity), the cardinality curve over a window ladder, and both verdicts — promote this key and index it, up to this width — computed upstream. Takes no lock and writes nothing, so it runs on a reader beside a live writer; PromotionCandidates feeds SetPromoted
Query surfaces
IMBH capability Go binding Notes
SQL over logs / spans / metrics_* ✅ Query(ctx, sql) lazy, zero-copy, cancellable
SQL UDFs (histogram_quantile, matches, json_get_str, hex) ✅ via SQL incl. computed quantiles
Typed log query ✅ QueryLogs, QueryLogsTyped → []LogEntry curated field subset (below)
Metric range (resampled) ✅ QueryMetrics, QueryMetricsTyped → Matrix GroupBy splits series
Raw metric samples ✅ QueryMetricPoints, …Typed → []MetricPoint unaggregated; histogram → raw Rows
Span RED metrics ✅ QuerySpanMetrics, …Typed → []SpanMetricPoint calls / errors / p50 / p95 / p99
One trace's spans ✅ GetTrace, GetTraceSpans → []Span composes with TraceQL matches
Assembled Trace tree ✅ GetTraceForest, AssembleTrace → []*TraceNode parent→child forest; orphans surface as roots
Log pagination (LogPage cursor) ✅ QueryLogPage → LogPage{Entries,Next,Stats} opaque resume Cursor + per-page QueryStats
logs().volume / volume_by ✅ LogVolume / LogVolumeBy → []VolumeBucket time-bucketed counts, optional group_by
logs().count ✅ CountLogs(ctx, LogQuery) → uint64 full count(*) over the filter; ignores Limit/Backward
Arrival axis (observed_after / LogOrder, imbh 0.6.0) ✅ LogQuery.ObservedAfter / LogQuery.OrderBy filter and order on ingest time instead of event time; LogEntry.ObservedTime carries it back
traces().search (TraceSummary) ✅ SearchTraces → []TraceSummary full attr-predicate set on TraceQuery
metrics().instant (Vector) ✅ QueryMetricInstant → []InstantSample last sample per series
metrics().catalog / series / exemplars ✅ MetricCatalog / MetricSeries / MetricExemplars
metrics().dimensions (imbh 0.7.0) ✅ MetricDimensions → []MetricDimension label axes + their distinct values; kind-agnostic, so it answers for a histogram, which PromQL cannot select bare; reports the promoted service axis that MetricSeries leaves out
attrs() discovery (names, values) ✅ AttrNames / AttrValues
LGTM query languages
Capability Go binding Notes
PromQL ✅ QueryPromQL, QueryPromQLSeries → []Series metric names auto-resolved from the catalog (dots→underscores)
LogQL — range aggregation (matrix) ✅ QueryLogQL, QueryLogQLSeries → []Series count_over_time, rate, …
LogQL — bare selector (streams) ✅ QueryLogQLLines → []LogEntry {service="x"} |= "err"
TraceQL ✅ QueryTraceQL, QueryTraceQLMatches → []TraceMatch returns trace + selected span ids
Out-of-profile constructs ❌ by design rejected with a stable diagnostic rather than approximated
Typed-query field coverage

LogQuery now covers IMBH's builder broadly: Service, Match, AttrEq, Start, End, Limit, Backward, plus TraceID/SpanID correlation, SeverityAtLeast, the richer attribute predicates (AttrExists, AttrMatches, AttrIn/AttrNotIn, AttrGt/Ge/Lt/Le, AttrRegex), and the arrival axis (ObservedAfter, OrderBy); cursor paging is via QueryLogPage. TraceQuery (trace search) exposes service / name / status / kind / duration bounds / time range / limit and the same attribute-predicate set. SpanMetricsQuery and MetricQuery cover the common fields.

Transport & correctness
Property Status
Zero-copy Arrow results (C Data Interface) ✅ all query surfaces
Lazy streaming (bounded memory) ✅ SQL only — typed/LGTM queries collect eagerly upstream
Context cancellation ✅ ctx first arg on every query
Error surfacing ✅ Rows.Err(), Go errors on byte ops
Leak-verified ownership ✅ live-batch counter + Valgrind gate

Limitations & roadmap

Honest current state:

  • API stability. Pre-1.0, so breaking changes land in minor releases. In v0.8.0, two behavior changes carry no compile error with them: DbOptions.PromoteKeys and DbOptions.RetentionDays left unset now inherit the database's stored setting instead of resetting it (imbh 0.8.0 made both durable state — use ResetPromote / ResetRetention to clear), and a filter on an attribute key promoted after some rows were sealed now matches those rows through the attributes JSON, where it used to silently match none of them. Also in v0.8.0: SegmentFiles on a read-only handle returns the real paths rather than an empty list. v0.7.0 changed (*DB).Close() to return error instead of nothing — a failed final seal used to be silent; existing defer db.Close() call sites still compile.
  • Packaging. Consumed as external deps — imbh from crates.io (0.8.0) and sable git-pinned to a main commit — so no local checkouts are required to build. CI runs the standard gate on every push and pull request; cold build is minutes; the static archive is large.
  • Durability. Covered by TestDurabilityReopen (ingest → flush → close → reopen the same path → query without re-ingest, exercising WAL replay / segment reload). Read-only opens are tested too.
  • Concurrency. Stress-tested by TestConcurrentQueries (48 goroutines × 60 iters, mixed SQL + typed queries against one shared Db, all under -race).
  • Portability. linux/{amd64,arm64} and windows/amd64 are gated in CI: full build + go test -race, with the Windows leg building for x86_64-pc-windows-gnu and linking through the cgo directives natively. Durable (on-disk) DBs need imbh 0.1.1 or newer on Windows — 0.1.0 failed every on-disk open there (imbh#3). The Apple cells are compile-checked in CI (cargo check on a macOS runner) and built on release tags, but the Go suite has never run there.
  • Query surface. SQL, typed queries, LGTM (PromQL / LogQL / TraceQL), cursor-paged logs (QueryLogPage
    • QueryStats), trace search, log volume, metric catalog / series / exemplars / instant, and attribute discovery are all exposed, alongside the read-only / builder-options / ops admin surface — see Feature coverage for the exact matrix. Remaining: typed and LGTM queries collect eagerly upstream (only SQL streams lazily), logs().count is reachable via SQL rather than a dedicated method, and the two host-runtime-Handle builder variants (async ingest, runtime-driven maintenance) are deferred.

Testing

make test            # go test -tags sable_extern_lib -race ./...
make leak-valgrind   # Valgrind buffer-leak gate (requires valgrind)

License

Licensed under the Apache License, Version 2.0. This binding also links IMBH (Apache-2.0) and sable (MIT).

Documentation

Overview

Package imbhgo is a Go binding for IMBH — an embeddable observability database — with zero-copy Arrow query results fused onto Go's scheduler via sable. (Binding plan M0: open → SQL → zero-copy rows. Ingest and typed queries land in M1/M2.)

Build with -tags sable_extern_lib so sable's Go package contributes no -lsable of its own; the combined staticlib below (which contains both imbhgo_* and sable_* symbols) is linked instead.

Index

Constants

View Source 
const (
	// PromoteYes means string-valued, cheap enough to store, and present on enough rows.
	PromoteYes = "yes"
	// PromoteYesRare means cheap and string-valued but rare: the column is mostly NULL, which costs
	// little but only pays if the key is actually queried.
	PromoteYesRare = "yes?"
	// PromoteMarginal and PromoteCostly mean the column is in, or firmly in, the expensive regime.
	PromoteMarginal = "marginal"
	PromoteCostly   = "costly"
	// PromoteNo means too few string values for the column to populate at all.
	PromoteNo = "no"
	// PromoteUnknown means no rows, or segments too small for repetition to be observable — a verdict
	// about the corpus rather than the key, so none is given.
	PromoteUnknown = "-"
)

Promote verdicts, as spelled by imbh_attrstats::promote_verdict. They are computed upstream, not re-derived here: the thresholds behind them belong in one place, and a threshold copied across an FFI boundary is a threshold that drifts.

View Source 
const IndexScaleAll = "all"

IndexScaleAll is the IndexScale answer meaning pruning pays at any query width, so there is no horizon to name.

View Source 
const Version = release.Version

Version is the module release this source corresponds to. It is bumped in lockstep with each tagged release, and the prebuilt libimbhgo.a assets on GitHub Releases are named after it. The canonical value lives in internal/release (which is cgo-free so cmd/imbhgo-fetch can share it without depending on the archive); this re-export lets consumers read imbhgo.Version at runtime.

Variables

View Source 
var ErrBackpressure = sable.ErrBackpressure

ErrBackpressure is returned by the Try* entry points and by Query when the fused runtime is at its in-flight cap (see SetMaxInFlight). No work was admitted; the caller should shed load or retry.

Functions

func SetMaxInFlight 

func SetMaxInFlight(max uint64)

SetMaxInFlight caps the number of concurrently in-flight *admitted* operations (0 = unbounded, the default). The cap is process-global (one fused runtime) and applies to every admission-controlled entry point: TryIngestOTLP* and Query — a live Rows holds a slot until Close, so the cap also bounds concurrent open result streams. The blocking IngestOTLP* path is never refused (it still counts toward the in-flight gauge). Observe rejections via RuntimeStats().Rejected.

Types

type AttrScope added in v0.8.0 

type AttrScope string

AttrScope names one of the three attribute columns a measurement can cover. Only ScopeAttributes is the scope a promote list applies to; the other two are reachable only by a segment index. 

const (
	// ScopeAttributes is the per-record attributes column — the promote scope. Its keys are reported
	// under their bare names, exactly as a promote list would spell them.
	ScopeAttributes AttrScope = "attributes"
	// ScopeResource is the resource attributes column. Its keys are reported prefixed "resource:".
	ScopeResource AttrScope = "resource"
	// ScopeScope is the instrumentation-scope attributes column, reported prefixed "scope:".
	ScopeScope AttrScope = "scope"
)

type AttrStatsOptions added in v0.8.0 

type AttrStatsOptions struct {
	// Scopes restricts the attribute columns read. Empty = all three.
	Scopes []AttrScope `json:"scopes,omitempty"`
	// StartNs and EndNs restrict the scan to segments overlapping that absolute window. Both zero =
	// every segment. Narrowing the range is part of the measurement, not a shortcut — sigma is
	// defined over the segments in a time range — and it is also what bounds the work on a large
	// database.
	StartNs int64 `json:"start,omitempty"`
	EndNs   int64 `json:"end,omitempty"`
	// Windows replaces the cardinality ladder, in imbh's own duration spelling: comma-separated,
	// innermost first, strictly increasing ("30s,5m,1h,7d"; bare digits are seconds). "none" disables
	// the ladder and the per-value memory it costs. "" keeps imbh's default. A malformed spec fails
	// the call rather than falling back.
	Windows string `json:"windows,omitempty"`
	// MaxKeys caps tracked keys per scan unit, MaxValues caps distinct values per key, before hash
	// sampling engages. 0 = imbh's default. Whenever a cap engages the affected row's sample rate
	// drops below 1 and its estimates become estimates — see KeyReport.ValuesSampleRate.
	MaxKeys   uint64 `json:"max_keys,omitempty"`
	MaxValues uint64 `json:"max_values,omitempty"`
	// BatchSize is the Parquet read batch size. 0 = imbh's default.
	BatchSize uint64 `json:"batch_size,omitempty"`
}

AttrStatsOptions bounds what an AttributeStats run measures and how much memory it spends. The zero value takes imbh's defaults: all three scopes, every sealed segment, the 1m/1h/24h ladder, 4096 keys and 50000 values per scan unit, 8192-row batches.

type AttrStatsReport added in v0.8.0 

type AttrStatsReport struct {
	// Dir is the database directory as the measuring process saw it.
	Dir       string      `json:"dir"`
	Scopes    []AttrScope `json:"scopes"`
	MaxKeys   int         `json:"max_keys"`
	MaxValues int         `json:"max_values"`
	// Range is the absolute [start, end] segment window, when the scan was restricted to one.
	Range  *[2]int64     `json:"range"`
	Levels []LevelReport `json:"levels"`
	// PendingWalFrames counts unsealed WAL frames at the time of the scan: rows in no segment yet,
	// and so not measured. Flush first to include them.
	PendingWalFrames int `json:"pending_wal_frames"`
	// SegmentsSkipped names segments that could not be read (retention or compaction removed them
	// mid-scan), with the reason. A skipped segment under-counts rather than aborting the run, so it
	// has to be visible.
	SegmentsSkipped []string `json:"segments_skipped"`
	// Tables is the per-table units; Global is the DB-wide roll-up, which is the scope a promote list
	// applies at.
	Tables []UnitReport `json:"tables"`
	Global UnitReport   `json:"global"`
	// PromotionCandidates is the actionable summary: the DB-wide attributes-scoped keys with both
	// verdicts, ordered by rows a column would populate.
	PromotionCandidates []PromotionCandidate `json:"promotion_candidates"`
}

AttrStatsReport is a whole measurement: every table, the DB-wide roll-up, the promotion verdicts, and what the run could not cover.

type CompactionReport 

type CompactionReport struct {
	// SegmentsMerged counts segments consumed by a merge — a partition that held more than one.
	SegmentsMerged uint64 `json:"segments_merged"`
	// SegmentsConverged counts segments rewritten in place (1 -> 1) because their schema lagged the
	// promoted key set, which is not a merge and so is counted apart (imbh 0.8.0). Before that a
	// single-segment partition was skipped outright, so a partition that never gained a second
	// segment never converged after a SetPromoted and kept answering through the JSON fallback
	// forever. Expect a burst here on the first Compact after promoting a key: every segment lacking
	// the new column is rewritten once.
	SegmentsConverged uint64 `json:"segments_converged"`
	SegmentsCreated   uint64 `json:"segments_created"`
}

CompactionReport summarizes a compact() pass (imbh: Db::compact).

type Cursor 

type Cursor []byte

Cursor is an opaque page-resume token. Obtain it from LogPage.Next and pass it back to QueryLogPage to fetch the following page (reusing the same filters/limit/direction). Treat it as opaque — do not construct or interpret it. The zero (nil) value requests the first page.

func (Cursor) HasMore 

func (c Cursor) HasMore() bool

HasMore reports whether a following page exists (i.e. this cursor points somewhere).

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB is a handle to an embedded IMBH database.

func Open 

func Open(path string) (*DB, error)

Open opens a durable, on-disk database at path (created if absent).

func OpenInMemory 

func OpenInMemory() (*DB, error)

OpenInMemory opens an ephemeral, process-local database (great for tests and dev loops).

func OpenReadOnly 

func OpenReadOnly(path string) (*DB, error)

OpenReadOnly opens an existing on-disk database at path read-only. It takes no writer lock, so it coexists with the single writer process and with other readers; queries see the writer's segments unioned with its live WAL tail (near-real-time). Every write returns an error.

Rejected if the writer had its WAL disabled (the reader could then get only seal-interval freshness, not near-real-time); use OpenWith with AllowStaleReads to accept that. (imbh: Db::open_read_only.)

func OpenWith 

func OpenWith(opts DbOptions) (*DB, error)

OpenWith opens a durable database configured by opts (imbh: Db::builder(path) + setters).

func (*DB) AttrNames 

func (db *DB) AttrNames(ctx context.Context) ([]string, error)

AttrNames returns every distinct attribute/label key present on any signal (logs, spans, metrics), plus "service.name" when any record carries a service. Sorted.

func (*DB) AttrValues 

func (db *DB) AttrValues(ctx context.Context, key string) ([]string, error)

AttrValues returns the distinct string values for one attribute key across every signal, sorted.

func (*DB) AttributeStats added in v0.8.0 

func (db *DB) AttributeStats(opts AttrStatsOptions) (AttrStatsReport, error)

AttributeStats measures attribute cardinality and per-segment selectivity over this database — the measurement behind a promote list (imbh 0.8.0: Db::attribute_stats).

It answers, per attribute key: how many distinct values it has, how much of that value space lives inside a single segment (sigma, so how much a segment index could prune), how the count grows with the query window, and what a promoted column would cost per row. Feed PromotionCandidates to SetPromoted to act on it.

Reads and changes nothing: it replays the manifest and opens each sealed segment read-only with just its attribute columns projected — no lock, no new column, no manifest edit — so it is safe on a read-only handle and safe while a writer is running, including this one. It is a full scan of those columns though; bound it with AttrStatsOptions.StartNs/EndNs rather than running it per keystroke.

Only sealed segments are covered (see PendingWalFrames), and an in-memory database is an error rather than an empty report — no segment exists to be selective within, which is a different answer from "no attributes".

func (*DB) Close 

func (db *DB) Close() error

Close shuts the database down and releases the handle. It performs IMBH's clean shutdown — drain the async-ingest worker, join the background maintenance thread, a final seal() and commit_pending() — and returns the error that shutdown reported, if any.

The handle is released whether or not the shutdown succeeded, so Close is not retryable and a second call is a no-op returning nil. An error here is a reporting signal, not data loss: rows the final seal could not write stay in the WAL for the next Open to replay, the same state a crash would have left. It does mean the on-disk state is not the tidy post-shutdown one, so a caller that cares (a supervisor deciding whether the process exited cleanly, a test asserting durability) should check it. `defer db.Close()` remains valid where you do not.

func (*DB) CommitPending added in v0.8.0 

func (db *DB) CommitPending() (PendingReport, error)

CommitPending applies the segment-rewrite records an out-of-process housekeeper prepared: validate each, swap it into the manifest, unlink what it replaced. Writer-only.

This is the cheap half of compaction — one manifest delta — because the reading and rewriting happened in the other process. Maintain also commits pending records on every pass (see MaintenanceReport.Pending*), so a host running background maintenance need not call this at all; it is here for a host that seals on its own schedule.

func (*DB) Compact 

func (db *DB) Compact() (CompactionReport, error)

Compact merges segments to reduce fragmentation, and rewrites segments whose schema lags the promoted key set. Writer-only.

func (*DB) CompactBounded added in v0.8.0 

func (db *DB) CompactBounded(maxPartitions int) (CompactionReport, error)

CompactBounded is Compact stopping after maxPartitions UTC-day partitions have been rewritten (imbh 0.8.0: Db::compact_bounded). Writer-only; maxPartitions must be positive.

A compaction pass costs the corpus, not the request: over a database with hundreds of day partitions, Compact rewrites every one. A bound makes it incremental — a slice now, the rest later — instead of forcing a choice between an hour-long call and never compacting. Partitions past the bound are untouched rather than half-done, so a capped pass is not a partial one and there is nothing to resume: call it again. A pass that rewrites nothing is how you learn there is nothing left to do.

func (*DB) CountLogs 

func (db *DB) CountLogs(ctx context.Context, q LogQuery) (uint64, error)

CountLogs returns the number of log records matching q — imbh's logs().count(filter), a full count(*) over the filter that ignores q.Limit and q.Backward (they bound/order returned rows, not the total). It scans without materializing rows, so it is cheaper than draining QueryLogs when you only need the tally. Equivalent to SELECT count(*) via Query, but driven by the same typed LogQuery.

func (*DB) DurableThrough 

func (db *DB) DurableThrough() (uint64, bool, error)

DurableThrough returns the highest LSN durably persisted, or (0, false) if nothing is durable yet (imbh: Db::durable_through).

func (*DB) Export 

func (db *DB) Export(table Table, startNs, endNs int64) ([]byte, error)

Export returns the given table's rows over [startNs, endNs) as an Arrow-IPC stream (a self-describing schema + record batches), buffer unioned with segments ordered by time. Pass startNs == endNs == 0 for the whole range. Use ExportRecords to decode. (imbh: Db::export.)

func (*DB) ExportRecords 

func (db *DB) ExportRecords(table Table, startNs, endNs int64) ([]arrow.RecordBatch, error)

ExportRecords is Export decoded into Arrow record batches. Each returned batch is Retained and owned by the caller — Release() it when done.

func (*DB) Flush 

func (db *DB) Flush() error

Flush seals the in-memory buffer into an immutable segment. Not required for queryability (queries see the buffer union segments); use it to bound memory or to exercise the on-disk read path.

func (*DB) GetTrace 

func (db *DB) GetTrace(ctx context.Context, traceID string) (*Rows, error)

GetTrace fetches one trace's spans as zero-copy Arrow rows. traceID is the 32-char hex id — the same form QueryTraceQLMatches returns, so a TraceQL match can be passed straight in.

func (*DB) GetTraceForest 

func (db *DB) GetTraceForest(ctx context.Context, traceID string) ([]*TraceNode, error)

GetTraceForest fetches a trace's spans (via GetTraceSpans) and returns them assembled into a parent→child forest. It is named GetTraceForest rather than GetTrace because DB.GetTrace already exists in lgtm.go and returns zero-copy Arrow *Rows.

func (*DB) GetTraceSpans 

func (db *DB) GetTraceSpans(ctx context.Context, traceID string) ([]Span, error)

GetTraceSpans fetches a trace and decodes its spans into []Span (ordered by start time).

func (*DB) IngestOTLPLogs 

func (db *DB) IngestOTLPLogs(otlp []byte) (Receipt, error)

IngestOTLPLogs ingests OTLP/HTTP logs export-request protobuf bytes (what a stock OTel exporter sends). Data is queryable immediately, before any Flush.

func (*DB) IngestOTLPMetrics 

func (db *DB) IngestOTLPMetrics(otlp []byte) (Receipt, error)

IngestOTLPMetrics ingests an OTLP metrics export-request.

func (*DB) IngestOTLPTraces 

func (db *DB) IngestOTLPTraces(otlp []byte) (Receipt, error)

IngestOTLPTraces ingests an OTLP traces export-request.

func (*DB) LogVolume 

func (db *DB) LogVolume(ctx context.Context, q LogQuery, stepNs int64) ([]VolumeBucket, error)

LogVolume returns log record counts per stepNs-sized time bucket over the filter q. The bucket start is floor(time/stepNs)*stepNs in unix nanos; buckets carry no labels (Labels == "{}").

func (*DB) LogVolumeBy 

func (db *DB) LogVolumeBy(ctx context.Context, q LogQuery, stepNs int64, groupBy []string) ([]VolumeBucket, error)

LogVolumeBy is LogVolume broken down by the given attribute keys — counts per (step-bucket, label set), each bucket carrying its Labels as canonical JSON. Empty groupBy is equivalent to LogVolume.

A key may name a record attribute or the service: "service.name" (the OTel resource key) and "service" (the column it is lifted into at ingest) both split per service. Requires imbh 0.3.0 — before that, either spelling silently collapsed the breakdown into one empty-labelled bucket set.

func (*DB) Maintain 

func (db *DB) Maintain() (MaintenanceReport, error)

Maintain runs a maintenance pass (seal + retention enforcement). Writer-only; a read-only handle returns an error.

func (*DB) MetricCatalog 

func (db *DB) MetricCatalog(ctx context.Context) ([]MetricInfo, error)

MetricCatalog returns the metric catalog — one MetricInfo per stored metric.

func (*DB) MetricDimensions added in v0.7.0 

func (db *DB) MetricDimensions(ctx context.Context, metric string) ([]MetricDimension, error)

MetricDimensions returns the label axes of a metric — every label its series carry, each with its distinct values, sorted by name. It is the axis list a "group by" / "filter by" picker needs.

Two things distinguish it from MetricSeries, which answers the raw per-series attribute sets:

  • It is kind-agnostic. PromQL cannot select a cumulative histogram bare (its buckets are reachable only through histogram_quantile), so a caller cannot discover a histogram's labels by evaluating a selector the way it can for a gauge or a sum; this answers for a histogram exactly as for a gauge.
  • It reports the promoted service column as the "service" label — the name PromQL evaluation gives it — whereas MetricSeries deliberately leaves the resource axis out.

Only string-valued attributes become labels, since that is what a PromQL label set can match on. Requires imbh 0.7.0.

func (*DB) MetricExemplars 

func (db *DB) MetricExemplars(ctx context.Context, metric string) ([]Exemplar, error)

MetricExemplars returns every exemplar recorded for a metric. Exemplars are carried by histogram/ exponential-histogram (and gauge/sum) points; a metric with none yields an empty slice.

func (*DB) MetricSeries 

func (db *DB) MetricSeries(ctx context.Context, metric string) ([]string, error)

MetricSeries returns the distinct label sets (series) carrying a metric, each rendered as its canonical JSON object string. Resource-level dimensions like service are separate axes and are not folded in.

func (*DB) Promoted added in v0.8.0 

func (db *DB) Promoted() ([]string, error)

Promoted returns the attribute keys promoted to dedicated columns, in column order (imbh 0.8.0: Db::promote).

Since imbh 0.8.0 this is the *database's* durable set, recorded on disk and read at open — not whatever DbOptions.PromoteKeys this handle asked for. A reader always adopts it, so two handles on one directory can no longer disagree about the column layout of the same segments.

func (*DB) Query 

func (db *DB) Query(ctx context.Context, sql string) (*Rows, error)

Query runs a SQL statement and returns a lazily-streamed, zero-copy result set. The query executes batch-by-batch on IMBH's engine (fused onto Go's scheduler via sable); each Rows.Next pulls one Arrow RecordBatch without copying its buffers. Cancelling ctx aborts a parked Next (interrupting a slow batch) and cancels the query on the IMBH side (releasing its pinned snapshot); after cancellation, Next returns ok=false with ctx.Err().

func (*DB) QueryLogPage 

func (db *DB) QueryLogPage(ctx context.Context, q LogQuery, after Cursor) (*LogPage, error)

QueryLogPage runs a single page of a log query. Pass a nil Cursor for the first page; pass the returned LogPage.Next (while HasMore) for each subsequent page, keeping the same LogQuery. Rows are materialized into []LogEntry (like QueryLogsTyped); the page's cursor and QueryStats come back alongside. (imbh: logs().query → LogPage, LogQuery::after.)

func (*DB) QueryLogQL 

func (db *DB) QueryLogQL(ctx context.Context, query string, start, end, step int64) (*Rows, error)

QueryLogQL evaluates a LogQL query over [start, end] at step, returning zero-copy Arrow rows. LogQL has two result shapes (as in Loki): a range aggregation (e.g. count_over_time, rate) yields labeled series (labels|timestamp|value), while a bare selector yields log lines (the logs projection). See QueryLogQLSeries and QueryLogQLLines for the decoded forms.

func (*DB) QueryLogQLLines 

func (db *DB) QueryLogQLLines(ctx context.Context, query string, start, end int64, limit int) ([]LogEntry, error)

QueryLogQLLines evaluates a bare LogQL selector (e.g. `{service="checkout"} |= "error"`) and decodes the matching log lines. This is LogQL's `streams` result shape, as opposed to the `matrix` shape a range aggregation produces (see QueryLogQLSeries). limit caps the lines returned (0 = engine default).

func (*DB) QueryLogQLSeries 

func (db *DB) QueryLogQLSeries(ctx context.Context, query string, start, end, step int64) ([]Series, error)

QueryLogQLSeries evaluates a LogQL range aggregation and decodes it into labeled series.

func (*DB) QueryLogs 

func (db *DB) QueryLogs(ctx context.Context, q LogQuery) (*Rows, error)

QueryLogs runs a typed log query, returning a zero-copy streamed result set (see Rows).

func (*DB) QueryLogsTyped 

func (db *DB) QueryLogsTyped(ctx context.Context, q LogQuery) ([]LogEntry, error)

QueryLogsTyped runs a typed log query and decodes the result rows into []LogEntry. Convenience over QueryLogs for callers that want Go structs rather than raw Arrow batches. Prefer QueryLogs (+ manual Arrow) for very large results, since this materializes all rows.

func (*DB) QueryMetricInstant 

func (db *DB) QueryMetricInstant(ctx context.Context, q MetricQuery) ([]InstantSample, error)

QueryMetricInstant runs an instant metric query (imbh's `metrics().instant`) over the same MetricQuery as QueryMetrics, returning one InstantSample per series (the last point in range).

func (*DB) QueryMetricPoints 

func (db *DB) QueryMetricPoints(ctx context.Context, q MetricPointsQuery) (*Rows, error)

QueryMetricPoints returns raw metric samples as zero-copy Arrow rows. Columns: point_time, metric, service, attributes, temporality, is_monotonic, then value (scalar kinds) or explicit_bounds + bucket_counts (histogram).

func (*DB) QueryMetricPointsTyped 

func (db *DB) QueryMetricPointsTyped(ctx context.Context, q MetricPointsQuery) ([]MetricPoint, error)

QueryMetricPointsTyped decodes raw scalar (gauge/sum) samples into []MetricPoint. For histogram metrics use QueryMetricPoints and read the bucket columns directly.

func (*DB) QueryMetrics 

func (db *DB) QueryMetrics(ctx context.Context, q MetricQuery) (*Rows, error)

QueryMetrics runs a typed metric range query, returning a zero-copy streamed result set.

func (*DB) QueryMetricsTyped 

func (db *DB) QueryMetricsTyped(ctx context.Context, q MetricQuery) (Matrix, error)

QueryMetricsTyped runs a metric range query and decodes it into a Matrix (rows grouped into series by the GroupBy label set). Convenience over QueryMetrics.

func (*DB) QueryPromQL 

func (db *DB) QueryPromQL(ctx context.Context, query string, start, end, step int64) (*Rows, error)

QueryPromQL evaluates a PromQL query over [start, end] at the given step (all unix nanoseconds), returning the result as zero-copy Arrow rows (columns labels | timestamp | value).

func (*DB) QueryPromQLSeries 

func (db *DB) QueryPromQLSeries(ctx context.Context, query string, start, end, step int64) ([]Series, error)

QueryPromQLSeries evaluates a PromQL query and decodes the result into labeled series.

func (*DB) QuerySpanMetrics 

func (db *DB) QuerySpanMetrics(ctx context.Context, q SpanMetricsQuery) (*Rows, error)

QuerySpanMetrics runs a span (RED) metrics query, returning a zero-copy streamed result set with columns bucket, [group labels], calls, errors, p50, p95, p99.

func (*DB) QuerySpanMetricsTyped 

func (db *DB) QuerySpanMetricsTyped(ctx context.Context, q SpanMetricsQuery) ([]SpanMetricPoint, error)

QuerySpanMetricsTyped runs a span (RED) metrics query and decodes the rows into []SpanMetricPoint.

func (*DB) QueryTraceQL 

func (db *DB) QueryTraceQL(ctx context.Context, query string, start, end int64) (*Rows, error)

QueryTraceQL evaluates a TraceQL query over the trace-start window [start, end] (unix nanoseconds), returning matches as zero-copy Arrow rows (columns trace_id | span_id).

func (*DB) QueryTraceQLMatches 

func (db *DB) QueryTraceQLMatches(ctx context.Context, query string, start, end int64) ([]TraceMatch, error)

QueryTraceQLMatches evaluates a TraceQL query and decodes the matches into []TraceMatch.

func (*DB) Retention added in v0.8.0 

func (db *DB) Retention() (RetentionPolicy, error)

Retention returns the retention policy in effect. Like Promoted, this is durable database state since imbh 0.8.0 rather than per-handle configuration, so an external housekeeper applies the host's rules instead of inventing its own — and omitting DbOptions.RetentionDays inherits the stored policy rather than clearing it (DbOptions.ResetRetention clears it).

func (*DB) SearchTraces 

func (db *DB) SearchTraces(ctx context.Context, q TraceQuery) ([]TraceSummary, error)

SearchTraces returns the trace summaries matching q, most-recent-first per imbh's ordering.

func (*DB) SegmentFiles 

func (db *DB) SegmentFiles(table Table) ([]string, error)

SegmentFiles lists the on-disk file paths backing the given table (imbh: Db::segment_files).

func (*DB) Segments 

func (db *DB) Segments() ([]SegmentRef, error)

Segments lists the database's current on-disk segments.

func (*DB) SetPromoted added in v0.8.0 

func (db *DB) SetPromoted(keys []string) ([]string, error)

SetPromoted changes the promoted attribute keys on a live database and returns the set now in effect (imbh 0.8.0: Db::set_promote). Writer-only. Passing an empty list demotes everything.

The whole set is sent rather than a delta: promotion is an ordered list — the order is the column order — so a delta would ask the database to guess placement and would let two callers silently lose each other's change.

This seals the buffer first, which is what makes the set mutable at all: batches are encoded at ingest against the set in effect then, and a buffer holding batches from both sides of the change cannot be concatenated safely. The new set is persisted, so it survives reopen and every other handle sees it.

Existing segments are not rewritten. Ones sealed before a key was promoted have no column for it and are read back out of the attributes JSON on exactly those rows, so history stays queryable in either direction — the fallback is correct but pays a JSON parse; Compact converges those segments (see CompactionReport.SegmentsConverged). Demotion is the safe direction, since the key never left the JSON.

Returns an error on a read-only handle, and after several attempts if ingest is busy enough that the buffer refills between every seal and the swap.

func (*DB) Snapshot 

func (db *DB) Snapshot(dir string) (SnapshotInfo, error)

Snapshot writes a consistent copy of the database's segments into dir (created if absent). Writer-only.

func (*DB) Stats 

func (db *DB) Stats() (DbStats, error)

Stats returns a snapshot of storage and ingest counters. Works on readers and writers.

func (*DB) TryIngestOTLPLogs 

func (db *DB) TryIngestOTLPLogs(otlp []byte) (Receipt, error)

TryIngestOTLPLogs is IngestOTLPLogs with backpressure: at the in-flight cap it returns ErrBackpressure immediately without ingesting (nothing admitted), so a producer can shed load or retry with backoff instead of piling on unbounded work.

func (*DB) TryIngestOTLPMetrics 

func (db *DB) TryIngestOTLPMetrics(otlp []byte) (Receipt, error)

TryIngestOTLPMetrics is IngestOTLPMetrics with backpressure.

func (*DB) TryIngestOTLPTraces 

func (db *DB) TryIngestOTLPTraces(otlp []byte) (Receipt, error)

TryIngestOTLPTraces is IngestOTLPTraces with backpressure.

type DbOptions 

type DbOptions struct {
	// Path is the on-disk directory (required).
	Path string `json:"path"`
	// ReadOnly opens as a reader (no writer lock; many may coexist with one writer).
	ReadOnly bool `json:"read_only,omitempty"`
	// AllowStaleReads lets a read-only open accept seal-interval freshness when the writer's WAL is
	// off (otherwise such an open is rejected).
	AllowStaleReads bool `json:"allow_stale_reads,omitempty"`
	// MemoryBudgetBytes caps the in-memory buffer (0 = imbh default, 128 MiB).
	MemoryBudgetBytes uint64 `json:"memory_budget_bytes,omitempty"`
	// Compression selects the segment codec: "none", "lz4", or "zstd" (with ZstdLevel). "" = default.
	Compression string `json:"compression,omitempty"`
	// ZstdLevel is the zstd level used when Compression == "zstd".
	ZstdLevel int32 `json:"zstd_level,omitempty"`
	// WalMode selects the write-ahead-log mode: "off", "always", or "interval" (with WalIntervalNs).
	WalMode string `json:"wal_mode,omitempty"`
	// WalIntervalNs is the flush interval when WalMode == "interval".
	WalIntervalNs int64 `json:"wal_interval_ns,omitempty"`
	// RetentionDays drops data older than N days. Since imbh 0.8.0 retention is durable database
	// state, so leaving this and MaxDiskBytes at 0 *inherits* the policy the database already carries
	// rather than resetting it to keep-everything — two handles on one directory must not disagree
	// about when data is deleted, and an external housekeeper has to be able to learn the host's
	// policy. Set ResetRetention to clear it instead. Retention reports what is in effect.
	RetentionDays uint64 `json:"retention_days,omitempty"`
	// MaxDiskBytes bounds on-disk segment bytes (0 = no size bound; see RetentionDays on inheritance).
	MaxDiskBytes uint64 `json:"max_disk_bytes,omitempty"`
	// ResetRetention clears the database's stored retention policy back to keep-everything, the one
	// thing an omitted RetentionDays/MaxDiskBytes can no longer express. Takes precedence over both.
	ResetRetention bool `json:"reset_retention,omitempty"`
	// Refresh controls read-only snapshot refresh: "onquery", "manual", or "ttl" (with RefreshTtlNs).
	Refresh string `json:"refresh,omitempty"`
	// RefreshTtlNs is the refresh TTL when Refresh == "ttl".
	RefreshTtlNs int64 `json:"refresh_ttl_ns,omitempty"`
	// MaintenanceBackgroundNs runs background maintenance on an owned OS thread every N ns (0 = manual).
	// This picks *who* runs the scheduler; Flush picks *when* it seals. imbh's default is manual, so
	// with this at 0 nothing seals (and no WAL fsync timer runs) unless you call Flush/Maintain yourself.
	MaintenanceBackgroundNs int64 `json:"maintenance_background_ns,omitempty"`
	// Flush is imbh's flush-policy spec: comma-separated key=value pairs, or the single word "manual"
	// ("off"/"none"/"never" alias it). Keys are interval/every (duration), buffer/bytes (size, or
	// "budget"/"off"), rows (count), wal (size), idle (duration), and tick (duration); the triggers OR
	// together. E.g. "interval=5s,wal=64MiB". Unlike the other string tags, a malformed spec fails the
	// open rather than falling back to a default. "" leaves imbh's own default, which seals on the
	// MaintenanceBackgroundNs tick and at the memory-budget-derived byte threshold. (imbh 0.2.0:
	// DbBuilder::flush.)
	Flush string `json:"flush,omitempty"`
	// Duplicates picks what happens when two metric datapoints share a series *and* a timestamp — a
	// pair PromQL has no meaning for, since series identity is service + __name__ + the string
	// attributes. "" (or "error_on_read") keeps imbh's historical behavior: ingest takes both and any
	// PromQL query touching that series fails, naming the metric, labels and instant. "last_wins"
	// collapses the duplicated instant at read time, degrading one point instead of the whole metric —
	// the only remedy for points already written. "reject" drops the repeat at ingest and counts it in
	// Receipt.Rejected and DbStats.IngestRejected; it costs a fixed ~13 MiB guard and takes an optional
	// lookback, "reject,recent=N" (default 262144 points). The guard is process-local, best-effort, and
	// never rejects out-of-order or late points — only an exact (series, timestamp) repeat. Like Flush,
	// a malformed spec fails the open rather than falling back. (imbh 0.5.0: DbBuilder::duplicates.)
	// Since imbh 0.6.0 "last_wins" collapses the typed metric reads (QueryMetricsTyped /
	// QueryMetricInstant) too, not just PromQL; before that they aggregated both points, so a
	// duplicated gauge averaged the pair and a duplicated sum added it.
	Duplicates string `json:"duplicates,omitempty"`
	// PromoteKeys promotes the given attribute keys to dedicated columns. Reserved names (including
	// "service") are dropped by imbh, and "service.name" needs no promotion: since imbh 0.3.0 both
	// spellings resolve to the built-in service column in every group-by and attribute predicate.
	//
	// Since imbh 0.8.0 the promoted set is durable database state, so an empty PromoteKeys inherits
	// the database's set rather than demoting everything (set ResetPromote for that), and a read-only
	// open ignores this field entirely — a reader adopts the durable set, because a reader that
	// disagreed with the writer would be reading the same segments a different way. SetPromoted
	// changes the set on a live database; Promoted reports it.
	PromoteKeys []string `json:"promote_keys,omitempty"`
	// ResetPromote demotes every promoted key at open — the empty-set spelling an omitted PromoteKeys
	// can no longer express. Takes precedence over PromoteKeys. Ignored on a read-only open.
	ResetPromote bool `json:"reset_promote,omitempty"`
}

DbOptions configures a durable open, mirroring imbh's DbBuilder. The zero value opens with imbh's defaults (equivalent to Open). Only set fields take effect; unrecognized string tags are ignored (leaving the default). The host-runtime option variants (async ingest, runtime-driven maintenance), which need an explicit tokio runtime handle, are intentionally not exposed here.

type DbStats 

type DbStats struct {
	Tables           []TableStats `json:"tables"`
	BufferBytes      uint64       `json:"buffer_bytes"`
	WalBytes         uint64       `json:"wal_bytes"`
	DurableLSN       *uint64      `json:"durable_lsn"`
	IngestQueueDepth uint64       `json:"ingest_queue_depth"`
	IngestDropped    uint64       `json:"ingest_dropped"`
	IngestErrors     uint64       `json:"ingest_errors"`
	// IngestRejected counts metric points dropped at ingest because their (series, timestamp) was
	// already accepted. Non-zero only under DbOptions.Duplicates == "reject"; every other policy
	// takes duplicates and resolves them (or fails) at read time. (imbh 0.5.0.)
	IngestRejected uint64 `json:"ingest_rejected"`
}

DbStats is a snapshot of the database's storage and ingest counters (imbh: Db::stats).

type Exemplar 

type Exemplar struct {
	Time       int64
	Value      float64
	TraceID    string
	SpanID     string
	Attributes string
}

Exemplar is one OTLP exemplar surfaced from a metric point — the trace link for metric→trace drill-down. TraceID/SpanID are "" when the exemplar carries none. Attributes is the exemplar's filtered attributes as canonical JSON ("" when none).

type InstantSample 

type InstantSample struct {
	Labels string
	Time   int64
	Value  float64
}

InstantSample is one series' instant value: the last sample of that series over the query range (Vector semantics — exactly one sample per series). Labels is the canonical JSON label-set string; Time is unix nanoseconds.

type KeyReport added in v0.8.0 

type KeyReport struct {
	Name  string    `json:"name"`
	Scope AttrScope `json:"scope"`
	// RowsPresent counts rows carrying the key; RowsString the subset whose value is a string — the
	// only ones a promoted column would hold.
	RowsPresent uint64 `json:"rows_present"`
	RowsString  uint64 `json:"rows_string"`
	// StrLenAvg and StrLenMax describe the string values' lengths.
	StrLenAvg float64 `json:"str_len_avg"`
	StrLenMax uint32  `json:"str_len_max"`
	// SegmentsPresent counts segments the key appears in.
	SegmentsPresent uint32 `json:"segments_present"`
	// ValuesTracked is the distinct values actually held in memory. ValuesSampleRate is 1 while that
	// was exact and below 1 once MaxValues forced hash sampling — which is what makes DistinctEst and
	// PostingsEst estimates rather than counts.
	ValuesTracked    int     `json:"values_tracked"`
	ValuesSampleRate float64 `json:"values_sample_rate"`
	// DistinctEst is the estimated true distinct-value count. PostingsEst is the (key, value, segment)
	// entries a segment index would hold for this key — the direct size bound.
	DistinctEst float64       `json:"distinct_est"`
	PostingsEst float64       `json:"postings_est"`
	Sigma       *SigmaSummary `json:"sigma"`
	// Curve is the mean distinct values within one window, one entry per configured ladder level,
	// innermost first. A nil entry is a level that opened no window.
	Curve []*float64 `json:"curve"`
	// CSegment is the innermost point of that curve — distinct values within a single segment.
	CSegment *float64 `json:"c_segment"`
	// Repetition is the mean rows per (value, segment) posting: in-segment repetition.
	Repetition float64 `json:"repetition"`
	// Runs counts how often the value differed from the previous row's, over the whole scan. Low
	// means the key arrives in runs, which a promoted column's index array compresses well; near
	// RowsPresent means interleaved, which it does not.
	Runs uint64 `json:"runs"`
	// EstBytesPerRow estimates the bytes per row a promoted column would occupy before compression —
	// the dictionary plus the per-row index array, the second term being the one a postings-per-row
	// figure cannot see. Absolute values run 2-5x high (zstd exploits structure the model does not);
	// it is calibrated for *ordering* keys by cost, which is what a verdict needs.
	EstBytesPerRow float64 `json:"est_bytes_per_row"`
}

KeyReport is one attribute key's measurement within one scan unit.

func (KeyReport) IsSampled added in v0.8.0 

func (k KeyReport) IsSampled() bool

IsSampled reports whether a cap engaged for this key, making its estimates estimates.

type LevelReport added in v0.8.0 

type LevelReport struct {
	Label      string `json:"label"`
	WidthNanos int64  `json:"width_nanos"`
	// Windows counts distinct windows this width opened over the scan. 1 means the width covers
	// everything and a value near the segment count means it has collapsed onto a single segment;
	// either way the level says nothing new.
	Windows uint32 `json:"windows"`
}

LevelReport is one rung of the cardinality ladder, as measured.

type LogEntry 

type LogEntry struct {
	Time int64 // event time, unix nanoseconds
	// ObservedTime is when ingest received the record (unix nanoseconds); 0 when the producer sent
	// none, since the column is nullable. This is the clock LogOrderObservedTime sorts by, so a follow
	// loop feeds the largest value it has seen back as the next query's LogQuery.ObservedAfter.
	ObservedTime int64
	Service      string // service.name (may be "")
	Severity     uint8  // OTLP severity number
	SeverityText string
	Body         string
	Attributes   string // attributes as a JSON object string
	TraceID      []byte // 16 bytes, or nil
	SpanID       []byte // 8 bytes, or nil
}

LogEntry is a decoded log record (a curated subset of IMBH's logs columns).

type LogOrder added in v0.6.0 

type LogOrder string

LogOrder selects which of a record's two instants a log query orders by, independently of LogQuery.Backward (which picks the direction along that axis). A record carries both: Time is when the event happened, ObservedTime is when ingest received it. They differ by up to one batch interval always, and by more whenever a record's own timestamp is trusted.

LogOrderObservedTime is what a tailer wants. Arrival order is monotone in the order rows became visible, so a watermark over it cannot be overtaken by a late-arriving record with an older event time — which is precisely how a follow loop on the event clock drops lines. Records with no observed_time sort last in either direction, and ObservedAfter never matches them (SQL NULL > t is unknown), so they are left out of a follow loop rather than replayed on every poll. 

const (
	LogOrderTime         LogOrder = "time"          // order by event time (the default)
	LogOrderObservedTime LogOrder = "observed_time" // order by arrival time, NULLs last
)

type LogPage 

type LogPage struct {
	Entries []LogEntry
	Next    Cursor
	Stats   QueryStats
}

LogPage is one page of a paged log query: the decoded rows, a resume cursor (empty when the page was short, i.e. no more rows), and the scan statistics.

type LogQuery 

type LogQuery struct {
	Service  string            `json:"service,omitempty"`  // exact service.name match
	Match    string            `json:"match,omitempty"`    // full-text match on the log body
	AttrEq   map[string]string `json:"attr_eq,omitempty"`  // attribute equality filters (AND)
	Start    int64             `json:"start,omitempty"`    // time range start (unix nanos, inclusive)
	End      int64             `json:"end,omitempty"`      // time range end (unix nanos)
	Limit    int               `json:"limit,omitempty"`    // max rows (0 = engine default)
	Backward bool              `json:"backward,omitempty"` // newest-first (default is oldest-first)

	// Trace correlation: filter logs down to a single trace or span.
	TraceID string `json:"trace_id,omitempty"` // hex trace id (32 hex chars); correlate logs to a trace
	SpanID  string `json:"span_id,omitempty"`  // hex span id (16 hex chars); correlate to a single span

	// Severity + attribute predicates (all AND-combined with the above).
	SeverityAtLeast int                 `json:"severity_at_least,omitempty"` // minimum OTEL severity number (1-24); 0 = unset
	AttrExists      []string            `json:"attr_exists,omitempty"`       // keys that must be present
	AttrMatches     map[string]string   `json:"attr_matches,omitempty"`      // key → full-text term match on that attribute
	AttrIn          map[string][]string `json:"attr_in,omitempty"`           // key → allowed value set
	AttrNotIn       map[string][]string `json:"attr_not_in,omitempty"`       // key → excluded value set
	AttrGt          map[string]float64  `json:"attr_gt,omitempty"`           // key → value must be > n
	AttrGe          map[string]float64  `json:"attr_ge,omitempty"`           // key → value must be >= n
	AttrLt          map[string]float64  `json:"attr_lt,omitempty"`           // key → value must be < n
	AttrLe          map[string]float64  `json:"attr_le,omitempty"`           // key → value must be <= n
	AttrRegex       map[string]string   `json:"attr_regex,omitempty"`        // key → RE2 pattern the value must match

	// Arrival (observed-time) axis — see LogOrder. Orthogonal to Start/End, which bound event time.
	ObservedAfter int64    `json:"observed_after,omitempty"` // keep only records with observed_time > t (unix nanos); 0 = unset
	OrderBy       LogOrder `json:"order_by,omitempty"`       // time axis to ORDER BY; "" = LogOrderTime
}

LogQuery is an endpoint-shaped log query (mirrors IMBH's LogQuery builder; a curated subset). The zero value matches all logs. Times are Unix nanoseconds; 0 means unset.

type MaintenanceReport 

type MaintenanceReport struct {
	Sealed          bool   `json:"sealed"`
	SegmentsDropped uint64 `json:"segments_dropped"`
	BytesFreed      uint64 `json:"bytes_freed"`
	// PendingApplied, PendingDiscarded and PendingSegmentsReplaced report the CommitPending half of
	// the pass, which since imbh 0.8.0 runs before retention: a maintenance tick now picks up an
	// external housekeeper's prepared segment rewrites on its own, where before this a long-running
	// host applied them only at open or close. Zero throughout unless something is preparing records.
	PendingApplied          uint64 `json:"pending_applied"`
	PendingDiscarded        uint64 `json:"pending_discarded"`
	PendingSegmentsReplaced uint64 `json:"pending_segments_replaced"`
}

MaintenanceReport summarizes a maintain() pass (imbh: Db::maintain).

type Matrix 

type Matrix struct {
	Series []Series
}

Matrix is the result of a metric range query — one series per distinct group-by label set.

type MetricDimension added in v0.7.0 

type MetricDimension struct {
	Name   string
	Values []string
}

MetricDimension is one label axis of a metric: the label name and the distinct values series of that metric carry on it, both sorted.

type MetricInfo 

type MetricInfo struct {
	Metric      string
	Unit        string
	Temporality string
	Kind        string
}

MetricInfo is one metric's catalog entry: its name, unit, temporality ("" when the kind carries none, e.g. summaries), and kind ("gauge" | "sum" | "histogram" | ...).

type MetricPoint 

type MetricPoint struct {
	Time       int64 // unix nanoseconds
	Metric     string
	Service    string
	Attributes string // JSON object string
	Value      float64
}

MetricPoint is one decoded raw metric sample (scalar kinds: gauge / sum).

type MetricPointsQuery 

type MetricPointsQuery struct {
	Metric  string            `json:"metric"`
	Kind    string            `json:"kind,omitempty"`
	Filters map[string]string `json:"filters,omitempty"` // attribute equality filters (AND)
	Start   int64             `json:"start,omitempty"`
	End     int64             `json:"end,omitempty"`
	Limit   int               `json:"limit,omitempty"`
}

MetricPointsQuery selects raw (unaggregated) metric samples — the counterpart to MetricQuery, which resamples into a range. Kind is "gauge" (default), "sum", or "histogram". Times are Unix nanoseconds.

type MetricQuery 

type MetricQuery struct {
	Metric  string   `json:"metric"`             // metric name
	Sum     bool     `json:"sum,omitempty"`      // false = gauge (default), true = sum
	Step    int64    `json:"step,omitempty"`     // resample step (nanos)
	Start   int64    `json:"start,omitempty"`    // range start (unix nanos)
	End     int64    `json:"end,omitempty"`      // range end (unix nanos)
	GroupBy []string `json:"group_by,omitempty"` // attribute keys to split series on
}

MetricQuery is a metric range query over a scalar metric (gauge or sum). Times are Unix nanoseconds; Step is the resampling interval in nanoseconds. GroupBy names attribute keys to split series on — a record attribute, or the service under either spelling ("service.name" / "service"), which imbh 0.3.0 resolves to the built-in service column (earlier versions merged every service into one empty-labelled series). The same holds for LogQuery's attribute predicates and SpanMetricsQuery.

type PendingReport added in v0.8.0 

type PendingReport struct {
	// Applied counts records validated and swapped into the manifest.
	Applied uint64 `json:"applied"`
	// Discarded counts records rejected as stale, corrupt, or built against a different promoted key
	// set. Their inputs are untouched, so the only cost is the preparer's wasted work.
	Discarded uint64 `json:"discarded"`
	// SegmentsReplaced counts input segments the applied records removed.
	SegmentsReplaced uint64 `json:"segments_replaced"`
}

PendingReport summarizes a CommitPending pass (imbh 0.8.0: Db::commit_pending).

type Point 

type Point struct {
	T int64
	V float64
}

Point is one (time, value) sample. T is unix nanoseconds.

type PromotionCandidate added in v0.8.0 

type PromotionCandidate struct {
	Name string `json:"name"`
	// RowsString is how many rows a promoted column would actually populate; the candidates are
	// ordered by it, descending.
	RowsString     uint64  `json:"rows_string"`
	EstBytesPerRow float64 `json:"est_bytes_per_row"`
	// Promote is one of the Promote* constants above.
	Promote string `json:"promote"`
	// IndexScale is the widest window width at which a segment index would still prune for this key
	// (IndexScaleAll when even the whole scan qualifies), or "" when no rung does. A segment index is
	// not something this binding can build — it is reported because the measurement answers it.
	IndexScale string `json:"index_scale"`
}

PromotionCandidate is one DB-wide, record-attributes-scoped key — exactly the keys SetPromoted may name — with both verdicts. The two are independent, not branches of one: a key can want a promoted column (fast filtering) and a segment index (pruning) at once.

type QueryStats 

type QueryStats struct {
	SegmentsScanned uint64 `json:"segments_scanned"`
	SegmentsPruned  uint64 `json:"segments_pruned"`
	RowsScanned     uint64 `json:"rows_scanned"`
	RowsReturned    uint64 `json:"rows_returned"`
	BytesScanned    uint64 `json:"bytes_scanned"`
	ElapsedNs       uint64 `json:"elapsed_ns"`
	UsedIndex       bool   `json:"used_index"`
}

QueryStats reports what a query scanned. Complete only for a fully drained query (which QueryLogPage always does). Mirrors imbh's QueryStats.

type Receipt 

type Receipt struct {
	Accepted uint64
	Rejected uint64
	LSN      uint64
	Durable  bool
	Queued   bool
}

Receipt is the outcome of an ingest call. When Queued is true (async ingest), LSN/Durable carry no information yet.

type RetentionPolicy added in v0.8.0 

type RetentionPolicy struct {
	// MaxAgeNs drops segments older than this many nanoseconds. nil = no age bound.
	MaxAgeNs *uint64 `json:"max_age_ns"`
	// MaxDiskBytes caps the total on-disk segment bytes. nil = no size bound.
	MaxDiskBytes *uint64 `json:"max_disk_bytes"`
}

RetentionPolicy is the retention rules in effect (imbh 0.8.0: Db::retention). Both bounds are optional and combine: a segment is dropped if it is older than MaxAge or if the total on-disk size exceeds MaxDiskBytes (oldest first). Both nil is "keep everything".

type Rows 

type Rows struct {
	// contains filtered or unexported fields
}

Rows is a streaming, zero-copy query result. Iterate with Next until it returns ok=false, then check Err (or Close). Not safe for concurrent use: iterate from one goroutine. Each returned RecordBatch is owned by the caller — call its Release() when done.

ZERO-COPY CAVEAT: a batch's Arrow buffers are IMBH-owned and freed by Release(). Scalar values read from a batch — especially strings/[]byte via arrow-go, which alias the buffer without copying — are only valid until that batch's Release(). Copy anything you need to outlive the batch (e.g. strings.Clone for strings; QueryLogsTyped does this for you).

func (*Rows) Close 

func (r *Rows) Close()

Close releases the cursor (and cancels the query if not fully drained). Idempotent.

Abandoning a stream before end-of-stream means finish never runs, and finish is what clears this query's Rust-side error slot — so Close does it instead. Without that, a query that had already recorded a terminal error (a plan error, say, which is stored the moment the handler starts) holds that entry until the process exits: a slow leak for any long-lived program that abandons streams. See TestAbandonedStreamClearsErrorSlot.

One window remains open by design: if the handler records an error *after* this fetch — it is still running, and only a send failure makes it stop without storing — that entry is never claimed. The Rust-side fix would be to skip storing once the consumer is gone (`tx.is_closed()`), at every store site; the pending-error count in TestNoLeak is the tripwire if it ever matters in practice.

func (*Rows) Err 

func (r *Rows) Err() error

Err returns the terminal error after Next has returned ok=false: nil (clean end), a query error, or context.Canceled/DeadlineExceeded. Meaningless before iteration ends.

func (*Rows) Next 

func (r *Rows) Next() (rec arrow.RecordBatch, ok bool, err error)

Next pulls the next result batch. ok=false marks the end of iteration; then Err reports whether it ended cleanly (nil), on a query error, or on context cancellation. On ok=true the RecordBatch wraps IMBH-allocated Arrow buffers zero-copy; the caller must Release() it.

type SegmentRef 

type SegmentRef struct {
	RelativePath    string `json:"relative_path"`
	MinTimeUnixNano int64  `json:"min_time_unix_nano"`
	MaxTimeUnixNano int64  `json:"max_time_unix_nano"`
	Rows            uint64 `json:"rows"`
}

SegmentRef identifies one on-disk segment and its covered time range (imbh: Db::segments).

type Series 

type Series struct {
	Labels map[string]string
	Points []Point
}

Series is one metric time series: its label set and its samples over time.

type SigmaSummary added in v0.8.0 

type SigmaSummary struct {
	P50  float64 `json:"p50"`
	P90  float64 `json:"p90"`
	Max  float64 `json:"max"`
	Mean float64 `json:"mean"`
	// FracLE25 is the fraction of values whose sigma is at or below 0.25 — the long tail a segment
	// index exploits.
	FracLE25 float64 `json:"frac_le_25"`
	// Histogram counts values in ten equal-width sigma buckets, [0,0.1) .. [0.9,1.0].
	Histogram [10]uint32 `json:"histogram"`
	Count     int        `json:"count"`
}

SigmaSummary is the sigma distribution for one key: one sample per distinct value, unweighted by how often that value occurs. Sigma is the fraction of a table's segments that contain a given (key, value) pair, so a segment index prunes 1 - sigma. Near 1 means the index buys nothing; near 1/segments means it prunes almost everything.

type SnapshotInfo 

type SnapshotInfo struct {
	Dir      string `json:"dir"`
	Segments uint64 `json:"segments"`
}

SnapshotInfo describes a snapshot written by Snapshot (imbh: Db::snapshot).

type Span 

type Span struct {
	TraceID       []byte
	SpanID        []byte
	ParentSpanID  []byte
	Name          string
	Kind          string
	StartTime     int64 // unix nanoseconds
	DurationNs    int64
	StatusCode    string
	StatusMessage string
	Service       string
	Attributes    string // JSON object string
}

Span is one decoded span of a trace.

type SpanMetricPoint 

type SpanMetricPoint struct {
	Bucket int64 // bucket start, unix nanoseconds
	Labels map[string]string
	Calls  uint64
	Errors uint64
	P50    float64 // latency percentiles, nanoseconds
	P95    float64
	P99    float64
}

SpanMetricPoint is one bucket of RED span metrics for a label set.

type SpanMetricsQuery 

type SpanMetricsQuery struct {
	Service string   `json:"service,omitempty"`
	Name    string   `json:"name,omitempty"`     // span name
	Kind    string   `json:"kind,omitempty"`     // span kind
	Status  string   `json:"status,omitempty"`   // status code filter
	GroupBy []string `json:"group_by,omitempty"` // attribute keys to split on
	Step    int64    `json:"step,omitempty"`     // bucket width (nanos)
	Start   int64    `json:"start,omitempty"`
	End     int64    `json:"end,omitempty"`
}

SpanMetricsQuery is a span (RED) metrics query: calls / errors / latency percentiles over spans, bucketed by Step (nanos) and optionally split by GroupBy attribute keys.

type Stats 

type Stats = sable.Stats

Stats is a snapshot of the fused runtime's counters.

func RuntimeStats 

func RuntimeStats() Stats

RuntimeStats returns a snapshot of runtime counters (InFlight, Rejected, MaxInFlight, …) for backpressure tuning and observability.

type Table 

type Table string

Table names one of imbh's storage tables (logs, spans, and the five metric families). The string values match imbh's Table::as_str form used on the wire. 

const (
	TableLogs                Table = "logs"
	TableSpans               Table = "spans"
	TableMetricsGauge        Table = "metrics_gauge"
	TableMetricsSum          Table = "metrics_sum"
	TableMetricsHistogram    Table = "metrics_histogram"
	TableMetricsExpHistogram Table = "metrics_exp_histogram"
	TableMetricsSummary      Table = "metrics_summary"
)

type TableStats 

type TableStats struct {
	Table           string `json:"table"`
	SegmentCount    uint64 `json:"segment_count"`
	SegmentRows     uint64 `json:"segment_rows"`
	BufferRows      uint64 `json:"buffer_rows"`
	MinTimeUnixNano *int64 `json:"min_time_unix_nano"`
	MaxTimeUnixNano *int64 `json:"max_time_unix_nano"`
}

TableStats is per-table storage accounting within DbStats.

type TraceMatch 

type TraceMatch struct {
	TraceID string
	SpanIDs []string
}

TraceMatch is a TraceQL match: a trace and the span ids its spanset selected.

type TraceNode 

type TraceNode struct {
	Span
	Children []*TraceNode
}

TraceNode is one span in an assembled trace tree. It embeds the decoded Span and holds the spans whose ParentSpanID names this span's SpanID.

func AssembleTrace 

func AssembleTrace(spans []Span) []*TraceNode

AssembleTrace rebuilds the parent→child forest from a flat span slice and returns the roots.

A span becomes a root when its ParentSpanID is empty OR when its parent's SpanID is not present in the input (an orphan is surfaced as a root so no span is ever dropped). Every other span is attached under the node whose SpanID equals its ParentSpanID. Spans are keyed by string(SpanID); on a duplicate SpanID the first occurrence wins and later ones are discarded.

The input slice is not mutated: nodes are built from copies of the Span values. Children of each node, and the returned roots, are sorted by StartTime then by string(SpanID) so the output is stable regardless of input order. An empty or nil input returns nil.

type TraceQuery 

type TraceQuery struct {
	Service       string            `json:"service,omitempty"`
	Name          string            `json:"name,omitempty"`
	Status        string            `json:"status,omitempty"`
	Kind          string            `json:"kind,omitempty"`
	MinDurationNs int64             `json:"min_duration_ns,omitempty"`
	MaxDurationNs int64             `json:"max_duration_ns,omitempty"`
	AttrEq        map[string]string `json:"attr_eq,omitempty"`

	// Attribute predicates (parity with LogQuery; all AND-combined with the above).
	AttrExists  []string            `json:"attr_exists,omitempty"`  // keys that must be present
	AttrMatches map[string]string   `json:"attr_matches,omitempty"` // key → full-text term match on that attribute
	AttrIn      map[string][]string `json:"attr_in,omitempty"`      // key → allowed value set
	AttrNotIn   map[string][]string `json:"attr_not_in,omitempty"`  // key → excluded value set
	AttrGt      map[string]float64  `json:"attr_gt,omitempty"`      // key → value must be > n
	AttrGe      map[string]float64  `json:"attr_ge,omitempty"`      // key → value must be >= n
	AttrLt      map[string]float64  `json:"attr_lt,omitempty"`      // key → value must be < n
	AttrLe      map[string]float64  `json:"attr_le,omitempty"`      // key → value must be <= n
	AttrRegex   map[string]string   `json:"attr_regex,omitempty"`   // key → RE2 pattern the value must match

	Start int64 `json:"start,omitempty"`
	End   int64 `json:"end,omitempty"`
	Limit int64 `json:"limit,omitempty"`
}

TraceQuery selects traces for SearchTraces. All fields are optional: a field is applied only when non-empty/non-zero, so a zero TraceQuery matches everything (up to the server's default limit). The JSON tags match the Rust `TraceQueryWire`.

type TraceSummary 

type TraceSummary struct {
	TraceID     string
	RootService string
	RootName    string
	StartTime   int64
	DurationNs  int64
	SpanCount   int64
	Error       bool
}

TraceSummary is one matched trace's summary row. RootService/RootName are "" when the root span carries none (SQL-NULL upstream). StartTime is unix nanoseconds; DurationNs is the trace's wall duration in nanoseconds.

type UnitReport added in v0.8.0 

type UnitReport struct {
	Label    string `json:"label"`
	Segments uint32 `json:"segments"`
	Rows     uint64 `json:"rows"`
	// KeysTracked, KeysSampleRate and KeysEst mirror the per-key sampling fields at key granularity
	// (MaxKeys is the cap that engages here).
	KeysTracked    int     `json:"keys_tracked"`
	KeysSampleRate float64 `json:"keys_sample_rate"`
	KeysEst        float64 `json:"keys_est"`
	// Windows is the windows opened per ladder level within this unit — the denominator behind Curve,
	// which differs per table.
	Windows   []uint32 `json:"windows"`
	SpanNanos int64    `json:"span_nanos"`
	// Keys is ordered most expensive segment index first (descending PostingsEst).
	Keys []KeyReport `json:"keys"`
}

UnitReport is one scan unit — a single table, or the whole database.

func (UnitReport) Key added in v0.8.0 

func (u UnitReport) Key(name string) *KeyReport

Key returns the named key's report within this unit, or nil.

func (UnitReport) RowsPerSegment added in v0.8.0 

func (u UnitReport) RowsPerSegment() float64

RowsPerSegment is the mean rows per segment — the ceiling on how much any value can repeat within one, and why a small corpus yields no promote verdict.

type VolumeBucket 

type VolumeBucket struct {
	Time   int64
	Labels string
	Count  int64
}

VolumeBucket is one time bucket of a log-volume query: the bucket start (unix nanos), the label set identifying this bucket's series as canonical JSON ("{}" when un-grouped), and the record count.

Source Files

View all Source files

Directories

Path Synopsis
cmd
imbhgo-fetch command
Command imbhgo-fetch downloads the prebuilt combined static library (libimbhgo.a) for the current platform so a consumer can build github.com/moriyoshi/imbh-go WITHOUT building the Rust side.
 Command imbhgo-fetch downloads the prebuilt combined static library (libimbhgo.a) for the current platform so a consumer can build github.com/moriyoshi/imbh-go WITHOUT building the Rust side.
internal
release
Package release holds the release coordinates for the prebuilt libimbhgo.a archives: the version this source corresponds to and the naming/URL scheme for the assets published on GitHub Releases.
 Package release holds the release coordinates for the prebuilt libimbhgo.a archives: the version this source corresponds to and the naming/URL scheme for the assets published on GitHub Releases.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.