ingot

README

ingot

An embedded time-series database for Go. SQLite for metrics.

db, _ := ingot.Open("./data", ingot.Options{
    Retention:     30 * 24 * time.Hour,
    BlockDuration: 2 * time.Hour,
})

// Write
app := db.Appender()
ref, _ := app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3)
app.Append(ref, nil, ts+15000, 71.4) // ref fast-path skips label hashing
app.Commit()

// Read
q, _ := db.Querier(mint, maxt)
ss := q.Select(labels.MustNewMatcher(labels.MatchEqual, "room", "office"))
for ss.Next() {
    it := ss.At().Iterator()
    for it.Next() {
        t, v := it.At()
        _ = t; _ = v
    }
}
q.Close()

db.Close()

Status

Alpha. The API is frozen (M4) and the system survives a 48h soak test under sustained load (M5), but this hasn't seen production use yet.

Milestone	State
M1 — Gorilla chunk encoding, fuzzed, benchmarked	Done
M2 — Head + WAL, kill -9 safe	Done
M3 — Immutable blocks, mmap reads	Done
M4 — Query path, API freeze, shippable alpha	Done
M5 — Compaction + retention, 48h soak	Done
M6 — ingotctl, HTTP layer, self-instrumentation	Done

See DESIGN.md for architecture, on-disk format, and the non-goals table. See ROADMAP.md for what's next.

Why

I needed a library to store time-series data locally and the options out there didn't quite work for my case. Prometheus was too heavy for what I needed but I wanted that level of compression. tstorage was close but it didn't have the compression or label indexing.

Install

go get github.com/davidhrinaldo/ingot

Features

• Gorilla XOR compression — ~1 byte/sample on regular metric data (see benchmarks below)
• Crash-safe — WAL with CRC32C records. Committed data is persisted
• Query by label matchers — equality, negation, regex, negative regex; merged across head and blocks
• Levelled compaction — 2h → 8h → 32h blocks, background merging, retention-based expiry
• Self-instrumentation — the DB records its own metrics (series/chunk counts, compactions, WAL fsync duration) through the normal write path, queryable like any other series
• Zero external dependencies

Tools

ingotctl

CLI for block inspection and diagnostics:

ingotctl blocks ./data                    # list blocks with ULID, time range, stats
ingotctl inspect ./data/01HXYZ.../        # series labels, chunk metadata, postings stats
ingotctl chunks ./data/01HXYZ.../ 42      # decode and print raw samples for series ref 42
ingotctl fsck ./data                      # CRC + index integrity check on all blocks

ingothttp

Minimal HTTP query server for Grafana integration:

ingothttp -data ./data -addr :9001

Endpoints:

• GET /api/v1/query_range?query=<name>&start=<ms>&end=<ms> — Prometheus-style JSON matrix response
• POST /api/v1/read — JSON read request with label matchers
• GET /api/v1/status — DB stats snapshot

This is a demo/bridge, not a full PromQL engine. The query parameter matches __name__ by equality.

Compression

Chunk encoding is Gorilla (Pelkonen et al., VLDB 2015): delta-of-delta timestamps, XOR floats. Measured on 120-sample chunks at a regular 15s interval:

Workload	bytes/sample
Constant value	0.42
Stepped sensor (repeats, occasional 0.1 steps)	~1.0
Integer counter	~2
Full-precision random walk (adversarial)	~7.5

Regenerate: go test -v -run TestBytesPerSample ./internal/chunkenc/

Two things worth knowing about XOR compression that the headline numbers hide:

• Decimal quantization doesn't help. 0.1-precision readings have mantissas as dirty as full-precision ones — 0.1 is non-terminating in binary. Compression comes from exact repeats (1 bit) and integer values (clean trailing zeros), not from "roundness" in base 10.
• The famous 1.37 bytes/sample figure assumes production metric traffic, where roughly half of consecutive values repeat exactly. Your data may not look like that; the adversarial row is what you pay when it doesn't.

Layout

ingot/                  public API: Open, Appender, Querier
├── internal/
│   ├── chunkenc/       Gorilla encoder/decoder, bitstream
│   ├── wal/            segmented write-ahead log
│   ├── head/           in-memory series, active chunks
│   ├── index/          symbols, postings, matchers
│   ├── block/          immutable block read/write, validation
│   ├── compact/        levelled merge + retention
│   └── postings/       sorted posting list operations
├── cmd/
│   ├── ingotctl/       block inspection, fsck
│   └── ingothttp/      HTTP query server
└── labels/             label types

Development

go test -race -short ./...                                           # all tests (skip soak)
go test -race ./...                                                  # all tests including soak (~5 min)
go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/     # fuzz the decoder
go test -bench=. ./internal/chunkenc/                                # benchmarks
go build ./cmd/ingotctl/                                             # build CLI tool
go build ./cmd/ingothttp/                                            # build HTTP server

The decoder is total: arbitrary bytes produce values or ErrShortStream and never panics. Fuzzing gates every change to chunkenc.

Inspiration

Most of the design is lifted from Prometheus TSDB — chunk encoding, index format, block/compaction model, label data model. The WAL is simpler (no page-level framing). The difference is that Prometheus TSDB is a storage engine inside a server; ingot is a library. See NOTICE.md.

The chunk encoding comes from the Gorilla paper (Pelkonen et al., VLDB 2015) via Prometheus, which adapted the bit-width buckets for millisecond timestamps. ingot uses the Prometheus variant.

tstorage is the closest existing embedded TSDB for Go. It doesn't do Gorilla compression or label-based indexing, which is most of why ingot exists.

Non-goals

Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows (sorry not my thing). Check DESIGN.md for reasoning.

License

Apache 2.0

Documentation

Overview

Package ingot is an embedded time-series database library for Go.

Index

• Constants
• type Appender
  • func (a *Appender) Append(ref uint64, ls []labels.Label, t int64, v float64) (uint64, error)
  • func (a *Appender) Commit() error
  • func (a *Appender) Rollback() error
• type DB
  • func Open(dataDir string, opts Options) (*DB, error)
  • func (db *DB) Appender() *Appender
  • func (db *DB) ApplyRetention()
  • func (db *DB) Close() error
  • func (db *DB) FlushOlderThan(maxT int64) (string, error)
  • func (db *DB) Querier(mint, maxt int64) (*Querier, error)
  • func (db *DB) RunCompaction() error
  • func (db *DB) Stats() DBStats
• type DBStats
• type Options
• type Querier
  • func (q *Querier) Close() error
  • func (q *Querier) Select(matchers ...*labels.Matcher) SeriesSet
• type SampleIterator
• type Series
• type SeriesSet

Constants

const (
	MetricHeadSeries        = "ingot_head_series"
	MetricHeadChunksActive  = "ingot_head_chunks_active"
	MetricBlocksTotal       = "ingot_blocks_total"
	MetricCompactionsTotal  = "ingot_compactions_total"
	MetricWALFsyncDurationS = "ingot_wal_fsync_duration_seconds"
)

Self-instrumentation metric names.

Types

type Appender

type Appender struct {
	// contains filtered or unexported fields
}

Appender buffers samples and new series for atomic commit.

func (*Appender) Append

func (a *Appender) Append(ref uint64, ls []labels.Label, t int64, v float64) (uint64, error)

Append adds a sample. If ref is 0, the series is resolved (or created) from ls.

func (*Appender) Commit

func (a *Appender) Commit() error

Commit writes the batch to the WAL and applies it to the head.

func (*Appender) Rollback

func (a *Appender) Rollback() error

Rollback discards the batch.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is an embedded time-series database.

func Open

func Open(dataDir string, opts Options) (*DB, error)

Open opens or creates a DB at the given directory.

func (*DB) Appender

func (db *DB) Appender() *Appender

Appender returns a new Appender for batching writes.

func (*DB) ApplyRetention

func (db *DB) ApplyRetention()

ApplyRetention drops blocks whose data is older than the retention window. Exported for testing.

func (*DB) Close

func (db *DB) Close() error

Close closes the DB, releasing all resources.

func (*DB) FlushOlderThan

func (db *DB) FlushOlderThan(maxT int64) (string, error)

FlushOlderThan flushes sealed head chunks to an immutable block.

func (*DB) Querier

func (db *DB) Querier(mint, maxt int64) (*Querier, error)

Querier returns a Querier over [mint, maxt].

func (*DB) RunCompaction

func (db *DB) RunCompaction() error

RunCompaction performs a single compaction cycle. Exported for testing.

func (*DB) Stats

func (db *DB) Stats() DBStats

Stats returns a snapshot of database statistics.

type DBStats

type DBStats struct {
	HeadSeries  int
	HeadChunks  int
	Blocks      int
	Compactions int
}

DBStats holds summary statistics for the database.

type Options

type Options struct {
	Retention     time.Duration
	BlockDuration time.Duration
	// Clock returns the current time in milliseconds. Defaults to
	// time.Now().UnixMilli(). Injected for testing with simulated time.
	Clock func() int64
}

Options configures a DB.

type Querier

type Querier struct {
	// contains filtered or unexported fields
}

Querier queries the DB over a time range.

func (*Querier) Close

func (q *Querier) Close() error

Close releases block references held by this querier.

func (*Querier) Select

func (q *Querier) Select(matchers ...*labels.Matcher) SeriesSet

Select returns a SeriesSet matching the given matchers.

type SampleIterator

type SampleIterator interface {
	Next() bool
	At() (int64, float64)
	Err() error
}

SampleIterator iterates over samples.

type Series

type Series interface {
	Labels() []labels.Label
	Iterator() SampleIterator
}

Series represents a single time series.

type SeriesSet

type SeriesSet interface {
	Next() bool
	At() Series
	Err() error
}

SeriesSet iterates over query results.