scandex

package module
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2026 License: MIT Imports: 23 Imported by: 0

README

scandex

scandex is an embedded, append-only inverted index for Go. It stores one kind of fact:

(value_type, field_name, field_value)  ->  document_id

and nothing else. There is no forward document -> values store. That single decision shapes the whole library: writes are cheap appends, the index doubles as a column store, and reads are linear scans that exploit sort order. On top of this substrate scandex layers a JSON document indexer, a typed query engine, and a group-by/aggregation engine.

It is a library, not a server. You embed it in your process, point it at a directory, and it manages the files.

import "github.com/gur-shatz/scandex"

Requires Go 1.23+. The only dependency is statekit, used by the optional metrics layer.

Contents

Data model

Every indexed fact is a tuple mapping a typed field/value to a document id. A single JSON document becomes many tuples, one per leaf value. For example:

{ "id": "evt-1", "type": "PushEvent", "actor": { "login": "alice" } }

flattens to dotted paths and produces (conceptually):

(String, "type",        "PushEvent") -> evt-1
(String, "actor.login", "alice")     -> evt-1

Strings, field paths, and document ids are interned through dictionaries and stored as uint64 ids, so the index itself is a compact stream of integer tuples. Because the tuples are sorted by (value_type, field_name, field_value), the slice of tuples for one field is exactly that field's column — which is why aggregation is a multi-column join on document_id rather than a forward lookup.

Value types are DocumentNull, DocumentBool, DocumentUint, DocumentInt, DocumentFloat, DocumentString, and DocumentArray. A field a document lacks is simply absent from the index (and folds into the Null group during aggregation). Arrays are stored whole as canonical JSON, not exploded per element.

Architecture

scandex is a stack of layers; you can use any layer directly.

Layer Package What it gives you
Tuple index scandex Open, Writer.Add, Scan, Search over raw (uint64…) -> id tuples
Document indexer scandex DocumentIndexer flattens JSON/maps into the tuple index
Query engine documentquery typed predicates (Eq, Gt, Between, And…) returning matching document ids
Aggregation documentquery/analytics GROUP BY + COUNT/SUM in two data passes
Metrics scandex (+ statekit) attachable counters/gauges for reader, writer, aggregation
Dictionaries internal/stringmap interning of strings/fields/document ids

The document layers all sit on the fixed three-field DocumentIndexSchema() = (value_type, field_name, field_value). The tuple layer is generic: you choose your own schema and uint64 encoding.

Quick start

Index two documents, build the index, then query and aggregate them.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/gur-shatz/scandex"
	"github.com/gur-shatz/scandex/documentquery"
	"github.com/gur-shatz/scandex/documentquery/analytics"
)

func main() {
	// 1. Open a document index in ./data with the storage prefix "events".
	db, err := scandex.Open("data", scandex.DocumentIndexSchema(), scandex.WithPrefix("events"))
	if err != nil {
		log.Fatal(err)
	}

	ix, err := scandex.OpenDocumentIndexer(db)
	if err != nil {
		log.Fatal(err)
	}
	mustAdd := func(doc map[string]any) {
		if _, _, err := ix.AddDocument(doc); err != nil {
			log.Fatal(err)
		}
	}
	mustAdd(map[string]any{"id": "evt-1", "type": "PushEvent", "actor": map[string]any{"login": "alice"}})
	mustAdd(map[string]any{"id": "evt-2", "type": "PushEvent", "actor": map[string]any{"login": "bob"}})
	mustAdd(map[string]any{"id": "evt-3", "type": "IssuesEvent", "actor": map[string]any{"login": "alice"}})

	if err := ix.Commit(scandex.Sync); err != nil {
		log.Fatal(err)
	}
	if err := ix.Close(); err != nil { // releases the writer lock
		log.Fatal(err)
	}
	if _, err := db.BuildIndex(); err != nil { // append log -> sorted segments
		log.Fatal(err)
	}
	db.Close()

	// 2. Query.
	eng, err := documentquery.Open("data", "events")
	if err != nil {
		log.Fatal(err)
	}
	defer eng.Close()

	res, err := eng.Search(context.Background(), documentquery.SourceQuery{
		Where: documentquery.Eq("type", documentquery.String("PushEvent")),
		Limit: 10,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("PushEvent docs:", res.Documents) // [evt-1 evt-2]

	// 3. Aggregate: count events per actor.
	agg, err := analytics.New(eng).Aggregate(context.Background(), analytics.Query{
		GroupBy: []string{"actor.login"},
		Aggs:    []analytics.Agg{{Op: analytics.Count}},
		OrderBy: []analytics.Sort{{OnAgg: true, Index: 0, Desc: true}},
	})
	if err != nil {
		log.Fatal(err)
	}
	for _, row := range agg.Rows {
		fmt.Printf("%s: %d\n", row.Keys[0], row.Count) // alice: 2 / bob: 1
	}
}

Indexing documents

OpenDocumentIndexer acquires the database's single writer and gives you three ways to add data. Every document must carry an id field, which becomes its source identifier.

db, _ := scandex.Open("data", scandex.DocumentIndexSchema(), scandex.WithPrefix("events"))
ix, _ := scandex.OpenDocumentIndexer(db)

// From a Go map. Returns (internalDocID, entriesWritten, error).
docID, n, err := ix.AddDocument(map[string]any{
	"id":      "evt-1",
	"type":    "PushEvent",
	"payload": map[string]any{"size": 5},
})

// From raw JSON bytes (decoded with UseNumber, then AddDocument).
_, _, err = ix.AddJSON([]byte(`{"id":"evt-2","type":"IssuesEvent"}`))

// A single pre-flattened entry, when you already know the path/type/value.
err = ix.AddFlatEntry("evt-3", "type", scandex.DocumentString, `"PushEvent"`)

Then make the data durable and queryable:

ix.Commit(scandex.Sync) // flush the append log; Sync fsyncs, Async does not
ix.Close()              // release the writer lock
db.BuildIndex()         // compact the append log into immutable sorted segments

Commit makes writes visible to readers (they scan the unsorted tail); BuildIndex turns that tail into sorted segments that readers can seek through. You can keep appending after a build — a later BuildIndex only processes the uncovered tail.

For high-volume loads, size the id caches to your cardinality:

ix, _ := scandex.OpenDocumentIndexer(db, scandex.WithDocumentIndexCaches(
	1<<20, // document-id cache entries
	1<<16, // field-path cache entries
	1<<20, // value cache entries
))

Querying documents

documentquery.Open(dir, prefix, opts...) opens a read engine over an existing index. Engine.Search evaluates a SourceQuery and returns the matching source document ids.

eng, _ := documentquery.Open("data", "events")
defer eng.Close()

res, err := eng.Search(context.Background(), documentquery.SourceQuery{
	Where: documentquery.And(
		documentquery.Eq("type", documentquery.String("PushEvent")),
		documentquery.Gt("payload.size", documentquery.Uint(3)),
	),
	Limit: 100, // 0 means no limit
})
// res.Documents []string  – source ids
// res.IDs       []uint64   – internal ids
// res.Stats                – segments/entries scanned, hits, duration
// res.Plan                 – how the query was translated/executed
Predicates

Build a documentquery.Expr from these constructors:

Constructor Meaning
Eq(field, v) field equals value
Ne(field, v) field exists and is not value
In(field, v1, v2, …) field equals any value
Gt / Gte / Lt / Lte(field, v) numeric/ordered comparison
Between(field, lo, hi) inclusive range
Exists(field) field is present
And(…) / Or(…) / Not(…) boolean composition

Values are typed: String, Array, Bool, Uint, Int, Float, Null. Field paths use dotted notation (actor.login, payload.size).

documentquery.Or(
	documentquery.Eq("type", documentquery.String("PushEvent")),
	documentquery.Between("payload.size", documentquery.Uint(10), documentquery.Uint(100)),
)

Aggregations

analytics.New(eng).Aggregate runs a GROUP BY with COUNT and SUM in exactly two data passes: one to find matching documents, one linear scan to fill the referenced columns. The reduce, ordering, and limiting happen in memory.

res, err := analytics.New(eng).Aggregate(context.Background(), analytics.Query{
	Where:   documentquery.Eq("type", documentquery.String("PushEvent")), // nil = all docs
	GroupBy: []string{"type", "actor.login"},
	Aggs: []analytics.Agg{
		{Op: analytics.Count},
		{Op: analytics.Sum, Field: "payload.size", Name: "bytes"},
	},
	Having:    func(r analytics.Row) bool { return r.Count >= 2 }, // optional
	OrderBy:   []analytics.Sort{{OnAgg: true, Index: 0, Desc: true}}, // top by count
	Limit:     20,   // max groups returned (0 = all)
	MaxGroups: 1<<20, // guardrail on group cardinality
})

for _, row := range res.Rows {
	// row.Keys []Cell (one per GroupBy), row.Vals []Num (one per Agg), row.Count uint64
	fmt.Printf("%v  count=%d  bytes=%s\n", row.Keys, row.Count, row.Vals[1])
}
fmt.Printf("matched=%d groups=%d in %s\n", res.Stats.Matched, res.Stats.Groups, res.Stats.Duration)

Notes:

  • Limit bounds the number of groups, not documents.
  • Sort.OnAgg selects an aggregation column by Index; otherwise Index selects a group-by column. Desc reverses order.
  • SUM stays exact in int64 while a column is all-integer, promoting to float64 only on the first float value or on int64 overflow. Results come back as a Num (IsInt, Int, Float, with Float64() and String() helpers).
  • A document missing a group field folds into the Null group.

See documentquery/analytics/README.md for the execution model in depth.

The low-level tuple index

Skip the document layers entirely when you want to control the schema and the uint64 encoding yourself. Define a schema, append tuples, and query with integer terms.

schema := scandex.Schema{Fields: []scandex.FieldDef{{Name: "user"}, {Name: "action"}}}
db, _ := scandex.Open("raw", schema)

w, _ := db.Writer() // single writer; flock-guarded
w.Add(scandex.Tuple{1, 42}, 1001) // (user=1, action=42) -> id 1001
w.Add(scandex.Tuple{1, 99}, 1002)
w.Commit(scandex.Sync)
w.Close()
db.BuildIndex()

// Predicate search. Terms wrap a uint64 with scandex.U.
res, _ := db.Search(context.Background(), scandex.Query{
	Where:  scandex.And(scandex.Eq("user", scandex.U(1)), scandex.Eq("action", scandex.U(42))),
	Limits: scandex.Limits{MaxHits: 100},
})
fmt.Println(res.IDs) // [1001]

Predicate constructors mirror the document layer but operate on Terms: Eq, In, RangePred, And, Or, Not, with U(uint64) to build terms.

For full control, Scan walks every tuple and lets a callback steer the cursor — sorted segments support ScanSeek/ScanSkipValue/ScanStop to skip ahead:

stats, _ := db.Scan(context.Background(), func(e scandex.ScanEntry) scandex.ScanDecision {
	// e.Tuple, e.ID, e.Sorted
	return scandex.ScanDecision{} // zero value = continue
})

This is the substrate the documentquery engine is built on.

Metrics

scandex maintains an optional, shareable metrics bundle backed by statekit. It is attached with an option and registered from the outside; a single bundle is safe to share across concurrent readers and a writer because every metric is atomic. Metrics are folded in once per completed operation, so the per-tuple scan loops pay nothing.

import "github.com/gur-shatz/statekit"

m := scandex.NewMetrics() // optionally scandex.WithSlowQueryThreshold(d)

db, _ := scandex.Open("data", scandex.DocumentIndexSchema(),
	scandex.WithPrefix("events"), scandex.WithMetrics(m))
eng, _ := documentquery.Open("data", "events", scandex.WithMetrics(m)) // shares m

// Register from the outside and expose over HTTP.
reg := statekit.NewRegistry()
reg.RegisterCollectors(m.Collectors()...)

mux := http.NewServeMux()
reg.Mount(mux, "/") // serves /metrics, /state, /health
http.ListenAndServe(":9090", mux)

The bundle tracks raw counts (scans, searches, entries, hits, documents added, commits, rotations, writer open/sessions) and derived state maintained inside the recording methods: the index-vs-AOF read split, the running scandex_aof_read_fraction (a compaction-pressure signal), scandex_read_selectivity, and slow-read/slow-aggregation counts against the configured threshold. All metrics are exported under the scandex_ prefix.

Storage, lifecycle, and concurrency

  • Append-only log (AOF). Writer.Add appends encoded records to an AOF file. Writes become visible to readers on Commit. Readers scan the unsorted tail directly (this is WithScanTail, on by default).
  • Immutable segments. db.BuildIndex() compacts AOF ranges into sorted, CRC-verifiable index segments. Segments are seekable, so predicate search and the aggregation fill pass skip past irrelevant tuples instead of reading them. Builds are incremental: only the uncovered tail is processed.
  • Rotation. The active AOF rotates when it reaches WithIndexGranularity entries.
  • Recovery. With WithRecoverOnOpen (default), opening the database scans the AOF tail and truncates a torn final record from a crash.
  • One writer, many readers. A database has at most one writer at a time, guarded by both an in-process lock and an OS flock; a second db.Writer() returns ErrWriterLocked. The writer may be closed and reopened freely (the metrics scandex_writer_open gauge tracks this). Readers (Scan, Search, documentquery, analytics) hold no locks and are safe to run concurrently across goroutines.

Options

Passed to scandex.Open(dir, schema, opts...) and forwarded by documentquery.Open(dir, prefix, opts...):

Option Default Effect
WithPrefix(string) "" namespace for all storage files in the directory
WithIndexGranularity(int) 75,000,000 entries per AOF file before rotation
WithBlockSizeTarget(int) 64 KiB target compressed block size in segments
WithVerifyPolicy(VerifyMagic|VerifyCRC|VerifyOpen) VerifyMagic how aggressively segment integrity is checked on read
WithMaxOpenSegments(int) 256 open segment file-descriptor cap
WithScanTail(bool) true also scan the unsorted AOF tail on reads
WithRecoverOnOpen(bool) true scan/repair the AOF tail when opening
WithLogger(func(level, msg string, kv ...any)) nil structured log sink
WithMetrics(*Metrics) nil attach a metrics bundle

Document-indexer-specific: WithDocumentIndexCaches(documents, fields, values int). Metrics-specific: WithSlowQueryThreshold(time.Duration) (default 250ms; ≤0 disables slow counting).

Examples

Three runnable programs live under examples/, each driven by make targets (see the Makefile).

examples/github_events — GitHub event stream

End-to-end document indexing, search, and aggregation over the GitHub Events dataset (newline-delimited JSON). Demonstrates the flatten → AOF → BuildIndex pipeline, configurable id caches, and pprof hooks.

make example-github-events-flat        # JSON -> flattened TSV
make example-github-events-rebuild     # clean, index, build immutable segments
make example-github-events-query  QUERY_FIELD=type QUERY_VALUE=PushEvent QUERY_LIMIT=10
make example-github-events-aggregate AGG_GROUP=type,actor.login AGG_SUM=payload.size
examples/bluesky — Bluesky firehose at scale

Loads the ClickHouse JSONBench Bluesky dataset (~1M Jetstream events per gzipped file) straight from compressed JSON, synthesizing document ids from did + time_us. Shows large-scale ingestion, incremental index building during a load (-build-interval), and aggregation over real social data.

make example-bluesky-full              # download (if needed), load 1M events, build index
make example-bluesky-index BSKY_LIMIT=10000
make example-bluesky-aggregate BSKY_AGG_GROUP=commit.collection
make example-bluesky-query BSKY_QUERY_FIELD=commit.collection BSKY_QUERY_VALUE=app.bsky.feed.post

Or directly:

go run ./examples/bluesky -limit 1000000 -build-index
go run ./examples/bluesky -aggregate -agg-group kind

See examples/bluesky/README.md for dataset notes.

examples/web_analytics — embedded HTTP analytics UI

A small browser UI over an existing index, showing scandex used as an embedded analytics backend: field discovery (via db.Scan), equality search, and group-by aggregates with optional sum.

go run ./examples/web_analytics -db examples/github_events/output -prefix github_events
# then open http://127.0.0.1:8090

Development

go build ./...
go test ./...

The Makefile holds the example pipelines and common developer tasks.

License

MIT. See LICENSE.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrSchemaMismatch       = errors.New("scandex: schema hash mismatch")
	ErrWriterLocked         = errors.New("scandex: writer lock held")
	ErrBudgetExceeded       = errors.New("scandex: tracked identifier budget exceeded")
	ErrUnsupportedPredicate = errors.New("scandex: predicate shape not supported in this mode")
	ErrBadOption            = errors.New("scandex: invalid option")
	ErrCorrupt              = errors.New("scandex: corruption detected")
)

Functions

func ReadFlatDocuments 

func ReadFlatDocuments(r io.Reader, add func(docID, path string, typ DocumentValueType, rawValue string) error) (entries int, err error)

func WriteFlatDocuments 

func WriteFlatDocuments(w io.Writer, r io.Reader, limit int) (documents int, entries int, err error)

Types

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func Open 

func Open(dir string, schema Schema, opts ...Option) (*DB, error)

func (*DB) BuildIndex 

func (db *DB) BuildIndex() ([]Range, error)

func (*DB) Close 

func (db *DB) Close() error

func (*DB) Indexes 

func (db *DB) Indexes() ([]IndexInfo, error)

func (*DB) Manifest 

func (db *DB) Manifest() (Manifest, error)

func (*DB) Metrics 

func (db *DB) Metrics() *Metrics

Metrics returns the Metrics bundle attached via WithMetrics, or nil. The returned value is safe to call recording methods on even when nil.

func (*DB) ReadManifest 

func (db *DB) ReadManifest() (Manifest, bool, error)

func (*DB) Scan 

func (db *DB) Scan(ctx context.Context, fn ScanFunc) (stats SearchStats, err error)

func (*DB) ScanWithOptions 

func (db *DB) ScanWithOptions(ctx context.Context, opts ExecutionOptions, fn ScanFunc) (stats SearchStats, err error)

func (*DB) Search 

func (db *DB) Search(ctx context.Context, q Query) (res *Result, err error)

func (*DB) WriteManifest 

func (db *DB) WriteManifest(mode SyncMode) error

func (*DB) WriteManifestForNextEntry 

func (db *DB) WriteManifestForNextEntry(nextEntry uint64, mode SyncMode) error

func (*DB) Writer 

func (db *DB) Writer() (*Writer, error)

type DocumentCommitStats 

type DocumentCommitStats struct {
	Fields    StringDictionarySaveStats
	Documents StringDictionarySaveStats
	AOF       time.Duration
	Total     time.Duration
}

type DocumentIndexCacheStats 

type DocumentIndexCacheStats struct {
	Documents gencache.Stats
	Fields    gencache.Stats
	Values    gencache.Stats
}

type DocumentIndexOption 

type DocumentIndexOption func(*documentIndexOptions)

func WithDocumentIndexCaches 

func WithDocumentIndexCaches(documents, fields, values int) DocumentIndexOption

type DocumentIndexer 

type DocumentIndexer struct {
	Strings   *StringDictionary
	Fields    *StringDictionary
	Documents *StringDictionary
	// contains filtered or unexported fields
}

func OpenDocumentIndexer 

func OpenDocumentIndexer(db *DB, opts ...DocumentIndexOption) (*DocumentIndexer, error)

func (*DocumentIndexer) AddDocument 

func (ix *DocumentIndexer) AddDocument(doc map[string]any) (uint64, int, error)

func (*DocumentIndexer) AddFlatEntry 

func (ix *DocumentIndexer) AddFlatEntry(docIDText, path string, typ DocumentValueType, rawValue string) error

func (*DocumentIndexer) AddJSON 

func (ix *DocumentIndexer) AddJSON(data []byte) (uint64, int, error)

func (*DocumentIndexer) CacheStats 

func (ix *DocumentIndexer) CacheStats() DocumentIndexCacheStats

func (*DocumentIndexer) Close 

func (ix *DocumentIndexer) Close() error

func (*DocumentIndexer) Commit 

func (ix *DocumentIndexer) Commit(mode SyncMode) error

func (*DocumentIndexer) CommitWithStats 

func (ix *DocumentIndexer) CommitWithStats(mode SyncMode) (DocumentCommitStats, error)

type DocumentValueType 

type DocumentValueType uint64
const (
	DocumentNull DocumentValueType = iota
	DocumentBool
	DocumentUint
	DocumentInt
	DocumentFloat
	DocumentString
	DocumentArray

	// DocumentValueTypeCount is one past the last value type. Anything that
	// enumerates every type should range over [DocumentNull, DocumentValueTypeCount).
	DocumentValueTypeCount
)

type ErrCorruptBlock 

type ErrCorruptBlock struct {
	File   string
	Block  uint32
	Offset uint64
	Reason string
}

func (ErrCorruptBlock) Error 

func (e ErrCorruptBlock) Error() string

func (ErrCorruptBlock) Unwrap 

func (e ErrCorruptBlock) Unwrap() error

type ExecutionOptions 

type ExecutionOptions struct {
	ReadPolicy ReadPolicy
	TimeBudget time.Duration
}

type FieldDef 

type FieldDef struct {
	Name string
}

type Hit 

type Hit struct {
	ID    uint64
	Tuple Tuple
}

type IndexInfo 

type IndexInfo struct {
	Range   Range
	File    string
	Entries uint64
	// OffsetRange is the byte span [start, end) in the source AOF that this
	// index's entries occupy; end is the resume point for the next entry. Zero
	// for legacy v2 indexes that predate the field.
	OffsetRange Range
}

type Limits 

type Limits struct {
	MaxHits int
}

type Manifest 

type Manifest struct {
	Version    int
	Selection  string
	AOF        []ManifestFile
	Indexes    []ManifestFile
	StringMaps []ManifestStringMap
}

type ManifestFile 

type ManifestFile struct {
	File  string
	Range string
	// OffsetRange is the source-AOF byte span "start-end" an index was
	// built from (end is the resume point for the next entry). Empty for AOF
	// entries and for legacy indexes without the offset recorded.
	OffsetRange string
}

type ManifestStringMap 

type ManifestStringMap struct {
	Name   string
	Prefix string
	Range  string
	Files  []string
}

type MatchMode 

type MatchMode uint8
const (
	RowMatch MatchMode = iota
	EntityMatch
)

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics is a concurrency-safe bundle of scandex runtime counters and gauges, backed by statekit. It is the "separate object" attach point: create one with NewMetrics, hand it to Open via WithMetrics, and register it with a statekit registry from the outside via reg.RegisterCollectors(m.Collectors()...).

The object is maintained, not merely reported: scandex updates it in place on the reader, writer, and aggregation paths. Every metric is an atomic statekit Counter or Gauge, so a single Metrics value may be shared by any number of concurrent readers (and a writer) without further locking.

All recording methods are nil-safe, so an unset DB option (no metrics) costs a single nil-receiver method call at each operation boundary and nothing in the per-tuple hot loops: reader and aggregation stats are folded in once per completed operation from the SearchStats/Stats the engine already computes.

Note that aggregation runs through the reader (Match -> Search/Scan, Fill -> Scan), so its passes also advance the reader counters below. The reader counters therefore measure all read activity, aggregation included; the aggregation counters add the query-level view on top.

Besides raw counts, the object maintains derived state inside the recording methods rather than leaving it to be computed downstream: the running fraction of entries served from the unsorted AOF tail (a compaction-pressure signal), the running read selectivity (hits per entry examined), and slow-operation counts against a configurable threshold.

func NewMetrics 

func NewMetrics(opts ...MetricsOption) *Metrics

NewMetrics builds a fresh, zeroed Metrics bundle with every collector created and ready to register.

func (*Metrics) Collectors 

func (this *Metrics) Collectors() []statekit.PrometheusCollector

Collectors returns the statekit collectors held by this bundle, for registration from the outside: 

reg.RegisterCollectors(m.Collectors()...)

func (*Metrics) ObserveAggregate 

func (this *Metrics) ObserveAggregate(matched, groups, rows int, dur time.Duration, err error)

ObserveAggregate folds one completed aggregation query into the aggregation counters. It is exported because the analytics layer lives in its own package; callers normally do not invoke it directly.

type MetricsOption 

type MetricsOption func(*Metrics)

MetricsOption configures a Metrics bundle at construction.

func WithSlowQueryThreshold 

func WithSlowQueryThreshold(d time.Duration) MetricsOption

WithSlowQueryThreshold sets the duration at or above which a read or aggregation is counted as slow. A non-positive threshold disables slow-query counting.

type Option 

type Option func(*Options) error

func WithBlockSizeTarget 

func WithBlockSizeTarget(n int) Option

func WithIndexBuildInterval 

func WithIndexBuildInterval(n int) Option

WithIndexBuildInterval makes the writer automatically build an index for the uncovered tail every n records. 0 disables automatic building.

func WithIndexGranularity 

func WithIndexGranularity(n int) Option

func WithLogger 

func WithLogger(fn func(level, msg string, kv ...any)) Option

func WithMaxOpenIndexes 

func WithMaxOpenIndexes(n int) Option

func WithMetrics 

func WithMetrics(m *Metrics) Option

WithMetrics attaches a Metrics bundle that scandex maintains on the reader, writer, and aggregation paths. The same bundle may be passed to several Open calls to aggregate their metrics into one registerable object.

func WithPrefix 

func WithPrefix(prefix string) Option

func WithRecoverOnOpen 

func WithRecoverOnOpen(v bool) Option

func WithScanTail 

func WithScanTail(v bool) Option

func WithVerifyPolicy 

func WithVerifyPolicy(v VerifyPolicy) Option

type Options 

type Options struct {
	IndexGranularity int
	// IndexBuildInterval, when > 0, makes the writer build an index for
	// the uncovered tail every IndexBuildInterval records appended. 0 (the
	// default) leaves index building entirely manual via DB.BuildIndex.
	IndexBuildInterval int
	BlockSizeTarget    int
	Verify             VerifyPolicy
	MaxOpenIndexes     int
	ScanTail           bool
	RecoverOnOpen      bool
	Prefix             string
	Logger             func(level, msg string, kv ...any)
	Metrics            *Metrics
}

type Predicate 

type Predicate interface {
	// contains filtered or unexported methods
}

func And 

func And(ps ...Predicate) Predicate

func Eq 

func Eq(field string, t Term) Predicate

func In 

func In(field string, ts ...Term) Predicate

func Not 

func Not(p Predicate) Predicate

func Or 

func Or(ps ...Predicate) Predicate

func RangePred 

func RangePred(field string, lo, hi Term) Predicate

type Query 

type Query struct {
	Scope     []Term
	Where     Predicate
	Mode      MatchMode
	Return    ReturnSpec
	Limits    Limits
	Execution ExecutionOptions
}

type Range 

type Range struct {
	First uint64
	Last  uint64
}

type ReadPolicy 

type ReadPolicy uint8
const (
	ReadCommitted ReadPolicy = iota
	ReadUncommitted
)

type Result 

type Result struct {
	IDs   []uint64
	Hits  []Hit
	Stats SearchStats
}

type ReturnSpec 

type ReturnSpec struct {
	Tuples bool
}

type ScanAction 

type ScanAction uint8
const (
	ScanContinue ScanAction = iota
	ScanStop
	ScanSkipValue
	ScanSeek
)

type ScanDecision 

type ScanDecision struct {
	Action ScanAction
	Field  int
	Target Tuple
}

type ScanEntry 

type ScanEntry struct {
	Tuple  Tuple
	ID     uint64
	Sorted bool
}

type ScanFunc 

type ScanFunc func(ScanEntry) ScanDecision

type Schema 

type Schema struct {
	Fields []FieldDef
}

func DocumentIndexSchema 

func DocumentIndexSchema() Schema

func (*Schema) FieldIndex 

func (s *Schema) FieldIndex(name string) (int, bool)

func (*Schema) Hash 

func (s *Schema) Hash() uint64

func (*Schema) Validate 

func (s *Schema) Validate() error

type SearchStats 

type SearchStats struct {
	Indexes        int
	TailFiles      int
	EntriesScanned uint64 // total entries examined (IndexEntries + TailEntries)
	IndexEntries   uint64 // entries examined in sorted indexes
	TailEntries    uint64 // entries examined in the unsorted AOF tail
	HitsEmitted    uint64
	Duration       time.Duration
}

type StringDictionary 

type StringDictionary struct {
	// contains filtered or unexported fields
}

func OpenStringDictionary 

func OpenStringDictionary(path string) (*StringDictionary, error)

func (*StringDictionary) Close 

func (d *StringDictionary) Close() error

func (*StringDictionary) ID 

func (d *StringDictionary) ID(s string) (uint64, error)

func (*StringDictionary) Lookup 

func (d *StringDictionary) Lookup(id uint64) (string, bool)

func (*StringDictionary) Save 

func (d *StringDictionary) Save() error

func (*StringDictionary) SaveWithStats 

func (d *StringDictionary) SaveWithStats() (StringDictionarySaveStats, error)

func (*StringDictionary) Stats 

func (d *StringDictionary) Stats() StringDictionaryStats

type StringDictionarySaveStats 

type StringDictionarySaveStats struct {
	Flush   time.Duration
	Total   time.Duration
	Entries int
}

type StringDictionaryStats 

type StringDictionaryStats struct {
	Calls      uint64
	Hits       uint64
	Misses     uint64
	Distinct   uint64
	Once       uint64
	Dup2To9    uint64
	Dup10To99  uint64
	Dup100Plus uint64
}

type SyncMode 

type SyncMode uint8
const (
	Async SyncMode = iota
	Sync
)

type Term 

type Term struct {
	// contains filtered or unexported fields
}

func U 

func U(v uint64) Term

type Tuple 

type Tuple []uint64

type VerifyPolicy 

type VerifyPolicy uint8
const (
	VerifyMagic VerifyPolicy = iota
	VerifyCRC
	VerifyOpen
)

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

func (*Writer) Add 

func (w *Writer) Add(t Tuple, id uint64) error

func (*Writer) Close 

func (w *Writer) Close() error

func (*Writer) Commit 

func (w *Writer) Commit(mode SyncMode) error

func (*Writer) Entries 

func (w *Writer) Entries() uint64

Source Files

View all Source files

Directories

Path Synopsis
analytics
Package analytics computes group-by aggregations on top of documentquery.
 Package analytics computes group-by aggregations on top of documentquery.
federation
Package federation runs documentquery / analytics operations across several independent scandex engines ("segments") and merges the results into a single logical answer.
 Package federation runs documentquery / analytics operations across several independent scandex engines ("segments") and merges the results into a single logical answer.
examples
bluesky command
Command bluesky loads the ClickHouse JSONBench Bluesky dataset (Jetstream firehose events, newline-delimited JSON) into a Scandex document index and lets you search or aggregate over it.
 Command bluesky loads the ClickHouse JSONBench Bluesky dataset (Jetstream firehose events, newline-delimited JSON) into a Scandex document index and lets you search or aggregate over it.
federation command
Command federation demonstrates querying across several independent scandex segments and merging the results.
 Command federation demonstrates querying across several independent scandex segments and merging the results.
github_events command
web_analytics command
Command web_analytics serves a small browser UI over an existing Scandex document index.
 Command web_analytics serves a small browser UI over an existing Scandex document index.
internal
gencache
stringmap
Package stringmap provides a content-addressed string interning store with file-backed storage, split into two roles tuned for opposite access patterns.
 Package stringmap provides a content-addressed string interning store with file-backed storage, split into two roles tuned for opposite access patterns.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.