storage

package
v0.0.0-...-effd846 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package storage implements the vec storage engine (spec 04): fixed-stride columnar vector segments, a separate columnar metadata store, and an id-map that mediates between stable application point ids and dense engine positions.

The engine owns the single copy of every vector (spec 04 §2.1, invariant I-1): one slot in one segment backs every index and every flat scan. Indexes ([07], [08]) operate on positions and call back through FetchVector/ScanVectors to read that single copy; they never persist their own vector bytes.

This build keeps segments, columns, and the id-map resident in memory with the exact stride math and position addressing the spec mandates (spec 04 §3.2-§3.4), so the layout is faithful and the seam to the pager ([05]) is a later slice. The in-memory column slices preserve the columnar access pattern (scan one column without touching vector bytes, spec 04 §5.1) that the on-disk format also gives.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound           = errors.New("vec: point id not found")
	ErrDuplicateID        = errors.New("vec: duplicate point id")
	ErrDeleted            = errors.New("vec: position is tombstoned")
	ErrNotVisible         = errors.New("vec: position not visible in snapshot")
	ErrPositionOutOfRange = errors.New("vec: position out of range")
	ErrSegmentCorrupt     = errors.New("vec: segment CRC mismatch")
	ErrDatabaseCorrupt    = errors.New("vec: database requires manual recovery")
	ErrDimensionMismatch  = errors.New("vec: vector dimension does not match collection")
	ErrSchemaMismatch     = errors.New("vec: metadata schema does not match collection")
	ErrCompactionActive   = errors.New("vec: compaction already in progress for collection")

	// ErrUnknownCollection is returned when a collID has no catalog entry.
	ErrUnknownCollection = errors.New("vec: unknown collection")
	// ErrUnknownColumn is returned when a ColID is not part of the collection schema.
	ErrUnknownColumn = errors.New("vec: unknown column")
	// ErrTxnClosed is returned when a committed or aborted transaction is reused.
	ErrTxnClosed = errors.New("vec: transaction already finished")
)

Engine error set (spec 04 §15.5). All errors are wrappable; callers use errors.Is and errors.As. The engine never panics on bad input, it returns one of these; panics are reserved for structural invariant violations (spec 04 §25.3), raised inline rather than through a separate assert package.

View Source 
var NullValue = Value{Kind: KindNull}

NullValue is the absent / NULL cell (spec 04 §5.3).

Functions

This section is empty.

Types

type And 

type And struct{ Terms []Predicate }

And passes when every child passes (spec 04 §10.5).

type CmpOp 

type CmpOp uint8

CmpOp is a scalar comparison operator in a metadata predicate (spec 04 §10.5). 

const (
	OpEq CmpOp = iota
	OpNe
	OpLt
	OpLe
	OpGt
	OpGe
)

type ColID 

type ColID uint32

ColID identifies a metadata column within a collection (spec 04 §5).

type ColType 

type ColType uint8

ColType is the logical type of a metadata column (spec 04 §5, spec 02 §4). 

const (
	ColInt64     ColType = 0
	ColFloat64   ColType = 1
	ColBool      ColType = 2
	ColTimestamp ColType = 3 // unix nanoseconds, stored as int64
	ColText      ColType = 4
	ColBytes     ColType = 5
)

type CollectionDef 

type CollectionDef struct {
	ID              uint64
	Name            string
	Dims            uint32
	Elem            ElemType
	Metric          distance.Metric
	Columns         []ColumnDef
	SegmentCapacity uint32
	// Int8Scale is the symmetric scale used to dequantize ElemInt8 segments on read
	// (spec 04 §18.5). Ignored for other element types; 0 defaults to 1.0.
	Int8Scale float32
}

CollectionDef declares the physical parameters of a collection (spec 04 §22.1). SegmentCapacity is the per-segment point capacity; 0 derives it from the default 256 MB segment target divided by the stride (spec 04 §3.5).

type CollectionStats 

type CollectionStats struct {
	TotalPoints        uint64
	LivePoints         uint64
	TombstoneCount     uint64
	SegmentCount       int
	Dims               uint32
	Stride             uint32
	StalenessScore     float64 // 0 fresh, 1 very stale (spec 04 §21.2)
	WritesSinceAnalyze uint64
}

CollectionStats are the aggregate statistics the planner consumes (spec 04 §12.2).

type ColumnDef 

type ColumnDef struct {
	ID       ColID
	Name     string
	Type     ColType
	Nullable bool
}

ColumnDef declares one metadata column of a collection (spec 04 §22.1).

type ColumnStats 

type ColumnStats struct {
	ColID         ColID
	NullFraction  float64
	DistinctCount uint64
	Min           Value
	Max           Value
	Histogram     []HistogramBucket
}

ColumnStats are per-column statistics for selectivity estimation (spec 04 §12.3).

type Compare 

type Compare struct {
	Col ColID
	Op  CmpOp
	Lit Value
}

Compare is a leaf predicate: column Col compared by Op against literal Lit (spec 04 §10.5). A comparison involving NULL is false (SQL UNKNOWN treated as not-passing for filter purposes).

type ElemType 

type ElemType uint8

ElemType is the stored element representation of a vector segment (spec 04 §3.3). 

const (
	ElemFP32   ElemType = 0 // 4 bytes/elem, 32-byte stride alignment
	ElemFP16   ElemType = 1 // 2 bytes/elem, 16-byte alignment
	ElemInt8   ElemType = 2 // 1 byte/elem, 16-byte alignment, dequantized on read
	ElemBinary ElemType = 3 // 1 bit/elem, 8-byte alignment, Hamming distance
)

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the storage engine for one database (spec 04 §15.1). It owns the collections, the MVCC clock, and the single-writer lock. Reads take the read lock and resolve through MVCC snapshots; writes serialize on the writer lock and publish atomically at commit (spec 04 §13.1).

func NewEngine 

func NewEngine() *Engine

NewEngine creates an empty engine (spec 04 §15.1).

func (*Engine) Analyze 

func (e *Engine) Analyze(collID uint64) error

Analyze rebuilds and caches statistics for a collection (spec 04 §12.5).

func (*Engine) Begin 

func (e *Engine) Begin(write bool) Txn

Begin starts a transaction (spec 04 §13.1). A write transaction takes the engine's writer lock; it must be released by Commit or Abort.

func (*Engine) Close 

func (e *Engine) Close() error

Close releases engine resources (spec 04 §15.1).

func (*Engine) CollectionStats 

func (e *Engine) CollectionStats(collID uint64) (CollectionStats, error)

CollectionStats returns aggregate statistics for the planner (spec 04 §12.2).

func (*Engine) ColumnStats 

func (e *Engine) ColumnStats(collID uint64, colID ColID) (ColumnStats, error)

ColumnStats returns per-column statistics, computing them on demand if Analyze has not cached them (spec 04 §12.3).

func (*Engine) Compact 

func (e *Engine) Compact(collID uint64, lo, hi uint32) error

Compact rewrites the collection's live points into a fresh, gap-free segment directory and reclaims tombstoned slots (spec 04 §9.4). It always performs a full-collection compaction in this build, so it produces a complete repoint table covering every live point; the lo/hi range is advisory. The repoint table is handed to the Index SPI through the registered hook so indexes renumber their positions in lockstep (spec 04 §9.5, §15.6).

func (*Engine) CreateCollection 

func (e *Engine) CreateCollection(def CollectionDef) error

CreateCollection registers a new collection (spec 04 §22.1). It is a DDL operation, not part of a data transaction in this build.

func (*Engine) Delete 

func (e *Engine) Delete(txn Txn, collID uint64, id PointID) error

Delete tombstones the point with the given id (spec 04 §8.4).

func (*Engine) Fetch 

func (e *Engine) Fetch(collID uint64, P uint32, proj []ColID, snap Snapshot) (PointRecord, error)

Fetch resolves a position to a full point record (spec 04 §7.1). proj selects metadata columns; nil means all.

func (*Engine) FetchBatch 

func (e *Engine) FetchBatch(collID uint64, positions []uint32, proj []ColID, snap Snapshot) ([]PointRecord, error)

FetchBatch resolves many positions (spec 04 §7.5). Positions that are out of range, tombstoned, or not visible are skipped; the result holds the resolvable records in the input order.

func (*Engine) FetchVector 

func (e *Engine) FetchVector(collID uint64, P uint32, buf []float32) error

FetchVector fills buf with the full-precision vector at position P (spec 04 §15.1). Used by the Index SPI for reranking; buf must hold dims floats.

func (*Engine) IdMapStats 

func (e *Engine) IdMapStats(collID uint64) (IdMapStats, error)

IdMapStats returns id-map statistics for a collection (spec 04 §15.3).

func (*Engine) Insert 

func (e *Engine) Insert(txn Txn, collID uint64, id PointID, vec []float32, meta MetaRow) (uint32, error)

Insert inserts a new point and returns its collection-level position (spec 04 §8.1). Returns ErrDuplicateID if the id is already live.

func (*Engine) InsertBatch 

func (e *Engine) InsertBatch(txn Txn, collID uint64, points []Point) ([]uint32, error)

InsertBatch inserts many points as one transaction (spec 04 §8.3).

func (*Engine) LookupID 

func (e *Engine) LookupID(collID uint64, id PointID) (uint32, error)

LookupID resolves a point id to its collection-level position (spec 04 §15.1).

func (*Engine) LookupPos 

func (e *Engine) LookupPos(collID uint64, P uint32) (PointID, error)

LookupPos resolves a position to the point id at that position (spec 04 §15.1).

func (*Engine) MetadataFilter 

func (e *Engine) MetadataFilter(collID uint64, pred Predicate, snap Snapshot) (*PositionBitmap, error)

MetadataFilter evaluates pred against all live, snapshot-visible positions and returns a bitmap of the matches (spec 04 §10.5). Zone maps skip whole blocks the predicate provably rejects (spec 04 §5.4).

func (*Engine) ScanVectors 

func (e *Engine) ScanVectors(collID uint64, snap Snapshot, cb func(pos uint32, vec []float32) bool) error

ScanVectors calls cb for every live, snapshot-visible position in ascending position order (spec 04 §10.2). The flat index and IVF training use this.

func (*Engine) SealSegment 

func (e *Engine) SealSegment(collID uint64) error

SealSegment manually seals the open segment (spec 04 §15.1).

func (*Engine) SetRepointHook 

func (e *Engine) SetRepointHook(h func(collID uint64, rp []Repoint) error)

SetRepointHook registers the compaction repoint callback (spec 04 §9.5).

func (*Engine) ShouldCompact 

func (e *Engine) ShouldCompact(collID uint64) bool

ShouldCompact reports whether a collection's tombstone fraction has crossed the compaction threshold (spec 04 §9.3).

func (*Engine) Snapshot 

func (e *Engine) Snapshot() Snapshot

Snapshot returns a read snapshot at the current commit point (spec 06 §2.1).

func (*Engine) UpdateMeta 

func (e *Engine) UpdateMeta(txn Txn, collID uint64, id PointID, meta MetaRow) error

UpdateMeta updates only the metadata of an existing point (spec 04 §8.7).

func (*Engine) Upsert 

func (e *Engine) Upsert(txn Txn, collID uint64, id PointID, vec []float32, meta MetaRow) (uint32, bool, error)

Upsert inserts or replaces the point (spec 04 §8.5). Replacing tombstones the old slot and appends a new one; the returned bool reports whether the id was new.

func (*Engine) VectorSegments 

func (e *Engine) VectorSegments(collID uint64) ([]VectorSegment, error)

VectorSegments returns the collection's segments as the VectorSegment interface (spec 04 §15.2), for the index build loop and diagnostics.

func (*Engine) WarmCache 

func (e *Engine) WarmCache(collID uint64) error

WarmCache is a no-op in the in-memory build: all segments are resident (spec 04 §15.1). The pager-backed build prefetches vector pages here.

type HistogramBucket 

type HistogramBucket struct {
	Lo    float64
	Hi    float64
	Count uint64
}

HistogramBucket is one equi-depth bucket of a numeric column histogram (spec 04 §21.2): values in [Lo, Hi] cover roughly Count points.

type IdMapStats 

type IdMapStats struct {
	LiveEntries       int64
	TombstonedEntries int64
}

IdMapStats reports id-map size (spec 04 §15.3).

type IsNullPred 

type IsNullPred struct {
	Col    ColID
	Negate bool // when true, passes when the column is NOT NULL
}

IsNullPred passes when the column is NULL (spec 04 §5.3).

type MetaRow 

type MetaRow map[ColID]Value

MetaRow is the metadata payload of a point, keyed by column id (spec 04 §15.1). A missing column is treated as NULL. Columns not in the schema are rejected.

type Not 

type Not struct{ Term Predicate }

Not inverts its child.

type Or 

type Or struct{ Terms []Predicate }

Or passes when any child passes.

type Point 

type Point struct {
	ID   PointID
	Vec  []float32
	Meta MetaRow
}

Point is one element of a bulk insert (spec 04 §15.1).

type PointID 

type PointID = mvcc.PointID

PointID is the stable, application-supplied identity of a point (spec 04 §6.1). It never changes and is never reused after delete. The id-map translates it to a dense engine position, which an ANN graph can store in a fixed 4 bytes per edge (spec 04 §20.4).

type PointRecord 

type PointRecord struct {
	Pos  uint32
	ID   PointID
	Vec  []float32
	Meta MetaRow
}

PointRecord is the resolved, snapshot-visible record returned by Fetch (spec 04 §7.1): the id, the full-precision vector, and the projected metadata.

type PositionBitmap 

type PositionBitmap struct {
	// contains filtered or unexported fields
}

PositionBitmap is a dense bitset over collection-level positions (spec 04 §10.6). The filter path produces one (1 = position passes the predicate) and the Index SPI consumes it as the pre-filter mask. It satisfies the index.Bitmap seam (Contains/Count) without the storage package importing index.

func NewPositionBitmap 

func NewPositionBitmap(n uint32) *PositionBitmap

NewPositionBitmap allocates a bitmap sized for n positions, all clear.

func (*PositionBitmap) And 

func (b *PositionBitmap) And(other *PositionBitmap)

And intersects in place with other (positions set in both survive).

func (*PositionBitmap) Clear 

func (b *PositionBitmap) Clear(p uint32)

Clear unmarks position p.

func (*PositionBitmap) Contains 

func (b *PositionBitmap) Contains(p uint32) bool

Contains reports whether position p is set (satisfies index.Bitmap).

func (*PositionBitmap) Count 

func (b *PositionBitmap) Count() int

Count returns the number of set positions (satisfies index.Bitmap).

func (*PositionBitmap) Len 

func (b *PositionBitmap) Len() uint32

Len returns the bitmap capacity (logical position count).

func (*PositionBitmap) Or 

func (b *PositionBitmap) Or(other *PositionBitmap)

Or unions other into b in place.

func (*PositionBitmap) Set 

func (b *PositionBitmap) Set(p uint32)

Set marks position p (no-op if out of range).

func (*PositionBitmap) SetAll 

func (b *PositionBitmap) SetAll()

SetAll marks every position in [0, n).

type Predicate 

type Predicate interface {
	// contains filtered or unexported methods
}

Predicate is a boolean metadata filter the engine evaluates against each live position (spec 04 §10.5, §21.1). The executor builds these from the WHERE clause ([11], [13]); the engine evaluates them and prunes whole zone blocks where it can.

type Repoint 

type Repoint struct {
	OldPos  uint32
	NewPos  uint32
	PointID PointID
}

Repoint describes one position mapping change during compaction (spec 04 §15.6). The slice handed to ApplyRepoint and to the Index SPI is sorted by OldPos ascending so position-sorted index structures can binary-search it.

type Snapshot 

type Snapshot = *mvcc.Snapshot

Snapshot is an MVCC read snapshot (spec 04 §13.2, spec 06). A read sees only versions committed at or before its watermark; in-flight writes are invisible.

type StorageEngine 

type StorageEngine interface {
	Fetch(collID uint64, P uint32, proj []ColID, snap Snapshot) (PointRecord, error)
	FetchBatch(collID uint64, positions []uint32, proj []ColID, snap Snapshot) ([]PointRecord, error)
	FetchVector(collID uint64, P uint32, buf []float32) error
	ScanVectors(collID uint64, snap Snapshot, cb func(pos uint32, vec []float32) bool) error

	Insert(txn Txn, collID uint64, id PointID, vec []float32, meta MetaRow) (uint32, error)
	InsertBatch(txn Txn, collID uint64, points []Point) ([]uint32, error)
	Delete(txn Txn, collID uint64, id PointID) error
	Upsert(txn Txn, collID uint64, id PointID, vec []float32, meta MetaRow) (uint32, bool, error)
	UpdateMeta(txn Txn, collID uint64, id PointID, meta MetaRow) error

	MetadataFilter(collID uint64, pred Predicate, snap Snapshot) (*PositionBitmap, error)

	LookupID(collID uint64, id PointID) (uint32, error)
	LookupPos(collID uint64, P uint32) (PointID, error)

	CollectionStats(collID uint64) (CollectionStats, error)
	ColumnStats(collID uint64, colID ColID) (ColumnStats, error)
	Analyze(collID uint64) error

	Compact(collID uint64, lo, hi uint32) error
	SealSegment(collID uint64) error
	WarmCache(collID uint64) error
	Close() error
}

StorageEngine is the central object the executor and Index SPI call (spec 04 §15.1). One StorageEngine exists per open database.

type True 

type True struct{}

True passes every row; the absence of a WHERE clause.

type Txn 

type Txn = *txn

Txn is the transaction handle the engine API takes (spec 04 §15.1). It is a pointer so the same handle threads through a multi-statement transaction.

type Value 

type Value struct {
	Kind  ValueKind
	I     int64
	F     float64
	B     bool
	S     string
	Bytes []byte
}

Value is a single metadata cell: a small tagged union covering the scalar and variable-length column types (spec 04 §5.2, §5.5). NullValue is the zero value.

func Bool 

func Bool(b bool) Value

Bool builds a bool value.

func BytesVal 

func BytesVal(b []byte) Value

BytesVal builds a bytes value.

func Float 

func Float(f float64) Value

Float builds a float64 value.

func Int 

func Int(i int64) Value

Int builds an int64 value.

func Text 

func Text(s string) Value

Text builds a text value.

func Timestamp 

func Timestamp(ns int64) Value

Timestamp builds a timestamp value (unix nanoseconds).

func (Value) IsNull 

func (v Value) IsNull() bool

IsNull reports whether the value is NULL.

type ValueKind 

type ValueKind uint8

ValueKind tags a metadata Value (spec 04 §5.2). 

const (
	KindNull      ValueKind = 0
	KindInt       ValueKind = 1
	KindFloat     ValueKind = 2
	KindBool      ValueKind = 3
	KindText      ValueKind = 4
	KindBytes     ValueKind = 5
	KindTimestamp ValueKind = 6
)

type VectorSegment 

type VectorSegment interface {
	SeqNum() uint32
	ElemType() ElemType
	Dims() uint32
	Stride() uint32
	Capacity() uint32
	LiveCount() uint32
	TombstoneCount() uint32
	IsSealed() bool

	FetchVector(slotPos uint32, buf []float32) error
	IsTombstoned(slotPos uint32) bool
	VersionAt(slotPos uint32) uint64
	BackwardMapLookup(slotPos uint32) (PointID, error)
	Scan(cb func(slotPos uint32, vec []float32) bool) error
	Pages() (firstPgno uint32, count uint32)
}

VectorSegment represents one sealed or open segment of vector data (spec 04 §15.2). Callers obtain segments through the engine; they do not create them.

type ZoneBlock 

type ZoneBlock struct {
	Min       Value
	Max       Value
	NullCount uint32
	Count     uint32 // non-null live values folded into Min/Max
}

ZoneBlock summarizes one run of positions in a column segment (spec 04 §5.4). Min and Max bound the live, non-null values; a block with Count == 0 holds no summarizable value and is never skippable. Invariant I-5 (spec 04 §25.1): the zone MAY be wider than the true range but MUST NOT be narrower.

type ZoneMap 

type ZoneMap struct {
	Blocks []ZoneBlock
}

ZoneMap is the per-column-segment array of zone blocks (spec 04 §5.4).

type ZoneMapStats 

type ZoneMapStats struct {
	BlockCount   int
	AvgZoneRange float64
	Sorted       bool
}

ZoneMapStats summarize zone map effectiveness for a column (spec 04 §12.4).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.