xvec

package module
v0.0.0-...-dfee7fc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2026 License: Apache-2.0 Imports: 38 Imported by: 0

README

xvec

CI codecov Go Reference Go Version License

xvec is a pure-Go reimplementation of Alibaba zvec, providing an embedded vector database with durable local storage. It runs inside your application without CGO, a separate database server, or prebuilt native libraries.

[!WARNING] xvec is under active development and is not ready for production use. Public APIs and on-disk formats may change before v1.0.

Features

  • Dense and sparse vector storage with exact and approximate nearest-neighbor search.
  • Flat, HNSW, HNSW-RaBitQ, IVF, Vamana, and DiskANN indexes.
  • L2, inner-product, cosine, and MIPS-L2 metrics with optional quantization and refinement.
  • Scalar filtering, block-max WAND BM25 full-text search, grouping, and hybrid multi-query retrieval.
  • Configurable WAL durability batching, crash recovery, segment-native incremental indexes, and atomic compaction.
  • Pure Go on Linux, macOS, and Windows.

Install

xvec requires Go 1.26 or later.

go get github.com/gorse-io/xvec

Then import it in your application:

import "github.com/gorse-io/xvec"

Vector storage tutorial

The following program creates a local collection, stores vectors with metadata, and returns the two nearest documents.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()

	schema := xvec.NewCollectionSchema("articles",
		xvec.NewField("title", xvec.DataTypeString),
		xvec.NewField("category", xvec.DataTypeString),
		xvec.FieldSchema{
			Name:      "embedding",
			DataType:  xvec.DataTypeVectorFP32,
			Dimension: 3,
			Index:     xvec.NewFlatIndexParams(xvec.MetricTypeCosine),
		},
	)

	collection, err := xvec.CreateAndOpen(
		ctx,
		"./data/articles",
		schema,
		xvec.NewCollectionOptions(),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer collection.Close()

	_, err = collection.Insert(ctx, []xvec.Document{
		{
			PrimaryKey: "go",
			Fields: map[string]any{
				"title":     "The Go Programming Language",
				"category":  "programming",
				"embedding": xvec.VectorFP32{1.0, 0.1, 0.0},
			},
		},
		{
			PrimaryKey: "vector",
			Fields: map[string]any{
				"title":     "Vector Search Fundamentals",
				"category":  "search",
				"embedding": xvec.VectorFP32{0.9, 0.2, 0.1},
			},
		},
		{
			PrimaryKey: "sql",
			Fields: map[string]any{
				"title":     "Database Internals",
				"category":  "database",
				"embedding": xvec.VectorFP32{0.0, 0.2, 1.0},
			},
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	results, err := collection.Query(ctx, xvec.VectorQuery{
		Field:       "embedding",
		DenseVector: xvec.VectorFP32{1.0, 0.0, 0.0},
		TopK:        2,
		Projection: xvec.Projection{
			OutputFields: []string{"title", "category"},
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	for _, result := range results {
		fmt.Printf("%s: %s (score %.4f)\n",
			result.PrimaryKey,
			result.Fields["title"],
			result.Score,
		)
	}
}

The collection is persisted under ./data/articles. Reopen it after restarting your application with:

collection, err := xvec.Open(
    context.Background(),
    "./data/articles",
    xvec.NewCollectionOptions(),
)

Use Insert, Upsert, Update, and Delete for document mutations. Call Flush to publish an immutable segment and Optimize to compact stored data; Close synchronizes pending WAL records. Set CollectionOptions.WALSyncEvery to synchronize automatically after a chosen number of successful records; zero disables automatic record-count-based synchronization. Query also accepts PrimaryKey as a vector target, a single FTS clause, or a filter-only request with no target. MultiQuery fuses dense, sparse, primary-key-vector, and FTS branches over one snapshot.

Choosing an index
Index Best for
Flat Exact search and small collections
HNSW General-purpose low-latency ANN search
HNSW-RaBitQ Memory-efficient graph search for larger vectors
IVF Tunable approximate search with list probing
Vamana Graph-based search with deterministic native persistence
DiskANN Disk-backed graph search with bounded node caching

Dense vectors support FP16 and FP32 storage, plus supported scalar quantization options. Sparse vectors support exact Flat and HNSW inner-product search. See the Collection API and vector query semantics for filters, radius queries, projections, ANN parameters, grouping, and refinement.

Documentation

Compatibility

The root xvec package is the public API. xvec uses native Go disk format v2 and does not read C++ zvec collection files. Version 1 collections are rejected; there is no compatibility, migration, fallback, or dual-write path.

License

Apache License 2.0. See LICENSE and NOTICE.

Documentation

Index

Examples

Constants

View Source 
const (
	// DefaultMultiQueryTopK is the pinned default final result count.
	DefaultMultiQueryTopK = 10
	// DefaultSubQueryCandidates is the pinned default candidate count per
	// MultiQuery branch.
	DefaultSubQueryCandidates = 10
	// MaxQueryTopK is the pinned upper bound for a MultiQuery final or branch
	// result count.
	MaxQueryTopK = 100_000
)
View Source 
const (
	// MinRuntimeMemoryLimit is the pinned minimum explicit process budget.
	MinRuntimeMemoryLimit uint64 = 100 << 20
	// MaxRuntimeConcurrency bounds configured query and optimize admission.
	MaxRuntimeConcurrency = 65_536
)
View Source 
const (
	DefaultHNSWM              = 50
	DefaultHNSWEFConstruction = 500
	DefaultHNSWEFSearch       = 300
	MaxHNSWM                  = 32767
	MaxGraphEFSearch          = 2048
	DefaultPrefetchOffset     = 8
	DefaultPrefetchLines      = 0

	DefaultIVFNList       = 1024
	DefaultIVFNIterations = 10

	DefaultRaBitQTotalBits   = 7
	DefaultRaBitQNumClusters = 16
	MaxRaBitQTotalBits       = 9

	DefaultDiskANNMaxDegree = 100
	DefaultDiskANNListSize  = 50
	DefaultDiskANNPQChunks  = 0

	DefaultVamanaMaxDegree        = 64
	DefaultVamanaSearchListSize   = 100
	DefaultVamanaMaxOcclusionSize = 750
	DefaultVamanaEFSearch         = 200
	MaxVamanaMaxDegree            = 65_535
)
View Source 
const (
	DefaultIVFNProbe            = 10
	DefaultRefinerScaleFactor   = 10
	DefaultDiskANNQueryListSize = 300
)
View Source 
const (
	MaxDenseDimensions       = 20_000
	MaxSparseDimensions      = 16_384
	MaxScalarFields          = 1_024
	MaxVectorFields          = 5
	DefaultMaxDocsPerSegment = 10_000_000
	MinMaxDocsPerSegment     = 1_000
	MinRaBitQDimensions      = 64
	MaxRaBitQDimensions      = 4_095
)
View Source 
const (
	// Version is the semantic version of this library release.
	Version = "0.5.0"

	// NativeDiskFormatVersion identifies the independent Go collection format.
	NativeDiskFormatVersion uint32 = 3
)
View Source 
const DefaultMaxBufferSize uint32 = 64 << 20

DefaultMaxBufferSize is the baseline-compatible DiskANN cache budget.

View Source 
const DefaultRRFRankConstant = 60

DefaultRRFRankConstant is the pinned reciprocal-rank-fusion constant.

View Source 
const DefaultVamanaAlpha float32 = 1.2
View Source 
const MaxPrimaryKeyBytes = 64 << 10

MaxPrimaryKeyBytes is the maximum UTF-8 primary-key size accepted by the native collection format.

Variables

View Source 
var (
	ErrNotFound           error = codeSentinel{ErrorCodeNotFound}
	ErrAlreadyExists      error = codeSentinel{ErrorCodeAlreadyExists}
	ErrInvalidArgument    error = codeSentinel{ErrorCodeInvalidArgument}
	ErrPermissionDenied   error = codeSentinel{ErrorCodePermissionDenied}
	ErrFailedPrecondition error = codeSentinel{ErrorCodeFailedPrecondition}
	ErrResourceExhausted  error = codeSentinel{ErrorCodeResourceExhausted}
	ErrUnavailable        error = codeSentinel{ErrorCodeUnavailable}
	ErrInternal           error = codeSentinel{ErrorCodeInternal}
	ErrNotSupported       error = codeSentinel{ErrorCodeNotSupported}
	ErrUnknown            error = codeSentinel{ErrorCodeUnknown}
)

Stable errors.Is targets for each non-success ErrorCode.

Functions

func ConfigureRuntime 

func ConfigureRuntime(config RuntimeConfig) error

ConfigureRuntime installs the process runtime configuration once. Like the pinned GlobalConfig, later calls are successful no-ops. Configuration must therefore happen before the first collection is created or opened.

func DefaultJiebaDictDir 

func DefaultJiebaDictDir() string

DefaultJiebaDictDir returns the current process-wide Jieba fallback.

func SetDefaultJiebaDictDir 

func SetDefaultJiebaDictDir(path string)

SetDefaultJiebaDictDir sets the process-wide lowest-priority Jieba resource directory. Per-field configuration and ZVEC_JIEBA_DICT_DIR take precedence.

Types

type AddColumnOptions 

type AddColumnOptions struct{ Concurrency int }

AddColumnOptions controls column backfill concurrency.

type AlterColumnOptions 

type AlterColumnOptions struct{ Concurrency int }

AlterColumnOptions controls column migration concurrency.

type BatchWriteError 

type BatchWriteError struct {
	Failed int
	// contains filtered or unexported fields
}

BatchWriteError summarizes per-document write failures. The returned WriteResult slice remains authoritative and preserves input order.

func (*BatchWriteError) Error 

func (e *BatchWriteError) Error() string

func (*BatchWriteError) Unwrap 

func (e *BatchWriteError) Unwrap() []error

Unwrap exposes every per-document cause to errors.Is and errors.As.

type Binary 

type Binary []byte

Binary is an arbitrary byte string. It is distinct from a UTF-8 String field and from binary vectors.

type BinaryArray 

type BinaryArray []Binary

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type BlockType 

type BlockType uint32

BlockType identifies a persisted collection block. 

const (
	BlockTypeUndefined           BlockType = 0
	BlockTypeScalar              BlockType = 1
	BlockTypeScalarIndex         BlockType = 2
	BlockTypeVectorIndex         BlockType = 3
	BlockTypeVectorIndexQuantize BlockType = 4
	BlockTypeFTSIndex            BlockType = 5
)

func (BlockType) String 

func (t BlockType) String() string

func (BlockType) Valid 

func (t BlockType) Valid() bool

Valid reports whether t is a value defined by the public API.

type BoolArray 

type BoolArray []bool

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type CallbackReranker 

type CallbackReranker func(ctx context.Context, batches []RerankBatch, topK int) ([]Document, error)

CallbackReranker adapts a function to the Reranker interface. The callback receives candidate batches in sub-query order and the requested final topK. Collection.MultiQuery validates the returned documents against its snapshot.

func NewCallbackReranker 

func NewCallbackReranker(callback func(context.Context, []RerankBatch, int) ([]Document, error)) CallbackReranker

NewCallbackReranker adapts callback to a Reranker. A nil callback remains invalid when invoked directly; a nil Reranker on MultiQuery selects RRF.

func (CallbackReranker) Rerank 

func (r CallbackReranker) Rerank(ctx context.Context, batches []RerankBatch, topK int) (documents []Document, err error)

Rerank invokes the callback and converts a panic into a structured internal error so caller code cannot unwind through the collection query boundary. Callback errors are returned unchanged.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec"
)

func main() {
	reranker := xvec.NewCallbackReranker(func(_ context.Context, batches []xvec.RerankBatch, topK int) ([]xvec.Document, error) {
		result := []xvec.Document{batches[1].Documents[0], batches[0].Documents[0]}
		if len(result) > topK {
			result = result[:topK]
		}
		return result, nil
	})
	results, err := reranker.Rerank(context.Background(), []xvec.RerankBatch{
		{Documents: []xvec.Document{{PrimaryKey: "vector", Score: 0.8}}},
		{Documents: []xvec.Document{{PrimaryKey: "keyword", Score: 2.1}}},
	}, 1)
	if err != nil {
		panic(err)
	}
	fmt.Println(results[0].PrimaryKey)

}

Output:
keyword

func (CallbackReranker) Validate 

func (r CallbackReranker) Validate() error

Validate rejects an empty callback.

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

Collection is one open native Go collection. Its methods are safe for concurrent use; mutations are serialized so queries see complete versions.

func CreateAndOpen 

func CreateAndOpen(ctx context.Context, path string, schema CollectionSchema, options CollectionOptions) (*Collection, error)

CreateAndOpen creates a native Go collection and opens its sole writable handle. The format is intentionally incompatible with C++ collections.

func Open 

func Open(ctx context.Context, path string, options CollectionOptions) (*Collection, error)

Open opens the version named by CURRENT and replays the valid WAL prefix.

func (*Collection) AddColumn 

func (c *Collection) AddColumn(ctx context.Context, field FieldSchema, expression string, options AddColumnOptions) error

AddColumn atomically adds a basic numeric field and backfills every live document with a baseline-compatible arithmetic expression. An empty expression is allowed only for nullable fields and writes explicit NULLs.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-add-column-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "rating", DataType: xvec.DataTypeInt32},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{{
		PrimaryKey: "book", Fields: map[string]any{"rating": int32(4)},
	}})
	if err != nil {
		panic(err)
	}
	field := xvec.FieldSchema{Name: "adjusted", DataType: xvec.DataTypeInt64}
	if err := collection.AddColumn(ctx, field, "rating * 2 + 1", xvec.AddColumnOptions{Concurrency: 2}); err != nil {
		panic(err)
	}
	documents, err := collection.Fetch(ctx, []string{"book"}, xvec.Projection{})
	if err != nil {
		panic(err)
	}
	fmt.Println(documents[0].Fields["adjusted"])
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
9

func (*Collection) AlterColumn 

func (c *Collection) AlterColumn(ctx context.Context, column, rename string, field *FieldSchema, options AlterColumnOptions) error

AlterColumn atomically renames or replaces one basic numeric field. A replacement may change its name, numeric type, nullability, and INVERT index parameters. Rename and replacement forms are mutually exclusive.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-alter-column-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "rating", DataType: xvec.DataTypeInt32},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{{
		PrimaryKey: "book", Fields: map[string]any{"rating": int32(4)},
	}})
	if err != nil {
		panic(err)
	}
	replacement := xvec.FieldSchema{Name: "adjusted", DataType: xvec.DataTypeInt64}
	if err := collection.AlterColumn(ctx, "rating", "", &replacement, xvec.AlterColumnOptions{Concurrency: 2}); err != nil {
		panic(err)
	}
	documents, err := collection.Fetch(ctx, []string{"book"}, xvec.Projection{})
	if err != nil {
		panic(err)
	}
	fmt.Println(documents[0].Fields["adjusted"])
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
4

func (*Collection) Close 

func (c *Collection) Close() error

Close releases files and the cross-process collection lock. It is idempotent; Close synchronizes pending WAL records, so writes remain recoverable without an explicit Flush.

func (*Collection) CreateIndex 

func (c *Collection) CreateIndex(ctx context.Context, column string, index IndexParams, options CreateIndexOptions) error

CreateIndex validates and backfills a currently implemented index, then atomically publishes the new schema in a manifest generation. At this stage Vector, INVERT, and FTS indexes are snapshot-local runtime indexes, so backfill validates the complete live snapshot and publication persists their parameters.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-index-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "rating", DataType: xvec.DataTypeInt32},
		xvec.FieldSchema{
			Name: "embedding", DataType: xvec.DataTypeVectorFP32, Dimension: 2,
			Index: xvec.NewFlatIndexParams(xvec.MetricTypeIP),
		},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{{
		PrimaryKey: "book", Fields: map[string]any{
			"rating": int32(5), "embedding": xvec.VectorFP32{1, 0},
		},
	}})
	if err != nil {
		panic(err)
	}
	if err := collection.CreateIndex(ctx, "rating", xvec.NewInvertIndexParams(), xvec.CreateIndexOptions{Concurrency: 2}); err != nil {
		panic(err)
	}
	field, _ := collection.Schema().Field("rating")
	fmt.Println(field.IndexType())
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
INVERT

func (*Collection) Delete 

func (c *Collection) Delete(ctx context.Context, primaryKeys []string) ([]WriteResult, error)

Delete removes current versions through the WAL. WALSyncEvery controls when records become durable.

func (*Collection) DeleteByFilter 

func (c *Collection) DeleteByFilter(ctx context.Context, filter string) error

DeleteByFilter removes every live document for which filter evaluates to SQL TRUE through the WAL. Selection and WAL-backed deletion are serialized under the collection write lock, so a matched version cannot be replaced between those two phases. WALSyncEvery controls when records become durable.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-delete-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "rating", DataType: xvec.DataTypeInt32, Index: xvec.NewInvertIndexParams()},
		xvec.FieldSchema{
			Name: "embedding", DataType: xvec.DataTypeVectorFP32, Dimension: 2,
			Index: xvec.NewFlatIndexParams(xvec.MetricTypeIP),
		},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{
		{PrimaryKey: "keep", Fields: map[string]any{"rating": int32(3), "embedding": xvec.VectorFP32{1, 0}}},
		{PrimaryKey: "remove", Fields: map[string]any{"rating": int32(5), "embedding": xvec.VectorFP32{0.5, 0}}},
	})
	if err != nil {
		panic(err)
	}
	if err := collection.DeleteByFilter(ctx, "rating >= 4"); err != nil {
		panic(err)
	}
	fmt.Println(collection.Stats().DocumentCount)
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
1

func (*Collection) Destroy 

func (c *Collection) Destroy(ctx context.Context) error

Destroy closes the handle and recursively removes only its validated collection directory. Read-only handles cannot destroy collections.

func (*Collection) DropColumn 

func (c *Collection) DropColumn(ctx context.Context, column string) error

DropColumn atomically removes one basic numeric field from the schema and every live document payload. Nonnumeric fields are reserved for their owning index milestones and are rejected until those implementations can migrate their auxiliary state safely.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-drop-column-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "title", DataType: xvec.DataTypeString},
		xvec.FieldSchema{Name: "legacy_score", DataType: xvec.DataTypeInt32},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{{
		PrimaryKey: "book", Fields: map[string]any{"title": "Go", "legacy_score": int32(4)},
	}})
	if err != nil {
		panic(err)
	}
	if err := collection.DropColumn(ctx, "legacy_score"); err != nil {
		panic(err)
	}
	documents, err := collection.Fetch(ctx, []string{"book"}, xvec.Projection{})
	if err != nil {
		panic(err)
	}
	_, found := documents[0].Fields["legacy_score"]
	fmt.Println(found)
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
false

func (*Collection) DropIndex 

func (c *Collection) DropIndex(ctx context.Context, column string) error

DropIndex atomically clears a scalar index or restores a vector field to the baseline unquantized Flat/IP definition. Existing documents are unchanged.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-drop-index-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "rating", DataType: xvec.DataTypeInt32, Index: xvec.NewInvertIndexParams()},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	if err := collection.DropIndex(ctx, "rating"); err != nil {
		panic(err)
	}
	field, _ := collection.Schema().Field("rating")
	fmt.Println(field.IndexType())
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
UNDEFINED

func (*Collection) Fetch 

func (c *Collection) Fetch(ctx context.Context, primaryKeys []string, projection Projection) ([]*Document, error)

Fetch returns one independently owned document pointer per requested key. Missing or deleted keys have a nil entry and are not errors.

func (*Collection) Flush 

func (c *Collection) Flush(ctx context.Context) error

Flush atomically publishes the current write segment and rotates its WAL.

func (*Collection) GroupByQuery 

func (c *Collection) GroupByQuery(ctx context.Context, query GroupByVectorQuery) ([]GroupResult, error)

GroupByQuery executes complete Flat/Linear grouping or native HNSW grouping. IVF, Vamana, and DiskANN group traversal remain unsupported to match the pinned native baseline.

func (*Collection) Insert 

func (c *Collection) Insert(ctx context.Context, documents []Document) ([]WriteResult, error)

Insert writes new primary keys through the WAL. WALSyncEvery controls when records become durable. Valid documents in a mixed batch are committed even when other entries fail validation or already exist.

func (*Collection) MultiQuery 

func (c *Collection) MultiQuery(ctx context.Context, query MultiQuery) ([]Document, error)

MultiQuery executes every branch over one live-document snapshot, applies one shared scalar filter, and delegates fusion to the configured or default Reranker. BM25 corpus statistics include every live document; the scalar filter masks FTS candidates without changing IDF or average document length.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-multi-query-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "title", DataType: xvec.DataTypeString, Index: xvec.NewFTSIndexParams()},
		xvec.FieldSchema{
			Name: "embedding", DataType: xvec.DataTypeVectorFP32, Dimension: 2,
			Index: xvec.NewFlatIndexParams(xvec.MetricTypeIP),
		},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{
		{PrimaryKey: "go", Fields: map[string]any{"title": "Go vector search", "embedding": xvec.VectorFP32{0.8, 0}}},
		{PrimaryKey: "ann", Fields: map[string]any{"title": "Approximate neighbors", "embedding": xvec.VectorFP32{1, 0}}},
	})
	if err != nil {
		panic(err)
	}
	results, err := collection.MultiQuery(ctx, xvec.MultiQuery{
		Queries: []xvec.SubQuery{
			{Field: "embedding", DenseVector: xvec.VectorFP32{1, 0}, NumCandidates: 2},
			{Field: "title", FTS: &xvec.FTSClause{Match: "go search"}, NumCandidates: 2},
		},
		TopK: 1, Projection: xvec.Projection{OutputFields: []string{"title"}},
	})
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s: %s\n", results[0].PrimaryKey, results[0].Fields["title"])
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
go: Go vector search

func (*Collection) Optimize 

func (c *Collection) Optimize(ctx context.Context, options OptimizeOptions) error

Optimize atomically compacts the current live snapshot into maximally sized contiguous-ID segments, reclaims superseded/deleted versions, rebuilds the implemented vector/INVERT/FTS runtime state, and removes obsolete storage files.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-optimize-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")
	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "rating", DataType: xvec.DataTypeInt32, Index: xvec.NewInvertIndexParams()},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{
		{PrimaryKey: "keep", Fields: map[string]any{"rating": int32(5)}},
		{PrimaryKey: "remove", Fields: map[string]any{"rating": int32(1)}},
	})
	if err != nil {
		panic(err)
	}
	if err := collection.Flush(ctx); err != nil {
		panic(err)
	}
	if _, err := collection.Delete(ctx, []string{"remove"}); err != nil {
		panic(err)
	}
	if err := collection.Optimize(ctx, xvec.OptimizeOptions{Concurrency: 2}); err != nil {
		panic(err)
	}
	documents, err := collection.Fetch(ctx, []string{"keep", "remove"}, xvec.Projection{})
	if err != nil {
		panic(err)
	}
	fmt.Println(collection.Stats().DocumentCount)
	fmt.Println(documents[0] != nil, documents[1] == nil)
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
1
true true

func (*Collection) Options 

func (c *Collection) Options() CollectionOptions

Options returns the effective options for this handle.

func (*Collection) Path 

func (c *Collection) Path() string

Path returns the absolute collection directory.

func (*Collection) Query 

func (c *Collection) Query(ctx context.Context, query VectorQuery) ([]Document, error)

Query executes a vector, document-ID vector, full-text, or filter-only search over the current live document versions. Filter is parsed and schema-bound before its candidate mask is applied.

Example 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "books")

	schema := xvec.NewCollectionSchema("books",
		xvec.FieldSchema{Name: "title", DataType: xvec.DataTypeString},
		xvec.FieldSchema{
			Name: "embedding", DataType: xvec.DataTypeVectorFP32, Dimension: 2,
			Index: xvec.NewFlatIndexParams(xvec.MetricTypeIP),
		},
	)
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{
		{PrimaryKey: "go", Fields: map[string]any{"title": "The Go Programming Language", "embedding": xvec.VectorFP32{1, 0}}},
		{PrimaryKey: "db", Fields: map[string]any{"title": "Database Internals", "embedding": xvec.VectorFP32{0.5, 0}}},
	})
	if err != nil {
		panic(err)
	}
	results, err := collection.Query(ctx, xvec.VectorQuery{
		Field: "embedding", DenseVector: xvec.VectorFP32{1, 0}, TopK: 1,
		Projection: xvec.Projection{OutputFields: []string{"title"}},
	})
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s: %s (%.1f)\n", results[0].PrimaryKey, results[0].Fields["title"], results[0].Score)
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
go: The Go Programming Language (1.0)
Example (Ann) 
package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gorse-io/xvec"
)

func main() {
	ctx := context.Background()
	directory, err := os.MkdirTemp("", "zvec-ann-example-")
	if err != nil {
		panic(err)
	}
	path := filepath.Join(directory, "items")
	index := xvec.NewHNSWIndexParams(xvec.MetricTypeL2)
	index.M = 8
	index.EFConstruction = 32
	index.Quantize = xvec.QuantizeTypeInt8
	index.Quantizer.EnableRotate = true
	schema := xvec.NewCollectionSchema("items", xvec.FieldSchema{
		Name: "embedding", DataType: xvec.DataTypeVectorFP32, Dimension: 4, Index: index,
	})
	collection, err := xvec.CreateAndOpen(ctx, path, schema, xvec.NewCollectionOptions())
	if err != nil {
		panic(err)
	}
	_, err = collection.Insert(ctx, []xvec.Document{
		{PrimaryKey: "nearest", Fields: map[string]any{"embedding": xvec.VectorFP32{1, 2, 3, 4}}},
		{PrimaryKey: "farther", Fields: map[string]any{"embedding": xvec.VectorFP32{4, 3, 2, 1}}},
	})
	if err != nil {
		panic(err)
	}
	params := xvec.NewHNSWQueryParams()
	params.EF = 32
	params.UseRefiner = true
	results, err := collection.Query(ctx, xvec.VectorQuery{
		Field: "embedding", DenseVector: xvec.VectorFP32{1, 2, 3, 4}, TopK: 1, Params: params,
	})
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s %.1f\n", results[0].PrimaryKey, results[0].Score)
	if err := collection.Destroy(ctx); err != nil {
		panic(err)
	}
	_ = os.Remove(directory)

}

Output:
nearest 0.0

func (*Collection) Schema 

func (c *Collection) Schema() CollectionSchema

Schema returns an independent schema copy.

func (*Collection) Stats 

func (c *Collection) Stats() CollectionStats

Stats returns document, segment, deletion, retained-memory, and current index completeness counters.

func (*Collection) Update 

func (c *Collection) Update(ctx context.Context, documents []Document) ([]WriteResult, error)

Update partially replaces fields on existing documents while retaining all unspecified fields from the current version.

func (*Collection) Upsert 

func (c *Collection) Upsert(ctx context.Context, documents []Document) ([]WriteResult, error)

Upsert inserts new documents and partially updates existing documents.

type CollectionOptions 

type CollectionOptions struct {
	ReadOnly      bool
	EnableMmap    bool
	MaxBufferSize uint32
	// WALSyncEvery synchronizes the WAL after this many successful records.
	// Zero disables automatic record-count-based synchronization; Flush and
	// Close still synchronize pending WAL records.
	WALSyncEvery uint64
}

CollectionOptions controls one collection handle. EnableMmap is persisted when creating a collection; MaxBufferSize bounds native DiskANN node-cache bytes. A zero MaxBufferSize selects DefaultMaxBufferSize.

func NewCollectionOptions 

func NewCollectionOptions() CollectionOptions

NewCollectionOptions returns baseline-compatible handle defaults.

type CollectionSchema 

type CollectionSchema struct {
	Name              string
	Fields            []FieldSchema
	MaxDocsPerSegment uint64
}

CollectionSchema describes a collection and preserves field order.

func NewCollectionSchema 

func NewCollectionSchema(name string, fields ...FieldSchema) CollectionSchema

NewCollectionSchema returns a deep-copied schema with the baseline segment size default.

func (CollectionSchema) Clone 

func (s CollectionSchema) Clone() CollectionSchema

Clone returns a deep copy of s.

func (CollectionSchema) Field 

func (s CollectionSchema) Field(name string) (FieldSchema, bool)

Field returns an independent copy of the named field.

func (CollectionSchema) Validate 

func (s CollectionSchema) Validate() error

Validate checks collection-level limits, duplicate names, and every field.

type CollectionStats 

type CollectionStats struct {
	DocumentCount      uint64
	IndexCompleteness  map[string]float32
	ImmutableSegments  uint64
	MutableDocuments   uint64
	DeletedDocuments   uint64
	StorageMemoryBytes uint64
}

CollectionStats is a point-in-time summary of live and retained in-memory collection state. StorageMemoryBytes is a conservative encoded-size estimate, not the Go process heap size.

type ColumnOp 

type ColumnOp uint32

ColumnOp identifies a schema mutation operation. 

const (
	ColumnOpUndefined ColumnOp = 0
	ColumnOpAdd       ColumnOp = 1
	ColumnOpAlter     ColumnOp = 2
	ColumnOpDrop      ColumnOp = 3
)

func (ColumnOp) String 

func (o ColumnOp) String() string

func (ColumnOp) Valid 

func (o ColumnOp) Valid() bool

Valid reports whether o is a value defined by the public API.

type CompareOp 

type CompareOp uint32

CompareOp identifies a scalar filter predicate. 

const (
	CompareOpNone          CompareOp = 0
	CompareOpEQ            CompareOp = 1
	CompareOpNE            CompareOp = 2
	CompareOpLT            CompareOp = 3
	CompareOpLE            CompareOp = 4
	CompareOpGT            CompareOp = 5
	CompareOpGE            CompareOp = 6
	CompareOpLike          CompareOp = 7
	CompareOpContainAll    CompareOp = 8
	CompareOpContainAny    CompareOp = 9
	CompareOpNotContainAll CompareOp = 10
	CompareOpNotContainAny CompareOp = 11
	CompareOpIsNull        CompareOp = 12
	CompareOpIsNotNull     CompareOp = 13
	CompareOpHasPrefix     CompareOp = 14
	CompareOpHasSuffix     CompareOp = 15
)

func (CompareOp) String 

func (o CompareOp) String() string

func (CompareOp) Valid 

func (o CompareOp) Valid() bool

Valid reports whether o is a value defined by the public API.

type CreateIndexOptions 

type CreateIndexOptions struct{ Concurrency int }

CreateIndexOptions controls index-build concurrency. Zero lets the library select an appropriate worker count.

type DataType 

type DataType uint32

DataType identifies a scalar, array, dense-vector, or sparse-vector field. Its numeric values match the public C++ header at baseline commit 58375ff. 

const (
	DataTypeUndefined DataType = 0

	DataTypeBinary DataType = 1
	DataTypeString DataType = 2
	DataTypeBool   DataType = 3
	DataTypeInt32  DataType = 4
	DataTypeInt64  DataType = 5
	DataTypeUint32 DataType = 6
	DataTypeUint64 DataType = 7
	DataTypeFloat  DataType = 8
	DataTypeDouble DataType = 9

	DataTypeVectorBinary32 DataType = 20
	DataTypeVectorBinary64 DataType = 21
	DataTypeVectorFP16     DataType = 22
	DataTypeVectorFP32     DataType = 23
	DataTypeVectorFP64     DataType = 24
	DataTypeVectorInt4     DataType = 25
	DataTypeVectorInt8     DataType = 26
	DataTypeVectorInt16    DataType = 27

	DataTypeSparseVectorFP16 DataType = 30
	DataTypeSparseVectorFP32 DataType = 31

	DataTypeArrayBinary DataType = 40
	DataTypeArrayString DataType = 41
	DataTypeArrayBool   DataType = 42
	DataTypeArrayInt32  DataType = 43
	DataTypeArrayInt64  DataType = 44
	DataTypeArrayUint32 DataType = 45
	DataTypeArrayUint64 DataType = 46
	DataTypeArrayFloat  DataType = 47
	DataTypeArrayDouble DataType = 48
)

func (DataType) ElementType 

func (t DataType) ElementType() DataType

ElementType returns the scalar element type of an array and returns t for non-array data types.

func (DataType) IsArray 

func (t DataType) IsArray() bool

IsArray reports whether t stores an array.

func (DataType) IsDenseVector 

func (t DataType) IsDenseVector() bool

IsDenseVector reports whether t stores a dense vector.

func (DataType) IsSparseVector 

func (t DataType) IsSparseVector() bool

IsSparseVector reports whether t stores a sparse vector.

func (DataType) IsVector 

func (t DataType) IsVector() bool

IsVector reports whether t stores either a dense or sparse vector.

func (DataType) String 

func (t DataType) String() string

func (DataType) Valid 

func (t DataType) Valid() bool

Valid reports whether t is a value defined by the public API.

type DenseVector 

type DenseVector interface {
	DataType() DataType
	Dimension() int
	// contains filtered or unexported methods
}

DenseVector is implemented by every explicit dense-vector value.

type DiskANNIndexParams 

type DiskANNIndexParams struct {
	Metric    MetricType
	MaxDegree int
	ListSize  int
	PQChunks  int
	Quantize  QuantizeType
	Quantizer QuantizerParams
}

DiskANNIndexParams configures a disk-backed graph index.

func NewDiskANNIndexParams 

func NewDiskANNIndexParams(metric MetricType) DiskANNIndexParams

func (DiskANNIndexParams) IndexType 

func (DiskANNIndexParams) IndexType() IndexType

func (DiskANNIndexParams) Validate 

func (p DiskANNIndexParams) Validate() error

type DiskANNQueryParams 

type DiskANNQueryParams struct {
	QueryOptions
	ListSize int
}

DiskANNQueryParams configures the disk graph search frontier.

func NewDiskANNQueryParams 

func NewDiskANNQueryParams() DiskANNQueryParams

func (DiskANNQueryParams) IndexType 

func (DiskANNQueryParams) IndexType() IndexType

func (DiskANNQueryParams) Validate 

func (p DiskANNQueryParams) Validate() error

type Document 

type Document struct {
	PrimaryKey string
	Fields     map[string]any
	Score      float32
	DocID      uint64
}

Document is the public document and query-result model. Fields contains scalar, array, dense-vector, and sparse-vector values using the explicit Go types declared by this package. Score and DocID are populated by queries; writes ignore them.

func NewDocument 

func NewDocument(primaryKey string, fields map[string]any) (Document, error)

NewDocument clones fields and returns a document ready for validation or a write operation.

func ProjectDocument 

func ProjectDocument(document Document, schema CollectionSchema, projection Projection) (Document, error)

ProjectDocument applies scalar selection and vector inclusion, cloning every retained value and preserving primary key, score, and internal document ID.

func (Document) Clone 

func (d Document) Clone() (Document, error)

Clone returns a deep, independently mutable document.

func (Document) Field 

func (d Document) Field(name string) (any, bool)

Field returns a cloned field value.

func (Document) FieldNames 

func (d Document) FieldNames() []string

FieldNames returns field names in bytewise ascending order.

func (Document) Validate 

func (d Document) Validate(schema CollectionSchema) error

Validate checks primary-key, field presence, nullability, exact Go value types, and dense-vector dimensions against schema. It applies full insert and upsert semantics: every non-nullable schema field must be present.

type Error 

type Error struct {
	Code    ErrorCode
	Op      string
	Path    string
	Message string
	Err     error
}

Error is the structured error returned by xvec operations.

Code is suitable for programmatic decisions. Op and Path identify the failed operation and collection path when available. Err retains the underlying error for errors.Is and errors.As traversal.

func (*Error) Error 

func (e *Error) Error() string

Error formats the structured context without losing the underlying cause.

func (*Error) Is 

func (e *Error) Is(target error) bool

Is makes errors.Is compare xvec errors by ErrorCode. An underlying cause is still considered by the standard library through Unwrap.

func (*Error) Unwrap 

func (e *Error) Unwrap() error

Unwrap returns the underlying cause.

type ErrorCode 

type ErrorCode uint32

ErrorCode classifies errors returned by xvec operations. The values and default messages match StatusCode in the C++ public API at commit 58375ff. 

const (
	ErrorCodeOK                 ErrorCode = 0
	ErrorCodeNotFound           ErrorCode = 1
	ErrorCodeAlreadyExists      ErrorCode = 2
	ErrorCodeInvalidArgument    ErrorCode = 3
	ErrorCodePermissionDenied   ErrorCode = 4
	ErrorCodeFailedPrecondition ErrorCode = 5
	ErrorCodeResourceExhausted  ErrorCode = 6
	ErrorCodeUnavailable        ErrorCode = 7
	ErrorCodeInternal           ErrorCode = 8
	ErrorCodeNotSupported       ErrorCode = 9
	ErrorCodeUnknown            ErrorCode = 10
)

func (ErrorCode) DefaultMessage 

func (c ErrorCode) DefaultMessage() string

DefaultMessage returns the stable default message for c.

func (ErrorCode) String 

func (c ErrorCode) String() string

func (ErrorCode) Valid 

func (c ErrorCode) Valid() bool

Valid reports whether c is a value defined by the public API.

type FTSClause 

type FTSClause struct {
	Query string
	Match string
}

FTSClause describes one full-text target. Exactly one of Query and Match must be non-empty. Query uses the FTS expression grammar; Match analyzes the text as natural language without interpreting operators.

type FTSIndexParams 

type FTSIndexParams struct {
	Tokenizer   string
	Filters     []string
	ExtraParams string
}

FTSIndexParams configures full-text tokenization and token filters.

func NewFTSIndexParams 

func NewFTSIndexParams() FTSIndexParams

NewFTSIndexParams returns the baseline defaults: standard tokenization and a lowercase filter.

func (FTSIndexParams) IndexType 

func (FTSIndexParams) IndexType() IndexType

func (FTSIndexParams) Validate 

func (p FTSIndexParams) Validate() error

type FTSQueryParams 

type FTSQueryParams struct {
	DefaultOperator string
}

FTSQueryParams configures parsing of adjacent bare terms. DefaultOperator is case-insensitive and may be empty, OR, or AND; empty means OR.

func NewFTSQueryParams 

func NewFTSQueryParams() FTSQueryParams

func (FTSQueryParams) IndexType 

func (FTSQueryParams) IndexType() IndexType

func (FTSQueryParams) Validate 

func (p FTSQueryParams) Validate() error

type FieldSchema 

type FieldSchema struct {
	Name      string
	DataType  DataType
	Nullable  bool
	Dimension uint32
	Index     IndexParams
}

FieldSchema describes one scalar, array, dense-vector, or sparse-vector field. Dimension is required only for dense vectors. A nil Index is valid; collection creation treats it as an exact Flat/IP index for vector fields.

func NewField 

func NewField(name string, dataType DataType) FieldSchema

NewField returns a scalar or array field schema.

func NewVectorField 

func NewVectorField(name string, dataType DataType, dimension uint32) FieldSchema

NewVectorField returns a dense-vector field schema.

func (FieldSchema) Clone 

func (f FieldSchema) Clone() FieldSchema

Clone returns a deep copy of f.

func (FieldSchema) EffectiveIndex 

func (f FieldSchema) EffectiveIndex() IndexParams

EffectiveIndex returns an independent configured index. Vector fields with no explicit index receive the baseline Flat/IP default. Scalar fields with no index return nil.

func (FieldSchema) IndexType 

func (f FieldSchema) IndexType() IndexType

IndexType reports the configured index type, or IndexTypeUndefined when no index is configured.

func (FieldSchema) Validate 

func (f FieldSchema) Validate() error

Validate checks the field name, type, dimension, index parameters, and all supported type/index/metric/quantization combinations.

type FileFormat 

type FileFormat uint32

FileFormat identifies an import or export file format. 

const (
	FileFormatUnknown FileFormat = 0
	FileFormatIPC     FileFormat = 1
	FileFormatParquet FileFormat = 2
)

func (FileFormat) String 

func (f FileFormat) String() string

func (FileFormat) Valid 

func (f FileFormat) Valid() bool

Valid reports whether f is a value defined by the public API.

type FlatIndexParams 

type FlatIndexParams struct {
	Metric    MetricType
	Quantize  QuantizeType
	Quantizer QuantizerParams
}

FlatIndexParams configures exact vector search.

func NewFlatIndexParams 

func NewFlatIndexParams(metric MetricType) FlatIndexParams

func (FlatIndexParams) IndexType 

func (FlatIndexParams) IndexType() IndexType

func (FlatIndexParams) Validate 

func (p FlatIndexParams) Validate() error

type FlatQueryParams 

type FlatQueryParams struct {
	QueryOptions
	ScaleFactor float32
}

FlatQueryParams configures exact vector search.

func NewFlatQueryParams 

func NewFlatQueryParams() FlatQueryParams

func (FlatQueryParams) IndexType 

func (FlatQueryParams) IndexType() IndexType

func (FlatQueryParams) Validate 

func (p FlatQueryParams) Validate() error

type Float16 

type Float16 uint16

Float16 stores the IEEE 754 binary16 bit representation of a number.

func Float16FromFloat32 

func Float16FromFloat32(value float32) Float16

Float16FromFloat32 converts value to binary16 using round-to-nearest-even.

func (Float16) Float32 

func (f Float16) Float32() float32

Float32 converts f to the exactly represented float32 value.

type Float32Array 

type Float32Array []float32

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type Float64Array 

type Float64Array []float64

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type GroupByVectorQuery 

type GroupByVectorQuery struct {
	Field        string
	DenseVector  DenseVector
	SparseVector SparseVector
	PrimaryKey   string
	Filter       string
	Projection   Projection
	Params       QueryParams
	GroupByField string
	GroupCount   int
	TopKPerGroup int
}

GroupByVectorQuery describes a vector search retaining the best documents from the best distinct scalar groups.

type GroupResult 

type GroupResult struct {
	Value     string
	Documents []Document
}

GroupResult contains one baseline-compatible string group value and its metric-ordered, projected documents.

type HNSWIndexParams 

type HNSWIndexParams struct {
	Metric              MetricType
	M                   int
	EFConstruction      int
	Quantize            QuantizeType
	UseContiguousMemory bool
	Quantizer           QuantizerParams
}

HNSWIndexParams configures a dense or sparse HNSW index.

func NewHNSWIndexParams 

func NewHNSWIndexParams(metric MetricType) HNSWIndexParams

func (HNSWIndexParams) IndexType 

func (HNSWIndexParams) IndexType() IndexType

func (HNSWIndexParams) Validate 

func (p HNSWIndexParams) Validate() error

type HNSWQueryParams 

type HNSWQueryParams struct {
	QueryOptions
	EF             int
	PrefetchOffset uint32
	PrefetchLines  uint32
}

HNSWQueryParams configures HNSW graph traversal.

func NewHNSWQueryParams 

func NewHNSWQueryParams() HNSWQueryParams

func (HNSWQueryParams) IndexType 

func (HNSWQueryParams) IndexType() IndexType

func (HNSWQueryParams) Validate 

func (p HNSWQueryParams) Validate() error

type HNSWRaBitQIndexParams 

type HNSWRaBitQIndexParams struct {
	Metric         MetricType
	TotalBits      int
	NumClusters    int
	SampleCount    int
	M              int
	EFConstruction int
}

HNSWRaBitQIndexParams configures an HNSW index backed by RaBitQ codes.

func NewHNSWRaBitQIndexParams 

func NewHNSWRaBitQIndexParams(metric MetricType) HNSWRaBitQIndexParams

func (HNSWRaBitQIndexParams) IndexType 

func (HNSWRaBitQIndexParams) IndexType() IndexType

func (HNSWRaBitQIndexParams) Validate 

func (p HNSWRaBitQIndexParams) Validate() error

type HNSWRaBitQQueryParams 

type HNSWRaBitQQueryParams struct {
	QueryOptions
	EF int
}

HNSWRaBitQQueryParams configures HNSW traversal over RaBitQ codes.

func NewHNSWRaBitQQueryParams 

func NewHNSWRaBitQQueryParams() HNSWRaBitQQueryParams

func (HNSWRaBitQQueryParams) IndexType 

func (HNSWRaBitQQueryParams) IndexType() IndexType

func (HNSWRaBitQQueryParams) Validate 

func (p HNSWRaBitQQueryParams) Validate() error

type IVFIndexParams 

type IVFIndexParams struct {
	Metric      MetricType
	NList       int
	NIterations int
	UseSOAR     bool
	Quantize    QuantizeType
	Quantizer   QuantizerParams
}

IVFIndexParams configures an inverted-file vector index.

func NewIVFIndexParams 

func NewIVFIndexParams(metric MetricType) IVFIndexParams

func (IVFIndexParams) IndexType 

func (IVFIndexParams) IndexType() IndexType

func (IVFIndexParams) Validate 

func (p IVFIndexParams) Validate() error

type IVFQueryParams 

type IVFQueryParams struct {
	QueryOptions
	NProbe      int
	ScaleFactor float32
}

IVFQueryParams configures inverted-list probing and optional refinement.

func NewIVFQueryParams 

func NewIVFQueryParams() IVFQueryParams

func (IVFQueryParams) IndexType 

func (IVFQueryParams) IndexType() IndexType

func (IVFQueryParams) Validate 

func (p IVFQueryParams) Validate() error

type IndexParams 

type IndexParams interface {
	IndexType() IndexType
	Validate() error
	// contains filtered or unexported methods
}

IndexParams is the common, sealed interface for field index parameters. Concrete values have value semantics and may be stored directly in a FieldSchema.

type IndexType 

type IndexType uint32

IndexType identifies the index implementation attached to a field.

The numeric values match the public C++ header at baseline commit 58375ff. In particular, DiskANN is 5 and Vamana is 6. The legacy C++ protobuf used the opposite values; Go disk codecs must therefore map them explicitly. 

const (
	IndexTypeUndefined  IndexType = 0
	IndexTypeHNSW       IndexType = 1
	IndexTypeIVF        IndexType = 2
	IndexTypeFlat       IndexType = 3
	IndexTypeHNSWRaBitQ IndexType = 4
	IndexTypeDiskANN    IndexType = 5
	IndexTypeVamana     IndexType = 6
	IndexTypeInvert     IndexType = 10
	IndexTypeFTS        IndexType = 11
)

func (IndexType) IsVector 

func (t IndexType) IsVector() bool

IsVector reports whether t is a vector index type.

func (IndexType) String 

func (t IndexType) String() string

func (IndexType) Valid 

func (t IndexType) Valid() bool

Valid reports whether t is a value defined by the public API.

type Int32Array 

type Int32Array []int32

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type Int64Array 

type Int64Array []int64

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type InvertIndexParams 

type InvertIndexParams struct {
	EnableRangeOptimization bool
	EnableExtendedWildcard  bool
}

InvertIndexParams configures a scalar inverted index.

func NewInvertIndexParams 

func NewInvertIndexParams() InvertIndexParams

NewInvertIndexParams returns baseline-compatible inverted-index defaults.

func (InvertIndexParams) IndexType 

func (InvertIndexParams) IndexType() IndexType

func (InvertIndexParams) Validate 

func (InvertIndexParams) Validate() error

type LogLevel 

type LogLevel uint32

LogLevel preserves the pinned public logging severity order. 

const (
	LogLevelDebug LogLevel = iota
	LogLevelInfo
	LogLevelWarn
	LogLevelError
	LogLevelFatal
)

func (LogLevel) String 

func (l LogLevel) String() string

func (LogLevel) Valid 

func (l LogLevel) Valid() bool

Valid reports whether l is a public logging level.

type MetricType 

type MetricType uint32

MetricType identifies the distance or similarity function used by a vector index. Lower scores rank first for L2, cosine distance, and MIPSL2; higher scores rank first for IP. 

const (
	MetricTypeUndefined MetricType = 0
	MetricTypeL2        MetricType = 1
	MetricTypeIP        MetricType = 2
	MetricTypeCosine    MetricType = 3
	MetricTypeMIPSL2    MetricType = 4
)

func (MetricType) String 

func (t MetricType) String() string

func (MetricType) Valid 

func (t MetricType) Valid() bool

Valid reports whether t is a value defined by the public API.

type MultiQuery 

type MultiQuery struct {
	Queries    []SubQuery
	TopK       int
	Filter     string
	Projection Projection
	Reranker   Reranker
}

MultiQuery combines vector, sparse-vector, and full-text candidate lists through a Reranker. Nil selects NewRRFReranker. At least two sub-queries are required. Zero TopK selects DefaultMultiQueryTopK.

type Operator 

type Operator uint32

Operator identifies a write operation. 

const (
	OperatorInsert Operator = 0
	OperatorUpsert Operator = 1
	OperatorUpdate Operator = 2
	OperatorDelete Operator = 3
)

func (Operator) String 

func (o Operator) String() string

func (Operator) Valid 

func (o Operator) Valid() bool

Valid reports whether o is a value defined by the public API.

type OptimizeOptions 

type OptimizeOptions struct{ Concurrency int }

OptimizeOptions controls segment-optimization concurrency.

type Projection 

type Projection struct {
	OutputFields   []string
	IncludeVectors bool
}

Projection describes result shaping. A nil OutputFields slice selects every scalar/array field, a non-nil empty slice selects none, and a non-empty slice selects the named scalar/array fields. IncludeVectors independently includes every vector field.

func (Projection) Clone 

func (p Projection) Clone() Projection

Clone returns an independent projection while preserving nil-versus-empty field selection semantics.

func (Projection) Validate 

func (p Projection) Validate(schema CollectionSchema) error

Validate checks output field count, duplicates, existence, and scalar type. The special field "*" selects every scalar field and must appear alone.

type QuantizeType 

type QuantizeType uint32

QuantizeType identifies a vector quantization scheme. 

const (
	QuantizeTypeUndefined QuantizeType = 0
	QuantizeTypeFP16      QuantizeType = 1
	QuantizeTypeInt8      QuantizeType = 2
	QuantizeTypeInt4      QuantizeType = 3
	QuantizeTypeRaBitQ    QuantizeType = 4
)

func (QuantizeType) String 

func (t QuantizeType) String() string

func (QuantizeType) Valid 

func (t QuantizeType) Valid() bool

Valid reports whether t is a value defined by the public API.

type QuantizerParams 

type QuantizerParams struct {
	// EnableRotate rotates vectors before INT8 or INT4 quantization.
	EnableRotate bool
}

QuantizerParams configures preprocessing shared by quantized vector indexes.

type QueryOptions 

type QueryOptions struct {
	Radius     float32
	Linear     bool
	UseRefiner bool
}

QueryOptions contains controls shared by vector query parameter types. A zero Radius disables radius filtering.

type QueryParams 

type QueryParams interface {
	IndexType() IndexType
	Validate() error
	// contains filtered or unexported methods
}

QueryParams is the common, sealed interface for index-specific search controls.

type RRFReranker 

type RRFReranker struct {
	RankConstant int
}

RRFReranker combines ranks without inspecting source scores. A document at zero-based rank r contributes 1/(RankConstant+r+1) in each batch where it occurs. Documents are fused by primary key, matching the pinned baseline.

func NewRRFReranker 

func NewRRFReranker() RRFReranker

NewRRFReranker returns the pinned rank-constant default.

func (RRFReranker) Rerank 

func (r RRFReranker) Rerank(ctx context.Context, batches []RerankBatch, topK int) ([]Document, error)

Rerank applies reciprocal rank fusion, returning at most topK distinct documents by descending fused score. Equal scores are ordered by primary key and then DocID so results remain deterministic across processes.

func (RRFReranker) Validate 

func (r RRFReranker) Validate() error

Validate rejects a negative rank constant. Zero is valid and intentionally differs from the default; use NewRRFReranker for baseline defaults.

type RelationOp 

type RelationOp uint32

RelationOp identifies the boolean relationship between filter expressions. 

const (
	RelationOpNone RelationOp = 0
	RelationOpAnd  RelationOp = 1
	RelationOpOr   RelationOp = 2
)

func (RelationOp) String 

func (o RelationOp) String() string

func (RelationOp) Valid 

func (o RelationOp) Valid() bool

Valid reports whether o is a value defined by the public API.

type RerankBatch 

type RerankBatch struct {
	Field     FieldSchema
	Documents []Document
}

RerankBatch contains one projected, score-ordered candidate list and the corresponding independent field schema. Batches retain SubQuery order. Documents and their field values are owned by the batch and may be changed by the reranker without mutating the collection snapshot.

type Reranker 

type Reranker interface {
	Rerank(ctx context.Context, batches []RerankBatch, topK int) ([]Document, error)
}

Reranker combines MultiQuery candidate batches. Implementations may adjust scores and order but must return distinct documents drawn from the supplied batches. Implementations shared by concurrent calls must be concurrency-safe.

type RuntimeConfig 

type RuntimeConfig struct {
	MemoryLimitBytes uint64
	Logger           *slog.Logger
	LogLevel         LogLevel

	QueryConcurrency   int
	QueryThreadBinding bool

	InvertToForwardScanRatio float32
	BruteForceByKeysRatio    float32
	FTSBruteForceByKeysRatio float32

	OptimizeConcurrency   int
	OptimizeThreadBinding bool
	JiebaDictionaryDir    string
}

RuntimeConfig controls process-wide query and maintenance resources. Call ConfigureRuntime before creating or opening the first collection. A zero MemoryLimitBytes leaves heap sizing to the Go runtime; a non-zero value is a conservative admission budget for collection query and maintenance scratch.

func CurrentRuntimeConfig 

func CurrentRuntimeConfig() RuntimeConfig

CurrentRuntimeConfig returns the configured value or defaults without freezing the one-shot configuration lifecycle.

func NewRuntimeConfig 

func NewRuntimeConfig() RuntimeConfig

NewRuntimeConfig returns native Go defaults aligned with the pinned planner ratios and current GOMAXPROCS. Explicit memory admission is disabled.

func (RuntimeConfig) Validate 

func (c RuntimeConfig) Validate() error

Validate checks process resource limits and planner ratios. CPU thread binding is explicit NotSupported because Go schedules goroutines onto its own cross-platform worker threads.

Example 
package main

import (
	"fmt"

	"github.com/gorse-io/xvec"
)

func main() {
	config := xvec.NewRuntimeConfig()
	config.MemoryLimitBytes = xvec.MinRuntimeMemoryLimit
	config.QueryConcurrency = 4
	config.OptimizeConcurrency = 2

	fmt.Println(config.Validate() == nil)

}

Output:
true

type RuntimeStats 

type RuntimeStats struct {
	MemoryLimitBytes uint64
	MemoryInUseBytes uint64
	PeakMemoryBytes  uint64
	MemoryWaiters    uint64

	ActiveQueries    uint64
	PeakQueries      uint64
	QueuedQueries    uint64
	CompletedQueries uint64

	ActiveOptimizeTasks    uint64
	PeakOptimizeTasks      uint64
	QueuedOptimizeTasks    uint64
	CompletedOptimizeTasks uint64
}

RuntimeStats is a concurrency-safe point-in-time view of process resource usage.

func CurrentRuntimeStats 

func CurrentRuntimeStats() RuntimeStats

CurrentRuntimeStats returns process admission and scratch-budget counters. Calling it initializes defaults when ConfigureRuntime has not run yet.

type SparseVector 

type SparseVector interface {
	DataType() DataType
	Len() int
	// contains filtered or unexported methods
}

SparseVector is implemented by both supported sparse-vector value types.

type SparseVectorFP16 

type SparseVectorFP16 struct {
	Indices []uint32
	Values  []Float16
}

SparseVectorFP16 stores matching coordinate and binary16 value slices.

func (SparseVectorFP16) Canonical 

func (v SparseVectorFP16) Canonical() (SparseVectorFP16, error)

Canonical returns an independent copy sorted by coordinate.

func (SparseVectorFP16) DataType 

func (SparseVectorFP16) DataType() DataType

func (SparseVectorFP16) Len 

func (v SparseVectorFP16) Len() int

func (SparseVectorFP16) Validate 

func (v SparseVectorFP16) Validate() error

Validate checks lengths, the coordinate-count limit, and uniqueness. Input coordinates may be unsorted; Canonical returns a sorted copy.

type SparseVectorFP32 

type SparseVectorFP32 struct {
	Indices []uint32
	Values  []float32
}

SparseVectorFP32 stores matching coordinate and float32 value slices.

func (SparseVectorFP32) Canonical 

func (v SparseVectorFP32) Canonical() (SparseVectorFP32, error)

Canonical returns an independent copy sorted by coordinate.

func (SparseVectorFP32) DataType 

func (SparseVectorFP32) DataType() DataType

func (SparseVectorFP32) Len 

func (v SparseVectorFP32) Len() int

func (SparseVectorFP32) Validate 

func (v SparseVectorFP32) Validate() error

Validate checks lengths, the coordinate-count limit, and uniqueness. Input coordinates may be unsorted; Canonical returns a sorted copy.

type StringArray 

type StringArray []string

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type SubQuery 

type SubQuery struct {
	Field         string
	DenseVector   DenseVector
	SparseVector  SparseVector
	PrimaryKey    string
	FTS           *FTSClause
	Params        QueryParams
	NumCandidates int
}

SubQuery describes one candidate-producing branch of MultiQuery. Exactly one of DenseVector, SparseVector, PrimaryKey, and FTS must be set. PrimaryKey resolves the query vector from Field in the same immutable collection snapshot used by every branch. A zero NumCandidates selects DefaultSubQueryCandidates.

type Uint32Array 

type Uint32Array []uint32

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type Uint64Array 

type Uint64Array []uint64

Explicit array types keep document values unambiguous when their element type is also used by a vector field.

type VamanaIndexParams 

type VamanaIndexParams struct {
	Metric              MetricType
	MaxDegree           int
	SearchListSize      int
	Alpha               float32
	MaxOcclusionSize    int
	SaturateGraph       bool
	UseContiguousMemory bool
	UseIDMap            bool
	Quantize            QuantizeType
	Quantizer           QuantizerParams
}

VamanaIndexParams configures an in-memory Vamana graph.

func NewVamanaIndexParams 

func NewVamanaIndexParams(metric MetricType) VamanaIndexParams

func (VamanaIndexParams) IndexType 

func (VamanaIndexParams) IndexType() IndexType

func (VamanaIndexParams) Validate 

func (p VamanaIndexParams) Validate() error

type VamanaQueryParams 

type VamanaQueryParams struct {
	QueryOptions
	EFSearch       int
	PrefetchOffset uint32
	PrefetchLines  uint32
}

VamanaQueryParams configures Vamana graph traversal.

func NewVamanaQueryParams 

func NewVamanaQueryParams() VamanaQueryParams

func (VamanaQueryParams) IndexType 

func (VamanaQueryParams) IndexType() IndexType

func (VamanaQueryParams) Validate 

func (p VamanaQueryParams) Validate() error

type VectorBinary32 

type VectorBinary32 []uint32

func (VectorBinary32) DataType 

func (VectorBinary32) DataType() DataType

func (VectorBinary32) Dimension 

func (v VectorBinary32) Dimension() int

type VectorBinary64 

type VectorBinary64 []uint64

func (VectorBinary64) DataType 

func (VectorBinary64) DataType() DataType

func (VectorBinary64) Dimension 

func (v VectorBinary64) Dimension() int

type VectorFP16 

type VectorFP16 []Float16

func (VectorFP16) DataType 

func (VectorFP16) DataType() DataType

func (VectorFP16) Dimension 

func (v VectorFP16) Dimension() int

type VectorFP32 

type VectorFP32 []float32

func (VectorFP32) DataType 

func (VectorFP32) DataType() DataType

func (VectorFP32) Dimension 

func (v VectorFP32) Dimension() int

type VectorFP64 

type VectorFP64 []float64

func (VectorFP64) DataType 

func (VectorFP64) DataType() DataType

func (VectorFP64) Dimension 

func (v VectorFP64) Dimension() int

type VectorInt4 

type VectorInt4 []int8

VectorInt4 stores one signed value per element, in the range [-8, 7]. Packing is a disk-codec concern and is not exposed in the Go API.

func (VectorInt4) DataType 

func (VectorInt4) DataType() DataType

func (VectorInt4) Dimension 

func (v VectorInt4) Dimension() int

func (VectorInt4) Validate 

func (v VectorInt4) Validate() error

Validate checks that every explicit INT4 element is representable.

type VectorInt8 

type VectorInt8 []int8

func (VectorInt8) DataType 

func (VectorInt8) DataType() DataType

func (VectorInt8) Dimension 

func (v VectorInt8) Dimension() int

type VectorInt16 

type VectorInt16 []int16

func (VectorInt16) DataType 

func (VectorInt16) DataType() DataType

func (VectorInt16) Dimension 

func (v VectorInt16) Dimension() int

type VectorQuery 

type VectorQuery struct {
	Field        string
	DenseVector  DenseVector
	SparseVector SparseVector
	PrimaryKey   string
	FTS          *FTSClause
	TopK         int
	Filter       string
	Projection   Projection
	Params       QueryParams
}

VectorQuery describes one collection search. Set exactly one of DenseVector, SparseVector, PrimaryKey, and FTS. PrimaryKey resolves the vector stored in Field from the query snapshot. Leaving all targets and Field empty performs a scalar filter scan in ascending document-ID order.

type WeightedReranker 

type WeightedReranker struct {
	Weights []float64
}

WeightedReranker normalizes each branch score according to its field metric, multiplies it by the corresponding weight, and sums by primary key. Weights may be negative but must be finite, and their count must match the batches.

func NewWeightedReranker 

func NewWeightedReranker(weights ...float64) WeightedReranker

NewWeightedReranker returns a reranker with an owned weight snapshot.

func (WeightedReranker) Rerank 

func (r WeightedReranker) Rerank(ctx context.Context, batches []RerankBatch, topK int) ([]Document, error)

Rerank applies baseline metric normalization and deterministic weighted score fusion. The first occurrence supplies the returned document payload.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec"
)

func main() {
	reranker := xvec.NewWeightedReranker(0.5, 0.5)
	results, err := reranker.Rerank(context.Background(), []xvec.RerankBatch{
		{
			Field: xvec.FieldSchema{
				Name: "embedding", DataType: xvec.DataTypeVectorFP32, Dimension: 2,
				Index: xvec.NewFlatIndexParams(xvec.MetricTypeL2),
			},
			Documents: []xvec.Document{
				{PrimaryKey: "a", DocID: 1, Score: 0},
				{PrimaryKey: "b", DocID: 2, Score: 1},
			},
		},
		{
			Field: xvec.FieldSchema{Name: "body", DataType: xvec.DataTypeString, Index: xvec.NewFTSIndexParams()},
			Documents: []xvec.Document{
				{PrimaryKey: "a", DocID: 1, Score: 1},
			},
		},
	}, 2)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s %.3f\n", results[0].PrimaryKey, results[0].Score)

}

Output:
a 0.750

func (WeightedReranker) Validate 

func (r WeightedReranker) Validate() error

Validate checks that every configured weight is finite.

type WriteResult 

type WriteResult struct {
	PrimaryKey string
	DocID      uint64
	Err        error
}

WriteResult reports one document mutation in input order.

Source Files

View all Source files

Directories

Path Synopsis
cmd
vector-db-bench command
internal
ailego/algorithm
Package algorithm provides reusable Ailego algorithms that do not depend on vector-index or database types.
 Package algorithm provides reusable Ailego algorithms that do not depend on vector-index or database types.
ailego/container
ailego/hash
ailego/io
ailego/math
ailego/parallel
ailego/utility
core
core/metric
Package metric provides dense-vector score computation and ordering.
 Package metric provides dense-vector score computation and ordering.
db
db/common
Package common provides the deliberately small Pebble surface used by immutable collection index artifacts.
 Package common provides the deliberately small Pebble surface used by immutable collection index artifacts.
db/index/column/fts_column
db/index/column/fts_column/tokenizer
db/index/common
db/index/segment
db/index/storage/wal
db/sqlengine
Package sql parses and evaluates the SQL-style scalar filter language used by xvec.
 Package sql parses and evaluates the SQL-style scalar filter language used by xvec.
floats
Package floats provides allocation-free float32 vector kernels.
 Package floats provides allocation-free float32 vector kernels.
thirdparty
jieba
Package jieba provides the pure-Go Jieba segmentation used by xvec.
 Package jieba provides the pure-Go Jieba segmentation used by xvec.
snowball
snowball/arabic
snowball/armenian
snowball/basque
snowball/catalan
snowball/czech
snowball/danish
snowball/dutch
snowball/dutch_porter
snowball/english
snowball/esperanto
snowball/estonian
snowball/finnish
snowball/french
snowball/german
snowball/greek
snowball/hindi
snowball/hungarian
snowball/indonesian
snowball/irish
snowball/italian
snowball/lithuanian
snowball/nepali
snowball/norwegian
snowball/persian
snowball/polish
snowball/porter
snowball/portuguese
snowball/romanian
snowball/russian
snowball/serbian
snowball/sesotho
snowball/spanish
snowball/stemmers
snowball/swedish
snowball/tamil
snowball/turkish
snowball/yiddish

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.