veclite

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 21, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package veclite provides an embeddable vector database with zero external dependencies. It stores vectors with metadata in a single file using gob encoding.

Basic usage: 

db, err := veclite.Open("data.veclite")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

coll := db.Collection("embeddings")
id, err := coll.Insert(vector, map[string]any{"file": "main.go"})

results, err := coll.Search(queryVector, veclite.TopK(10))

Index

Constants

View Source 
const Version = "0.1.0"

Version is the library version.

Variables

View Source 
var (
	// ErrNotFound is returned when a record or collection is not found.
	ErrNotFound = errors.New("veclite: not found")

	// ErrDimensionMismatch is returned when vector dimensions don't match.
	ErrDimensionMismatch = errors.New("veclite: dimension mismatch")

	// ErrEmptyVector is returned when an empty vector is provided.
	ErrEmptyVector = errors.New("veclite: empty vector")

	// ErrCollectionExists is returned when trying to create a collection that already exists.
	ErrCollectionExists = errors.New("veclite: collection already exists")

	// ErrDatabaseClosed is returned when operations are attempted on a closed database.
	ErrDatabaseClosed = errors.New("veclite: database closed")

	// ErrInvalidPath is returned when an invalid file path is provided.
	ErrInvalidPath = errors.New("veclite: invalid path")

	// ErrCorruptedFile is returned when the database file is corrupted.
	ErrCorruptedFile = errors.New("veclite: corrupted file")

	// ErrInvalidVersion is returned when the file version is not supported.
	ErrInvalidVersion = errors.New("veclite: unsupported file version")

	// ErrBatchSizeMismatch is returned when batch operation input sizes don't match.
	ErrBatchSizeMismatch = errors.New("veclite: batch size mismatch")
)

Sentinel errors for common conditions.

Functions

This section is empty.

Types

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

Collection represents a collection of vectors with the same dimension.

func (*Collection) All 

func (c *Collection) All() []*Record

All returns all records in the collection.

func (*Collection) Clear 

func (c *Collection) Clear()

Clear removes all records from the collection.

func (*Collection) Count 

func (c *Collection) Count() int

Count returns the number of records in the collection.

func (*Collection) Delete 

func (c *Collection) Delete(id uint64) error

Delete removes a record by ID.

func (*Collection) DeleteWhere 

func (c *Collection) DeleteWhere(filters ...Filter) (int, error)

DeleteWhere removes all records matching the filters. Returns the number of deleted records.

func (*Collection) Dimension 

func (c *Collection) Dimension() int

Dimension returns the vector dimension. Returns 0 if no vectors have been inserted yet.

func (*Collection) DistanceType 

func (c *Collection) DistanceType() floats.DistanceType

DistanceType returns the distance metric type.

func (*Collection) Find 

func (c *Collection) Find(filters ...Filter) ([]*Record, error)

Find retrieves all records matching the filters.

func (*Collection) FindOne 

func (c *Collection) FindOne(filters ...Filter) (*Record, error)

FindOne retrieves the first record matching the filters.

func (*Collection) Get 

func (c *Collection) Get(id uint64) (*Record, error)

Get retrieves a record by ID.

func (*Collection) GetVector 

func (c *Collection) GetVector(id uint64) ([]float32, error)

GetVector retrieves just the vector for a record.

func (*Collection) Insert 

func (c *Collection) Insert(vector []float32, payload map[string]any) (uint64, error)

Insert adds a vector with optional payload to the collection. Returns the assigned record ID.

func (*Collection) InsertBatch 

func (c *Collection) InsertBatch(vectors [][]float32, payloads []map[string]any) ([]uint64, error)

InsertBatch adds multiple vectors with payloads to the collection. Returns the assigned record IDs. If payloads is nil or shorter than vectors, missing payloads are treated as nil.

func (*Collection) Name 

func (c *Collection) Name() string

Name returns the collection name.

func (*Collection) Search 

func (c *Collection) Search(query []float32, opts ...SearchOption) ([]Result, error)

Search finds the most similar vectors to the query vector.

func (*Collection) Stats 

func (c *Collection) Stats() CollectionStats

Stats returns statistics about the collection.

func (*Collection) Update 

func (c *Collection) Update(id uint64, payload map[string]any) error

Update updates the payload for a record.

type CollectionOption 

type CollectionOption interface {
	// contains filtered or unexported methods
}

CollectionOption configures a collection.

func WithDimension 

func WithDimension(dim int) CollectionOption

WithDimension sets the vector dimension for the collection. If set, all vectors must match this dimension. If not set (0), the dimension is determined by the first insert.

func WithDistanceType 

func WithDistanceType(t floats.DistanceType) CollectionOption

WithDistanceType sets the distance metric for the collection. Default is cosine similarity.

type CollectionSnapshot 

type CollectionSnapshot struct {
	// Name is the collection name.
	Name string

	// Dimension is the vector dimension.
	Dimension int

	// DistanceType is the distance metric.
	DistanceType floats.DistanceType

	// NextID is the next record ID to assign.
	NextID uint64

	// Records contains all records in the collection.
	Records []*RecordSnapshot

	// CreatedAt is when the collection was created.
	CreatedAt time.Time

	// UpdatedAt is when the collection was last modified.
	UpdatedAt time.Time
}

CollectionSnapshot is the serializable state of a collection.

func NewCollectionSnapshot 

func NewCollectionSnapshot(name string, dimension int, distanceType floats.DistanceType) *CollectionSnapshot

NewCollectionSnapshot creates a new empty collection snapshot.

type CollectionStats 

type CollectionStats struct {
	// Name is the collection name.
	Name string

	// Count is the number of records in the collection.
	Count int

	// Dimension is the vector dimension (0 if not yet set).
	Dimension int

	// DistanceType is the distance metric used.
	DistanceType string
}

CollectionStats contains statistics about a collection.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a VecLite database.

func Open 

func Open(path string, opts ...Option) (*DB, error)

Open opens or creates a VecLite database at the given path. Use ":memory:" for an in-memory database that won't be persisted.

func (*DB) Close 

func (db *DB) Close() error

Close closes the database, syncing any pending changes.

func (*DB) Collection 

func (db *DB) Collection(name string) *Collection

Collection returns a collection by name, creating it if it doesn't exist. This is the preferred way to get collections for most use cases.

func (*DB) Collections 

func (db *DB) Collections() []string

Collections returns the names of all collections.

func (*DB) CreateCollection 

func (db *DB) CreateCollection(name string, opts ...CollectionOption) (*Collection, error)

CreateCollection creates a new collection with the given options. Returns an error if the collection already exists.

func (*DB) DropCollection 

func (db *DB) DropCollection(name string) error

DropCollection removes a collection and all its data.

func (*DB) GetCollection 

func (db *DB) GetCollection(name string) (*Collection, error)

GetCollection returns an existing collection or ErrNotFound.

func (*DB) HasCollection 

func (db *DB) HasCollection(name string) bool

HasCollection returns true if a collection exists.

func (*DB) IsClosed 

func (db *DB) IsClosed() bool

IsClosed returns true if the database is closed.

func (*DB) Path 

func (db *DB) Path() string

Path returns the database file path.

func (*DB) Stats 

func (db *DB) Stats() DatabaseStats

Stats returns statistics about the database.

func (*DB) Sync 

func (db *DB) Sync() error

Sync writes all pending changes to storage.

type DatabaseSnapshot 

type DatabaseSnapshot struct {
	// Version is the file format version.
	Version uint32

	// Collections maps collection names to their snapshots.
	Collections map[string]*CollectionSnapshot

	// CreatedAt is when the database was created.
	CreatedAt time.Time

	// UpdatedAt is when the database was last modified.
	UpdatedAt time.Time
}

DatabaseSnapshot is the serializable state of the database.

func NewDatabaseSnapshot 

func NewDatabaseSnapshot() *DatabaseSnapshot

NewDatabaseSnapshot creates a new empty database snapshot.

type DatabaseStats 

type DatabaseStats struct {
	// Path is the database file path (":memory:" for in-memory).
	Path string

	// Collections is the number of collections.
	Collections int

	// TotalRecords is the total number of records across all collections.
	TotalRecords int

	// CollectionStats contains stats for each collection.
	CollectionStats []CollectionStats
}

DatabaseStats contains statistics about the database.

type DimensionError 

type DimensionError struct {
	Expected int
	Got      int
}

DimensionError provides details about dimension mismatches.

func (*DimensionError) Error 

func (e *DimensionError) Error() string

func (*DimensionError) Unwrap 

func (e *DimensionError) Unwrap() error

type FileStorage 

type FileStorage struct {
	// contains filtered or unexported fields
}

FileStorage is a file-based storage implementation. Uses gob encoding with atomic writes for durability.

func NewFileStorage 

func NewFileStorage(path string) *FileStorage

NewFileStorage creates a new file storage for the given path.

func (*FileStorage) Close 

func (f *FileStorage) Close() error

Close is a no-op for file storage.

func (*FileStorage) Delete 

func (f *FileStorage) Delete() error

Delete removes the database file and any backup files.

func (*FileStorage) Exists 

func (f *FileStorage) Exists() bool

Exists returns true if the database file exists.

func (*FileStorage) Load 

func (f *FileStorage) Load() (*DatabaseSnapshot, error)

Load reads the database from the file. Returns nil, nil if the file doesn't exist yet.

func (*FileStorage) Path 

func (f *FileStorage) Path() string

Path returns the file path.

func (*FileStorage) Save 

func (f *FileStorage) Save(snapshot *DatabaseSnapshot) error

Save writes the database to the file using atomic write pattern. Writes to .tmp file, then renames old to .bak, then renames .tmp to final.

type Filter 

type Filter interface {
	// Match returns true if the record matches the filter criteria.
	Match(r *Record) bool
}

Filter is an interface for filtering records based on payload values.

func And 

func And(filters ...Filter) Filter

And creates a filter that matches records matching all given filters.

func Contains 

func Contains(key, substr string) Filter

Contains creates a filter that matches records where payload[key] contains the substring.

func Equal 

func Equal(key string, value any) Filter

Equal creates a filter that matches records where payload[key] equals value.

func Exists 

func Exists(key string) Filter

Exists creates a filter that matches records where payload[key] exists.

func Glob 

func Glob(key, pattern string) Filter

Glob creates a filter that matches records where payload[key] matches the glob pattern.

func In 

func In(key string, values ...any) Filter

In creates a filter that matches records where payload[key] is in the given values.

func Not 

func Not(filter Filter) Filter

Not creates a filter that negates the given filter.

func NotEqual 

func NotEqual(key string, value any) Filter

NotEqual creates a filter that matches records where payload[key] does not equal value.

func NotIn 

func NotIn(key string, values ...any) Filter

NotIn creates a filter that matches records where payload[key] is not in the given values.

func Or 

func Or(filters ...Filter) Filter

Or creates a filter that matches records matching any given filter.

func Prefix 

func Prefix(key, prefix string) Filter

Prefix creates a filter that matches records where payload[key] has the given prefix.

func Suffix 

func Suffix(key, suffix string) Filter

Suffix creates a filter that matches records where payload[key] has the given suffix.

type FilterFunc 

type FilterFunc func(r *Record) bool

FilterFunc is a function adapter for the Filter interface.

func (FilterFunc) Match 

func (f FilterFunc) Match(r *Record) bool

Match implements Filter interface.

type MemoryStorage 

type MemoryStorage struct {
	// contains filtered or unexported fields
}

MemoryStorage is an in-memory storage implementation. Data is not persisted and will be lost when the database is closed.

func NewMemoryStorage 

func NewMemoryStorage() *MemoryStorage

NewMemoryStorage creates a new in-memory storage.

func (*MemoryStorage) Close 

func (m *MemoryStorage) Close() error

Close is a no-op for memory storage.

func (*MemoryStorage) Load 

func (m *MemoryStorage) Load() (*DatabaseSnapshot, error)

Load returns the stored snapshot or nil if none exists.

func (*MemoryStorage) Save 

func (m *MemoryStorage) Save(snapshot *DatabaseSnapshot) error

Save stores the snapshot in memory.

type NotFoundError 

type NotFoundError struct {
	Type string // "record", "collection", etc.
	ID   string // Identifier that was not found
}

NotFoundError provides details about what was not found.

func (*NotFoundError) Error 

func (e *NotFoundError) Error() string

func (*NotFoundError) Unwrap 

func (e *NotFoundError) Unwrap() error

type Option 

type Option interface {
	// contains filtered or unexported methods
}

Option configures the database.

func WithReadOnly 

func WithReadOnly(enabled bool) Option

WithReadOnly opens the database in read-only mode. Write operations will return an error.

func WithSyncOnWrite 

func WithSyncOnWrite(enabled bool) Option

WithSyncOnWrite enables automatic sync after each write operation. This is slower but ensures durability.

type Record 

type Record struct {
	// ID is the unique identifier for this record.
	ID uint64

	// Vector is the embedding vector.
	Vector []float32

	// Payload contains arbitrary metadata associated with the vector.
	Payload map[string]any

	// CreatedAt is when the record was inserted.
	CreatedAt time.Time

	// UpdatedAt is when the record was last updated.
	UpdatedAt time.Time
}

Record represents a stored vector with its metadata.

func (*Record) Clone 

func (r *Record) Clone() *Record

Clone creates a deep copy of the record.

type RecordSnapshot 

type RecordSnapshot struct {
	// ID is the record's unique identifier.
	ID uint64

	// Vector is the embedding vector.
	Vector []float32

	// Payload contains arbitrary metadata.
	Payload map[string]any

	// CreatedAt is when the record was inserted.
	CreatedAt time.Time

	// UpdatedAt is when the record was last updated.
	UpdatedAt time.Time
}

RecordSnapshot is the serializable state of a record.

type Result 

type Result struct {
	// Record is the matched record.
	Record *Record

	// Score is the similarity/distance score.
	// For cosine/dot: higher is more similar.
	// For euclidean: lower is more similar.
	Score float32
}

Result represents a search result with its similarity score.

type SearchOption 

type SearchOption interface {
	// contains filtered or unexported methods
}

SearchOption configures search behavior.

func Threshold 

func Threshold(t float32) SearchOption

Threshold sets the minimum similarity score for results. For cosine/dot: results with score >= threshold are returned. For euclidean: results with score <= threshold are returned.

func TopK 

func TopK(k int) SearchOption

TopK sets the maximum number of results to return. Default is 10.

func WithFilter 

func WithFilter(f Filter) SearchOption

WithFilter adds a filter to the search. Multiple filters are combined with AND logic.

func WithFilters 

func WithFilters(filters ...Filter) SearchOption

WithFilters adds multiple filters to the search. All filters are combined with AND logic.

type Storage 

type Storage interface {
	// Load reads the database from storage.
	// Returns nil, nil if the database doesn't exist yet.
	Load() (*DatabaseSnapshot, error)

	// Save writes the database to storage.
	Save(snapshot *DatabaseSnapshot) error

	// Close releases any resources held by the storage.
	Close() error
}

Storage is the interface for database persistence.

type StorageError 

type StorageError struct {
	Op  string // Operation that failed
	Err error  // Underlying error
}

StorageError wraps storage-related errors with context.

func (*StorageError) Error 

func (e *StorageError) Error() string

func (*StorageError) Unwrap 

func (e *StorageError) Unwrap() error

Source Files

View all Source files

Directories

Path Synopsis
cmd
veclite command
Command veclite provides a CLI for inspecting VecLite databases.
 Command veclite provides a CLI for inspecting VecLite databases.
internal
floats
Package floats provides optimized floating-point vector operations.
 Package floats provides optimized floating-point vector operations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.