Documentation
¶
Overview ¶
Package vectorstore defines provider-neutral semantic indexing and search. Four independent interfaces split the surface by capability:
- Indexer indexes documents.
- Searcher finds similar documents by query + metadata filter.
- IDDeleter removes documents by identifier.
- FilterDeleter removes documents matching a metadata filter.
- Batcher supplies an order-preserving ingestion partition policy.
There is deliberately no aggregate Store interface: consumers depend only on the capabilities they call, and providers implement only what they can support. Batching remains an injected capability rather than a framework dependency. IndexRequest owns the shared validation and batching boundary before provider I/O. SearchRequest validates search input, while SearchResponse validates ranked provider output against that request.
Indexed documents have a caller-assigned, non-empty ID and non-empty text. Providers preserve both values so every successful SearchResult is immediately usable by retrieval pipelines. Providers never generate IDs, and a vector-index-plus-external-document-store architecture must hydrate results explicitly outside these capabilities rather than return partial Documents. Provider distances and similarities are converted to the common Score contract with the ScoreFrom functions in this package.
Metadata filtering uses the filter mini-language: build predicates with typed constructors or parse them from text with filter.Parse. See github.com/Tangerg/scope/core/vectorstore/filter.
Quick start:
expr, _ := filter.Parse(`category == 'tech' AND year >= 2020`)
req := &vectorstore.SearchRequest{
Query: "attention",
Options: vectorstore.SearchOptions{TopK: 5, MinScore: 0.7, Filter: expr},
}
response, err := searcher.Search(ctx, req)
Example ¶
package main
import (
"fmt"
"github.com/Tangerg/scope/core/vectorstore"
"github.com/Tangerg/scope/core/vectorstore/filter"
)
func main() {
expr := filter.EQ("category", "wildlife")
request := &vectorstore.SearchRequest{
Query: "scope habitat",
Options: vectorstore.SearchOptions{
TopK: 5, MinScore: 0.7, Filter: expr,
},
}
if err := request.Validate(); err != nil {
panic(err)
}
fmt.Println(request.Query, request.Options.TopK, request.Options.MinScore, expr.Operator())
}
Output: scope habitat 5 0.7 ==
Index ¶
- Constants
- Variables
- type Batcher
- type FilterDeleter
- type IDDeleter
- type IndexRequest
- type Indexer
- type Score
- func ScoreFromCosineDistance(distance float64) Score
- func ScoreFromCosineSimilarity(similarity float64) Score
- func ScoreFromDistance(distance float64) Score
- func ScoreFromInnerProduct(product float64) Score
- func ScoreFromNegativeInnerProductDistance(distance float64) Score
- func ScoreFromOneMinusInnerProductDistance(distance float64) Score
- func ScoreFromValue(value float64) Score
- type SearchOptions
- type SearchRequest
- type SearchResponse
- func (s *SearchResponse) Documents() []*document.Document
- func (s *SearchResponse) First() *SearchResult
- func (s SearchResponse) MarshalJSON() ([]byte, error)
- func (s *SearchResponse) UnmarshalJSON(data []byte) error
- func (s *SearchResponse) Validate() error
- func (s *SearchResponse) ValidateFor(request *SearchRequest) error
- type SearchResult
- type Searcher
Examples ¶
Constants ¶
const ( // DefaultTopK is used when [SearchOptions.TopK] is zero. DefaultTopK = 5 // MinSimilarityScore is the lowest valid score. MinSimilarityScore = 0.0 // MaxSimilarityScore is the highest valid score. MaxSimilarityScore = 1.0 )
Similarity-score range for SearchOptions.MinScore and search defaults.
Variables ¶
var ( ErrInvalidOptions = errors.New("vectorstore: invalid options") ErrInvalidRequest = errors.New("vectorstore: invalid request") ErrInvalidResponse = errors.New("vectorstore: invalid response") ErrInvalidScore = errors.New("vectorstore: invalid score") ErrEmptyDocuments = errors.New("vectorstore: documents must not be empty") ErrInvalidDocument = errors.New("vectorstore: invalid document") ErrMissingDocumentID = errors.New("vectorstore: document ID is required") ErrDuplicateDocumentID = errors.New("vectorstore: duplicate document ID") ErrMissingFilter = errors.New("vectorstore: filter is required") )
var ErrInvalidBatcherOutput = errors.New("vectorstore: invalid batcher output")
Functions ¶
This section is empty.
Types ¶
type Batcher ¶
type Batcher interface {
// Batch partitions the supplied pointers without cloning or retaining them.
// Every input pointer must occur exactly once in the output, global order is
// preserved, and empty batches are invalid. Context cancellation remains
// identifiable through errors.Is.
Batch(ctx context.Context, documents []*document.Document) ([][]*document.Document, error)
}
Batcher partitions documents for ingestion. It must preserve every document pointer exactly once and in input order, and it must not return empty batches. Implementations commonly come from document pipelines; stores depend only on this narrow capability contract.
type FilterDeleter ¶
type FilterDeleter interface {
// DeleteWhere removes every document matching predicate. Implementations return
// [ErrMissingFilter] for nil and reject invalid expressions.
DeleteWhere(ctx context.Context, predicate filter.Predicate) error
}
FilterDeleter removes documents selected by a metadata expression. It is a separate capability because some providers can search but cannot mutate their managed index.
type IDDeleter ¶
type IDDeleter interface {
// DeleteIDs removes the documents with the given ids. Unknown ids
// are ignored (idempotent); an empty slice is a no-op.
DeleteIDs(ctx context.Context, ids []string) error
}
IDDeleter removes documents by identifier. It is independent from FilterDeleter: providers frequently expose only one of the two paths.
type IndexRequest ¶
IndexRequest is one atomic indexing operation. It owns the complete provider-independent validation and batching lifecycle for its documents.
func NewIndexRequest ¶
func NewIndexRequest(documents []*document.Document) (*IndexRequest, error)
func (*IndexRequest) Batch ¶
func (i *IndexRequest) Batch(ctx context.Context, batcher Batcher) ([]*IndexRequest, error)
Batch delegates to batcher and returns validated, order-preserving child requests. The receiver itself must be valid.
func (IndexRequest) MarshalJSON ¶
func (i IndexRequest) MarshalJSON() ([]byte, error)
func (*IndexRequest) UnmarshalJSON ¶
func (i *IndexRequest) UnmarshalJSON(data []byte) error
func (*IndexRequest) Validate ¶
func (i *IndexRequest) Validate() error
type Indexer ¶
type Indexer interface {
// Index persists request documents using caller-assigned IDs. Existing IDs
// are replaced according to the backend's upsert semantics. Implementations
// validate the complete request before external I/O.
//
// Index never invents document IDs: its error-only result has no channel for
// returning generated identities to the caller.
Index(ctx context.Context, request *IndexRequest) error
}
Indexer embeds and indexes documents in the vector store. The store runs:
- Embedding (text → vector)
- Indexing (vector + metadata → searchable record)
- Storage (record → durable backend)
type Score ¶
type Score float64
Score is a provider-neutral similarity value in [0, 1].
func ScoreFromCosineDistance ¶
ScoreFromCosineDistance maps 1-cosine-similarity from [0, 2] to [0, 1].
func ScoreFromCosineSimilarity ¶
ScoreFromCosineSimilarity maps cosine similarity from [-1, 1] to [0, 1].
func ScoreFromDistance ¶
ScoreFromDistance maps a non-negative, unbounded distance to (0, 1], where zero is an exact match. Tiny negative values caused by floating-point error are treated as zero.
func ScoreFromInnerProduct ¶
ScoreFromInnerProduct maps an unbounded dot product monotonically into (0, 1).
func ScoreFromNegativeInnerProductDistance ¶
ScoreFromNegativeInnerProductDistance maps a provider distance defined as the negative dot product into the similarity range.
func ScoreFromOneMinusInnerProductDistance ¶
ScoreFromOneMinusInnerProductDistance maps a provider distance defined as 1-dot-product into the similarity range.
func ScoreFromValue ¶
ScoreFromValue clamps a finite provider score to the common range. Non-finite input becomes NaN so result validation reports the contract breach.
func (Score) MarshalJSON ¶
func (*Score) UnmarshalJSON ¶
type SearchOptions ¶
type SearchOptions struct {
// TopK limits the result count. Zero uses DefaultTopK.
TopK int `json:"top_k,omitempty"`
MinScore Score `json:"min_score,omitempty"`
Filter filter.Predicate `json:"-"`
}
SearchOptions owns the policies applied to a semantic search.
func (SearchOptions) MarshalJSON ¶
func (s SearchOptions) MarshalJSON() ([]byte, error)
func (SearchOptions) ResultLimit ¶
func (s SearchOptions) ResultLimit() int
ResultLimit returns the explicit TopK or DefaultTopK when it is omitted.
func (*SearchOptions) UnmarshalJSON ¶
func (s *SearchOptions) UnmarshalJSON(data []byte) error
func (SearchOptions) Validate ¶
func (s SearchOptions) Validate() error
type SearchRequest ¶
type SearchRequest struct {
Query string `json:"query,omitempty"`
Options SearchOptions `json:"options"`
}
SearchRequest describes one semantic search and owns its input validation.
func NewSearchRequest ¶
func NewSearchRequest(query string) (*SearchRequest, error)
func (SearchRequest) MarshalJSON ¶
func (s SearchRequest) MarshalJSON() ([]byte, error)
func (*SearchRequest) UnmarshalJSON ¶
func (s *SearchRequest) UnmarshalJSON(data []byte) error
func (*SearchRequest) Validate ¶
func (s *SearchRequest) Validate() error
type SearchResponse ¶
type SearchResponse struct {
Results []*SearchResult `json:"results"`
}
SearchResponse owns a complete ranked result set.
func NewSearchResponse ¶
func NewSearchResponse(results []*SearchResult) (*SearchResponse, error)
func (*SearchResponse) Documents ¶
func (s *SearchResponse) Documents() []*document.Document
func (*SearchResponse) First ¶
func (s *SearchResponse) First() *SearchResult
func (SearchResponse) MarshalJSON ¶
func (s SearchResponse) MarshalJSON() ([]byte, error)
func (*SearchResponse) UnmarshalJSON ¶
func (s *SearchResponse) UnmarshalJSON(data []byte) error
func (*SearchResponse) Validate ¶
func (s *SearchResponse) Validate() error
func (*SearchResponse) ValidateFor ¶
func (s *SearchResponse) ValidateFor(request *SearchRequest) error
type SearchResult ¶
type SearchResult struct {
Document *document.Document `json:"document"`
Score Score `json:"score"`
}
SearchResult relates a document to one search operation. Score is deliberately kept outside document.Document: relevance belongs to a query/result pair, not to the indexed content itself.
func NewSearchResult ¶
func NewSearchResult(matched *document.Document, score Score) (*SearchResult, error)
func (SearchResult) MarshalJSON ¶
func (s SearchResult) MarshalJSON() ([]byte, error)
func (*SearchResult) UnmarshalJSON ¶
func (s *SearchResult) UnmarshalJSON(data []byte) error
func (*SearchResult) Validate ¶
func (s *SearchResult) Validate() error
type Searcher ¶
type Searcher interface {
// Search returns a response honoring the score threshold, metadata filter,
// and result cap owned by [SearchRequest.Options].
Search(ctx context.Context, request *SearchRequest) (*SearchResponse, error)
}
Searcher pulls documents similar to a query out of a vector store. Results are ranked by similarity score in descending order.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
|
Package filter defines the stable metadata-filter expression vocabulary used by vector stores. |
|
Package inmemory provides an in-process vector store backed by a map and a configurable similarity function.
|
Package inmemory provides an in-process vector store backed by a map and a configurable similarity function. |
|
Package storetest contains provider-independent contract tests for vector-store implementations and their filter visitors.
|
Package storetest contains provider-independent contract tests for vector-store implementations and their filter visitors. |