vector

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 6, 2025 License: MIT Imports: 8 Imported by: 1

README

Upstash Vector Go Client

Go Reference

[!NOTE]
This project is in GA Stage.

The Upstash Professional Support fully covers this project. It receives regular updates, and bug fixes. The Upstash team is committed to maintaining and improving its functionality.

Upstash Vector is a serverless vector database designed for working with vector embeddings.

This is the HTTP-based Go client for Upstash Vector.

Documentation

Installation

Use go get to install the Upstash Vector package:

go get github.com/upstash/vector-go

Import the Upstash Vector package in your project:

import "github.com/upstash/vector-go"

Usage

In order to use this client, head out to Upstash Console and create a vector database.

Initializing the client

The REST token and REST URL configurations are required to initialize an Upstash Vector index client. Find your configuration values in the console dashboard at Upstash Console.

import (
	"github.com/upstash/vector-go"
)

func main() {
	index := vector.NewIndex("<UPSTASH_VECTOR_REST_URL>", "<UPSTASH_VECTOR_REST_TOKEN>")
}

Alternatively, you can set following environment variables:

UPSTASH_VECTOR_REST_URL="your_rest_url"
UPSTASH_VECTOR_REST_TOKEN="your_rest_token"

and then create index client by using:

import (
	"github.com/upstash/vector-go"
)

func main() {
	index := vector.NewIndexFromEnv()
}
Using a custom HTTP client

By default, http.DefaultClient will be used for doing requests. It is possible to use custom HTTP client, by passing it in the options while constructing the client.

import (
	"net/http"

	"github.com/upstash/vector-go"
)

func main() {
	opts := vector.Options{
		Url:    "<UPSTASH_VECTOR_REST_URL>",
		Token:  "<UPSTASH_VECTOR_REST_TOKEN>",
		Client: &http.Client{},
	}
	index := vector.NewIndexWith(opts)
}

Index operations

Upstash vector indexes support operations for working with vector data using operations such as upsert, query, fetch, and delete.

import (
	"github.com/upstash/vector-go"
)

func main() {
	index := vector.NewIndex("<UPSTASH_VECTOR_REST_URL>", "<UPSTASH_VECTOR_REST_TOKEN>")
}

Upstash Vector allows you to partition a single index into multiple isolated namespaces.

You can specify a namespace for an index client with Namespace(ns string) function. When you create a Namespace client, all index operations executed through this client become associated with the specified namespace.

By default, the Index client is associated with the default namespace.

import (
	"github.com/upstash/vector-go"
)

func main() {
	index := vector.NewIndex("<UPSTASH_VECTOR_REST_URL>", "<UPSTASH_VECTOR_REST_TOKEN>")
	
	// Returns a new Namespace client associated with the given namespace
	ns := index.Namespace("<NAMESPACE>")
Upserting Vectors

Upsert can be used to insert new vectors into index or to update existing vectors.

Upsert Many

Dense Indexes

err := index.UpsertMany([]vector.Upsert{
	{
		Id:     "0",
		Vector: []float32{0.6, 0.8},
	},
	{
		Id:       "1",
		Vector:   []float32{0.0, 1.0},
		Metadata: map[string]any{"foo": "bar"}, // optional metadata
		Data:     "vector data",                // optional data
	},
})

Sparse Indexes

err := index.UpsertMany([]vector.Upsert{
	{
		Id: "0",
		SparseVector: &vector.SparseVector{
			Indices: []int32{0, 1},
			Values:  []float32{0.5, 0.6},
		},
	},
	{
		Id: "1",
		SparseVector: &vector.SparseVector{
			Indices: []int32{5},
			Values:  []float32{0.1},
		},
		Metadata: map[string]any{"foo": "bar"}, // optional metadata
		Data:     "vector data",                // optional data
	},
})

Hybrid Indexes

err = index.UpsertMany([]vector.Upsert{
	{
		Id:     "0",
		Vector: []float32{0.6, 0.8},
		SparseVector: &vector.SparseVector{
			Indices: []int32{0, 1},
			Values:  []float32{0.5, 0.6},
		},
	},
	{
		Id:     "1",
		Vector: []float32{0.0, 1.0},
		SparseVector: &vector.SparseVector{
			Indices: []int32{5},
			Values:  []float32{0.1},
		},
		Metadata: map[string]any{"foo": "bar"}, // optional metadata
		Data:     "vector data",                // optional data
	},
})
Upsert One

Dense Indexes

err := index.Upsert(vector.Upsert{
	Id:     "2",
	Vector: []float32{1.0, 0.0},
})

Sparse Indexes

err = index.Upsert(vector.Upsert{
	Id: "2",
	SparseVector: &vector.SparseVector{
		Indices: []int32{0, 1},
		Values:  []float32{0.5, 0.6},
	},
})

Hybrid Indexes

err = index.Upsert(vector.Upsert{
	Id:     "2",
	Vector: []float32{1.0, 0.0},
	SparseVector: &vector.SparseVector{
		Indices: []int32{0, 1},
		Values:  []float32{0.5, 0.6},
	},
})
Upserting with Raw Data

If the vector index is created with an embedding model, it can be populated using the raw data without explicitly converting it to an embedding. Upstash server will create the embedding and index the generated vectors.

Upsert can be used to insert new vectors into index or to update existing vectors.

Upsert Many
err := index.UpsertDataMany([]vector.UpsertData{
	{
		Id:   "0",
		Data: "Capital of Turkey is Ankara.",
	},
	{
		Id:       "1",
		Data:     "Capital of Japan is Tokyo.",
		Metadata: map[string]any{"foo": "bar"}, // optional metadata
	},
})
Upsert One
err := index.UpsertData(vector.UpsertData{
	Id:   "2",
	Data: "Capital of Turkey is Ankara.",
})
Querying Vectors

When TopK is specified, at most that many vectors will be returned.

When IncludeVectors is true, the response will contain the vector values.

When IncludeMetadata is true, the response will contain the metadata of the vectors, if any.

When IncludeData is true, the response will contain the data of the vectors, if any.

Dense Indexes

scores, err := index.Query(vector.Query{
	Vector:          []float32{0.0, 1.0},
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})

Sparse Indexes

scores, err := index.Query(vector.Query{
	SparseVector: &vector.SparseVector{
		Indices: []int32{0, 1},
		Values:  []float32{0.5, 0.5},
	},
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})

Hybrid Indexes

scores, err := index.Query(vector.Query{
	Vector: []float32{0.0, 1.0},
	SparseVector: &vector.SparseVector{
		Indices: []int32{0, 1},
		Values:  []float32{0.5, 0.5},
	},
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})

Additionally, a metadata filter can be specified in queries. When Filter is given, the response will contain only the values whose metadata matches the given filter. See Metadata Filtering docs for more information.

scores, err := index.Query(vector.Query{
	..., 
	Filter: `foo = 'bar'`
})
Querying with Raw Data

If the vector index is created with an embedding model, a query can be executed using the raw data without explicitly converting it to an embedding. Upstash server will create the embedding and run the query.

When TopK is specified, at most that many vectors will be returned.

When IncludeVectors is true, the response will contain the vector values.

When IncludeMetadata is true, the response will contain the metadata of the vectors, if any.

When IncludeData is true, the response will contain the data of the vectors, if any.

scores, err := index.QueryData(vector.QueryData{
	Data:            "Where is the capital of Turkey?",
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
	Filter:          `foo = 'bar'`,
})
Resumable Querying Vectors

With a similar interface to query and query data, query results can be fetched page by page with resumable queries.

Resumalbe Query

When a resumable query is started, it returns the first page of the query results, and returns a handle that can be used to fetch next pages. When enough query results are fetched, handle can be closed to release the resources acquired in the index to facilitate the resumable query.

Dense Indexes

scores, handle, err := index.ResumableQuery(vector.ResumableQuery{
	Vector:          []float32{0.0, 1.0},
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})
defer handle.Close()

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 3,
})

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 5,
})

Sparse Indexes

scores, handle, err := index.ResumableQuery(vector.ResumableQuery{
	SparseVector: &vector.SparseVector{
		Indices: []int32{0, 1},
		Values:  []float32{0.5, 0.5},
	},
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})
defer handle.Close()

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 3,
})

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 5,
})

Hybrid Indexes

scores, handle, err := index.ResumableQuery(vector.ResumableQuery{
	Vector: []float32{0.0, 1.0},
	SparseVector: &vector.SparseVector{
		Indices: []int32{0, 1},
		Values:  []float32{0.5, 0.5},
	},
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})
defer handle.Close()

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 3,
})

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 5,
})
Resumable Query with Data

If the vector index is created with an embedding model, a resumable query can be started using the raw data without explicitly converting it to an embedding. Upstash server will create the embedding and start the query.

scores, handle, err := index.ResumableQueryData(vector.ResumableQueryData{
	Data:            "Where is the capital of Turkey?",
	TopK:            2,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})
defer handle.Close()

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 3,
})

scores, err = handle.Next(vector.ResumableQueryNext{
	AdditionalK: 5,
})
Fetching Vectors

Vectors can be fetched individually by providing the unique vector ids.

When IncludeVectors is true, the response will contain the vector values.

When IncludeMetadata is true, the response will contain the metadata of the vectors, if any.

When IncludeData is true, the response will contain the data of the vectors, if any.

vectors, err := index.Fetch(vector.Fetch{
	Ids:             []string{"0", "1"},
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})
Deleting Vectors

Vectors can be deleted from the index.

Delete many
count, err := index.DeleteMany([]string{"0", "999"})
Delete One
ok, err := index.Delete("2")
Scanning Vectors

All or some of the vectors in the index can scanned by fetching range of vectors.

While starting the scan, the initial cursor value of "0" should be used.

When IncludeVectors is true, the response will contain the vector values.

When IncludeMetadata is true, the response will contain the metadata of the vectors, if any.

When IncludeData is true, the response will contain the data of the vectors, if any.

vectors, err := index.Range(vector.Range{
	Cursor:          "0",
	Limit:           10,
	IncludeVectors:  false,
	IncludeMetadata: false,
	IncludeData:     false,
})

for vectors.NextCursor != "" {
	for _, v := range vectors.Vectors {
		// process individual vectors
	}

	// Fetch the next range batch
	vectors, err = index.Range(vector.Range{
		Cursor:          vectors.NextCursor,
		Limit:           10,
		IncludeVectors:  false,
		IncludeMetadata: false,
		IncludeData:     false,
	})
}
Updating Vectors

Any combination of vector value, sparse vector value, data, or metadata can be updated.

ok, err := index.Update(vector.Update{
	Id:       "id",
	Metadata: map[string]any{"new": "metadata"},
})
Resetting the Index

Reset will delete all the vectors and reset the index to its initial state.

err := index.Reset()
Getting Index Information
info, err := index.Info()
List Namespaces

All the names of active namespaces can be listed.

namespaces, err := index.ListNamespaces()
for _, ns : range namespaces {
	fmt.Println(ns)
}
Delete Namespaces

A namespace can be deleted entirely if it exists. The default namespaces cannot be deleted.

err := index.Namespace("ns").DeleteNamespace()

Documentation

Index

Constants

View Source 
const (
	UrlEnvProperty   = "UPSTASH_VECTOR_REST_URL"
	TokenEnvProperty = "UPSTASH_VECTOR_REST_TOKEN"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Fetch 

type Fetch struct {
	// Unique vectors ids to fetch.
	Ids []string `json:"ids"`

	// Whether to include vector values in the fetch response.
	IncludeVectors bool `json:"includeVectors,omitempty"`

	// Whether to include metadata in the fetch response, if any.
	IncludeMetadata bool `json:"includeMetadata,omitempty"`

	// Whether to include data in the query response, if any.
	IncludeData bool `json:"includeData,omitempty"`
}

type FusionAlgorithm added in v0.7.0 

type FusionAlgorithm string

FusionAlgorithm specifies the algorithm to use while fusing scores from dense and sparse components of a hybrid index. 

const (
	// FusionAlgorithmRRF is reciprocal rank fusion.
	//
	// Each sorted score from the dense and sparse indexes are
	// mapped to 1 / (rank + K), where rank is the order of the
	// score in the dense or sparse scores and K is a constant
	// with the value of 60.
	//
	// Then, scores from the dense and sparse components are
	// deduplicated (i.e. if a score for the same vector is present
	// in both dense and sparse scores, the mapped scores are
	// added; otherwise individual mapped scores are used)
	// and the final result is returned as the topK values
	// of this final list.
	//
	// In short, this algorithm just takes the order of the scores
	// into consideration.
	FusionAlgorithmRRF FusionAlgorithm = "RRF"

	// FusionAlgorithmDBSF is distribution based score fusion.
	//
	// Each sorted score from the dense and sparse indexes are
	// normalized as
	// (s - (mean - 3 * stddev)) / ((mean + 3 * stddev) - (mean - 3 * stddev))
	// where s is the score, (mean - 3 * stddev) is the minimum,
	// and (mean + 3 * stddev) is the maximum tail ends of the distribution.
	//
	// Then, scores from the dense and sparse components are
	// deduplicated (i.e. if a score for the same vector is present
	// in both dense and sparse scores, the normalized scores are
	// added; otherwise individual normalized scores are used)
	// and the final result is returned as the topK values
	// of this final list.
	//
	// In short, this algorithm takes distribution of the scores
	// into consideration as well, as opposed to the RRF.
	FusionAlgorithmDBSF FusionAlgorithm = "DBSF"
)

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index is a client for Upstash Vector index.

func NewIndex 

func NewIndex(url string, token string) *Index

NewIndex returns an index client to be used with Upstash Vector with the given url and token.

func NewIndexFromEnv 

func NewIndexFromEnv() *Index

NewIndexFromEnv returns an index client to be used with Upstash Vector by reading URL and token from the environment variables.

func NewIndexWith 

func NewIndexWith(options Options) *Index

NewIndexWith returns an index client to be used with Upstash Vector with the given options.

func (*Index) Delete 

func (ix *Index) Delete(id string) (ok bool, err error)

Delete deletes the vector with the given id in the default namespace and reports whether the vector is deleted. If a vector with the given id is not found, Delete returns false.

func (*Index) DeleteMany 

func (ix *Index) DeleteMany(ids []string) (count int, err error)

DeleteMany deletes the vectors with the given ids in the default namespace and reports how many of them are deleted.

func (*Index) Fetch 

func (ix *Index) Fetch(f Fetch) (vectors []Vector, err error)

Fetch fetches one or more vectors in the default namespace with the ids passed into f. If IncludeVectors is set to true, the vector values are also returned. If IncludeMetadata is set to true, any associated metadata of the vectors is also returned, if any.

func (*Index) Info 

func (ix *Index) Info() (info IndexInfo, err error)

Info returns some information about the index, including:

  • Total number of vectors across all namespaces
  • Total number of vectors waiting to be indexed across all namespaces
  • Total size of the index on disk in bytes
  • Vector dimension
  • Similarity function used
  • per-namespace vector and pending vector counts

func (*Index) ListNamespaces added in v0.5.0 

func (ix *Index) ListNamespaces() (namespaces []string, err error)

ListNamespaces returns the list of names of namespaces.

func (*Index) Namespace added in v0.5.0 

func (ix *Index) Namespace(namespace string) (i *Namespace)

Namespace returns a new client associated with the given namespace.

func (*Index) Query 

func (ix *Index) Query(q Query) (scores []VectorScore, err error)

Query returns the result of the query for the given vector in the default namespace. When q.TopK is specified, the result will contain at most q.TopK many vectors. The returned list will contain vectors sorted in descending order of score, which correlates with the similarity of the vectors to the given query vector. When q.IncludeVectors is true, values of the vectors are also returned. When q.IncludeMetadata is true, metadata of the vectors are also returned, if any.

func (*Index) QueryData added in v0.4.0 

func (ix *Index) QueryData(q QueryData) (scores []VectorScore, err error)

QueryData returns the result of the query for the given data by converting it to an embedding on the server. When q.TopK is specified, the result will contain at most q.TopK many vectors. The returned list will contain vectors sorted in descending order of score, which correlates with the similarity of the vectors to the given query vector. When q.IncludeVectors is true, values of the vectors are also returned. When q.IncludeMetadata is true, metadata of the vectors are also returned, if any.

func (*Index) Range 

func (ix *Index) Range(r Range) (vectors RangeVectors, err error)

Range returns a range of vectors, starting with r.Cursor (inclusive), until the end of the vectors in the index or until the given q.Limit. The initial cursor should be set to "0", and subsequent calls to Range might use the next cursor returned in the response. When r.IncludeVectors is true, values of the vectors are also returned. When r.IncludeMetadata is true, metadata of the vectors are also returned, if any.

func (*Index) Reset 

func (ix *Index) Reset() (err error)

Reset deletes all the vectors in the default namespace of the index and resets it to initial state.

func (*Index) ResumableQuery added in v0.7.0 

func (ix *Index) ResumableQuery(q ResumableQuery) (scores []VectorScore, handle *ResumableQueryHandle, err error)

ResumableQuery starts a resumable query and returns the first page of the result of the query for the given vector in the default namespace. Then, next pages of the query results can be fetched over the returned handle. After all the needed pages of the results are fetched, it is recommended to close to handle to release the acquired resources.

func (*Index) ResumableQueryData added in v0.7.0 

func (ix *Index) ResumableQueryData(q ResumableQueryData) (scores []VectorScore, handle *ResumableQueryHandle, err error)

ResumableQueryData starts a resumable query and returns the first page of the result of the query for the given text data in the default namespace. Then, next pages of the query results can be fetched over the returned handle. After all the needed pages of the results are fetched, it is recommended to close to handle to release the acquired resources.

func (*Index) Update added in v0.7.0 

func (ix *Index) Update(u Update) (ok bool, err error)

Update updates a vector value, data, or metadata for the given id for the default namespace of the index and reports whether the vector is updated. If a vector with the given id is not found, Update returns false.

func (*Index) Upsert 

func (ix *Index) Upsert(u Upsert) (err error)

Upsert updates or inserts a vector to the default namespace of the index. Additional metadata can also be provided while upserting the vector.

func (*Index) UpsertData added in v0.4.0 

func (ix *Index) UpsertData(u UpsertData) (err error)

UpsertData updates or inserts a vector to the default namespace of the index by converting given raw data to an embedding on the server. Additional metadata can also be provided while upserting the vector.

func (*Index) UpsertDataMany added in v0.4.0 

func (ix *Index) UpsertDataMany(u []UpsertData) (err error)

UpsertDataMany updates or inserts some vectors to the default namespace of the index by converting given raw data to an embedding on the server. Additional metadata can also be provided for each vector.

func (*Index) UpsertMany 

func (ix *Index) UpsertMany(u []Upsert) (err error)

UpsertMany updates or inserts some vectors to the default namespace of the index. Additional metadata can also be provided for each vector.

type IndexInfo 

type IndexInfo struct {
	// The number of vectors in the index.
	VectorCount int `json:"vectorCount"`

	// The number of vectors that are pending to be indexed.
	PendingVectorCount int `json:"pendingVectorCount"`

	// The size of the index on disk in bytes
	IndexSize int `json:"indexSize"`

	// The dimension of the vectors.
	Dimension int `json:"dimension"`

	// Name of the similarity function used in indexing and queries.
	SimilarityFunction string `json:"similarityFunction"`

	// Per-namespace vector and pending vector counts
	Namespaces map[string]NamespaceInfo `json:"namespaces"`
}

type MetadataUpdateMode added in v0.7.0 

type MetadataUpdateMode string

MetadataUpdateMode specifies whether to overwrite the whole metadata while updating it, or patch the metadata (insert new fields or update or delete existing fields) according to the RFC 7396 JSON Merge Patch algorithm. 

const (
	// MetadataUpdateModeOverwrite overwrites the metadata,
	// and set it to a new value.
	MetadataUpdateModeOverwrite MetadataUpdateMode = "OVERWRITE"

	// MetadataUpdateModePatch patches the metadata according
	// to the JSON Merge Patch algorithm.
	MetadataUpdateModePatch MetadataUpdateMode = "PATCH"
)

type Namespace added in v0.5.0 

type Namespace struct {
	// contains filtered or unexported fields
}

func (*Namespace) Delete added in v0.5.0 

func (ns *Namespace) Delete(id string) (ok bool, err error)

Delete deletes the vector with the given id in the namespace and reports whether the vector is deleted. If a vector with the given id is not found, Delete returns false.

func (*Namespace) DeleteMany added in v0.5.0 

func (ns *Namespace) DeleteMany(ids []string) (count int, err error)

DeleteMany deletes the vectors with the given ids in the namespace and reports how many of them are deleted.

func (*Namespace) DeleteNamespace added in v0.5.0 

func (ns *Namespace) DeleteNamespace() error

DeleteNamespace deletes the given namespace of index if it exists.

func (*Namespace) Fetch added in v0.5.0 

func (ns *Namespace) Fetch(f Fetch) (vectors []Vector, err error)

Fetch fetches one or more vectors in the namespace with the ids passed into f. If IncludeVectors is set to true, the vector values are also returned. If IncludeMetadata is set to true, any associated metadata of the vectors is also returned, if any.

func (*Namespace) Query added in v0.5.0 

func (ns *Namespace) Query(q Query) (scores []VectorScore, err error)

Query returns the result of the query for the given vector in the namespace. When q.TopK is specified, the result will contain at most q.TopK many vectors. The returned list will contain vectors sorted in descending order of score, which correlates with the similarity of the vectors to the given query vector. When q.IncludeVectors is true, values of the vectors are also returned. When q.IncludeMetadata is true, metadata of the vectors are also returned, if any.

func (*Namespace) QueryData added in v0.5.0 

func (ns *Namespace) QueryData(q QueryData) (scores []VectorScore, err error)

QueryData returns the result of the query for the given data by converting it to an embedding on the server. When q.TopK is specified, the result will contain at most q.TopK many vectors. The returned list will contain vectors sorted in descending order of score, which correlates with the similarity of the vectors to the given query vector. When q.IncludeVectors is true, values of the vectors are also returned. When q.IncludeMetadata is true, metadata of the vectors are also returned, if any.

func (*Namespace) Range added in v0.5.0 

func (ns *Namespace) Range(r Range) (vectors RangeVectors, err error)

Range returns a range of vectors, starting with r.Cursor (inclusive), until the end of the vectors in the index or until the given q.Limit. The initial cursor should be set to "0", and subsequent calls to Range might use the next cursor returned in the response. When r.IncludeVectors is true, values of the vectors are also returned. When r.IncludeMetadata is true, metadata of the vectors are also returned, if any.

func (*Namespace) Reset added in v0.5.0 

func (ns *Namespace) Reset() (err error)

Reset deletes all the vectors in the namespace of the index and resets it to initial state.

func (*Namespace) ResumableQuery added in v0.7.0 

func (ns *Namespace) ResumableQuery(q ResumableQuery) (scores []VectorScore, handle *ResumableQueryHandle, err error)

ResumableQuery starts a resumable query and returns the first page of the result of the query for the given vector in the default namespace. Then, next pages of the query results can be fetched over the returned handle. After all the needed pages of the results are fetched, it is recommended to close to handle to release the acquired resources.

func (*Namespace) ResumableQueryData added in v0.7.0 

func (ns *Namespace) ResumableQueryData(q ResumableQueryData) (scores []VectorScore, handle *ResumableQueryHandle, err error)

ResumableQueryData starts a resumable query and returns the first page of the result of the query for the given text data in the default namespace. Then, next pages of the query results can be fetched over the returned handle. After all the needed pages of the results are fetched, it is recommended to close to handle to release the acquired resources.

func (*Namespace) Update added in v0.7.0 

func (ns *Namespace) Update(u Update) (ok bool, err error)

Update updates a vector value, data, or metadata for the given id in the namespace and reports whether the vector is updated. If a vector with the given id is not found, Update returns false.

func (*Namespace) Upsert added in v0.5.0 

func (ns *Namespace) Upsert(u Upsert) (err error)

Upsert updates or inserts a vector to the namespace of the index. Additional metadata can also be provided while upserting the vector.

func (*Namespace) UpsertData added in v0.5.0 

func (ns *Namespace) UpsertData(u UpsertData) (err error)

UpsertData updates or inserts a vector to the namespace of the index by converting given raw data to an embedding on the server. Additional metadata can also be provided while upserting the vector.

func (*Namespace) UpsertDataMany added in v0.5.0 

func (ns *Namespace) UpsertDataMany(u []UpsertData) (err error)

UpsertDataMany updates or inserts some vectors to the default namespace of the index by converting given raw data to an embedding on the server. Additional metadata can also be provided for each vector.

func (*Namespace) UpsertMany added in v0.5.0 

func (ns *Namespace) UpsertMany(u []Upsert) (err error)

UpsertMany updates or inserts some vectors to the default namespace of the index. Additional metadata can also be provided for each vector.

type NamespaceInfo added in v0.5.0 

type NamespaceInfo struct {
	// The number of vectors in the namespace of the index.
	VectorCount int `json:"vectorCount"`

	// The number of vectors that are pending to be indexed.
	PendingVectorCount int `json:"pendingVectorCount"`
}

type Options 

type Options struct {
	// URL of the Upstash Vector index.
	Url string

	// Token of the Upstash Vector index.
	Token string

	// The HTTP client to use for requests.
	Client *http.Client
}

type Query 

type Query struct {
	// The dense query vector for dense and hybrid indexes.
	Vector []float32 `json:"vector,omitempty"`

	// The sparse query vector for sparse and hybrid indexes.
	SparseVector *SparseVector `json:"sparseVector,omitempty"`

	// The maximum number of vectors that will
	// be returned for the query response.
	TopK int `json:"topK,omitempty"`

	// Whether to include vector values in the query response.
	IncludeVectors bool `json:"includeVectors,omitempty"`

	// Whether to include metadata in the query response, if any.
	IncludeMetadata bool `json:"includeMetadata,omitempty"`

	// Whether to include data in the query response, if any.
	IncludeData bool `json:"includeData,omitempty"`

	// Query filter
	Filter any `json:"filter,omitempty"`

	// Weighting strategy to be used for sparse vectors.
	// If not provided, no weighting will be used.
	WeightingStrategy WeightingStrategy `json:"weightingStrategy,omitempty"`

	// Fusion algorithm to use while fusing scores
	// from dense and sparse components of a hybrid index.
	// If not provided, defaults to RRF.
	FusionAlgorithm FusionAlgorithm `json:"fusionAlgorithm,omitempty"`
}

type QueryData added in v0.4.0 

type QueryData struct {
	// Raw data.
	// Data will be converted to the vector embedding on the server.
	Data string `json:"data"`

	// The maximum number of vectors that will
	// be returned for the query response.
	TopK int `json:"topK,omitempty"`

	// Whether to include vector values in the query response.
	IncludeVectors bool `json:"includeVectors,omitempty"`

	// Whether to include metadata in the query response, if any.
	IncludeMetadata bool `json:"includeMetadata,omitempty"`

	// Whether to include data in the query response, if any.
	IncludeData bool `json:"includeData,omitempty"`

	// Query filter
	Filter any `json:"filter,omitempty"`

	// Weighting strategy to be used for sparse vectors.
	// If not provided, no weighting will be used.
	WeightingStrategy WeightingStrategy `json:"weightingStrategy,omitempty"`

	// Fusion algorithm to use while fusing scores
	// from dense and sparse components of a hybrid index.
	// If not provided, defaults to RRF.
	FusionAlgorithm FusionAlgorithm `json:"fusionAlgorithm,omitempty"`

	// Specifies whether to run the query in only the
	// dense index, only the sparse index, or in both for hybrid
	// indexes with Upstash-hosted embedding models.
	// If not provided, defaults to hybrid query mode.
	QueryMode QueryMode `json:"queryMode,omitempty"`
}

type QueryMode added in v0.7.0 

type QueryMode string

QueryMode for hybrid indexes with Upstash-hosted embedding models.

It specifies whether to run the query in only the dense index, only the sparse index, or in both. 

const (
	// QueryModeHybrid runs the query in hybrid index mode,
	// after embedding the raw text data into dense and sparse vectors.
	//
	// Query results from the dense and sparse index components
	// of the hybrid index are fused before returning the result.
	QueryModeHybrid QueryMode = "HYBRID"

	// QueryModeDense runs the query in dense index mode,
	// after embedding the raw text data into a dense vector.
	//
	// Only the query results from the dense index component
	// of the hybrid index is returned.
	QueryModeDense QueryMode = "DENSE"

	// QueryModeSparse runs the query in sparse index mode,
	// after embedding the raw text data into a sparse vector.
	//
	// Only the query results from the sparse index component
	// of the hybrid index is returned.
	QueryModeSparse QueryMode = "SPARSE"
)

type Range 

type Range struct {
	// The cursor to start returning range from (inclusive).
	Cursor string `json:"cursor,omitempty"`

	// The maximum number of vectors that will be returned for
	// the range response.
	Limit int `json:"limit,omitempty"`

	// Whether to include vector values in the range response.
	IncludeVectors bool `json:"includeVectors,omitempty"`

	// Whether to include metadata in the fetch response, if any.
	IncludeMetadata bool `json:"includeMetadata,omitempty"`

	// Whether to include data in the query response, if any.
	IncludeData bool `json:"includeData,omitempty"`
}

type RangeVectors 

type RangeVectors struct {
	// The cursor that should be used for the subsequent range requests.
	NextCursor string `json:"nextCursor"`

	// List of vectors in the range.
	Vectors []Vector `json:"vectors,omitempty"`
}

type ResumableQuery added in v0.7.0 

type ResumableQuery struct {
	// The dense query vector for dense and hybrid indexes.
	Vector []float32 `json:"vector,omitempty"`

	// The sparse query vector for sparse and hybrid indexes.
	SparseVector *SparseVector `json:"sparseVector,omitempty"`

	// The maximum number of vectors that will
	// be returned for the query response.
	TopK int `json:"topK,omitempty"`

	// Whether to include vector values in the query response.
	IncludeVectors bool `json:"includeVectors,omitempty"`

	// Whether to include metadata in the query response, if any.
	IncludeMetadata bool `json:"includeMetadata,omitempty"`

	// Whether to include data in the query response, if any.
	IncludeData bool `json:"includeData,omitempty"`

	// Query filter
	Filter any `json:"filter,omitempty"`

	// Weighting strategy to be used for sparse vectors.
	// If not provided, no weighting will be used.
	WeightingStrategy WeightingStrategy `json:"weightingStrategy,omitempty"`

	// Fusion algorithm to use while fusing scores
	// from dense and sparse components of a hybrid index.
	// If not provided, defaults to RRF.
	FusionAlgorithm FusionAlgorithm `json:"fusionAlgorithm,omitempty"`

	// Maximum idle time for the resumable query in seconds.
	// If not provided, defaults to 1 hour.
	MaxIdle uint32 `json:"maxIdle,omitempty"`
}

type ResumableQueryData added in v0.7.0 

type ResumableQueryData struct {
	// Raw data.
	// Data will be converted to the vector embedding on the server.
	Data string `json:"data"`

	// The maximum number of vectors that will
	// be returned for the query response.
	TopK int `json:"topK,omitempty"`

	// Whether to include vector values in the query response.
	IncludeVectors bool `json:"includeVectors,omitempty"`

	// Whether to include metadata in the query response, if any.
	IncludeMetadata bool `json:"includeMetadata,omitempty"`

	// Whether to include data in the query response, if any.
	IncludeData bool `json:"includeData,omitempty"`

	// Query filter
	Filter any `json:"filter,omitempty"`

	// Weighting strategy to be used for sparse vectors.
	// If not provided, no weighting will be used.
	WeightingStrategy WeightingStrategy `json:"weightingStrategy,omitempty"`

	// Fusion algorithm to use while fusing scores
	// from dense and sparse components of a hybrid index.
	// If not provided, defaults to RRF.
	FusionAlgorithm FusionAlgorithm `json:"fusionAlgorithm,omitempty"`

	// Specifies whether to run the query in only the
	// dense index, only the sparse index, or in both for hybrid
	// indexes with Upstash-hosted embedding models.
	// If not provided, defaults to hybrid query mode.
	QueryMode QueryMode `json:"queryMode,omitempty"`

	// Maximum idle time for the resumable query in seconds.
	// If not provided, defaults to 1 hour.
	MaxIdle uint32 `json:"maxIdle,omitempty"`
}

type ResumableQueryHandle added in v0.7.0 

type ResumableQueryHandle struct {
	// contains filtered or unexported fields
}

func (*ResumableQueryHandle) Close added in v0.7.0 

func (h *ResumableQueryHandle) Close() (err error)

Close stops the resumable query and releases the acquired resources.

func (*ResumableQueryHandle) Next added in v0.7.0 

func (h *ResumableQueryHandle) Next(n ResumableQueryNext) (scores []VectorScore, err error)

Next fetches the next page of the query result.

type ResumableQueryNext added in v0.7.0 

type ResumableQueryNext struct {
	AdditionalK int `json:"additionalK,omitempty"`
}

type SparseVector added in v0.7.0 

type SparseVector struct {
	// List of dimensions that have non-zero values.
	Indices []int32 `json:"indices"`

	// Values of the non-zero dimensions.
	// It must be of the same size as the indices.
	Values []float32 `json:"values"`
}

type Update added in v0.7.0 

type Update struct {
	// The id of the vector to update.
	Id string `json:"id"`

	// The new dense vector values for dense and hybrid indexes.
	Vector []float32 `json:"vector,omitempty"`

	// The new sparse vector values for sparse and hybrid indexes.
	SparseVector *SparseVector `json:"sparseVector,omitempty"`

	// The new data of the vector.
	Data string `json:"data,omitempty"`

	// The new metadata of the vector.
	Metadata map[string]any `json:"metadata,omitempty"`

	// Whether to overwrite the whole metadata while updating it,
	// or patch the metadata according to the JSON Merge Patch algorithm.
	// If not provided, defaults to overwrite.
	MetadataUpdateMode MetadataUpdateMode `json:"metadataUpdateMode,omitempty"`
}

type Upsert 

type Upsert struct {
	// Unique id of the vector.
	Id string `json:"id"`

	// Dense vector values for dense and hybrid indexes.
	Vector []float32 `json:"vector,omitempty"`

	// Sparse vector values for sparse and hybrid indexes.
	SparseVector *SparseVector `json:"sparseVector,omitempty"`

	// Optional data of the vector.
	Data string `json:"data,omitempty"`

	// Optional metadata of the vector.
	Metadata map[string]any `json:"metadata,omitempty"`
}

type UpsertData added in v0.4.0 

type UpsertData struct {
	// Unique id of the vector.
	Id string `json:"id"`

	// Raw data.
	// Data will be converted to the vector embedding on the server.
	Data string `json:"data"`

	// Optional metadata of the vector.
	Metadata map[string]any `json:"metadata,omitempty"`
}

type Vector 

type Vector struct {
	// Unique id of the vector.
	Id string `json:"id"`

	// Dense vector values for dense and hybrid indexes.
	Vector []float32 `json:"vector,omitempty"`

	// Sparse vector values for sparse and hybrid indexes.
	SparseVector *SparseVector `json:"sparseVector,omitempty"`

	// Optional metadata of the vector, if any.
	Metadata map[string]any `json:"metadata,omitempty"`

	// Optional data of the vector.
	Data string `json:"data,omitempty"`
}

type VectorScore 

type VectorScore struct {
	// Unique id of the vector.
	Id string `json:"id"`

	// Similarity score of the vector to the query vector.
	// Vectors more similar to the query vector have higher score.
	Score float32 `json:"score"`

	// Optional dense vector values for dense and hybrid indexes.
	Vector []float32 `json:"vector,omitempty"`

	// Optional sparse vector values for sparse and hybrid indexes.
	SparseVector *SparseVector `json:"sparseVector,omitempty"`

	// Optional metadata of the vector, if any.
	Metadata map[string]any `json:"metadata,omitempty"`

	// Optional data of the vector.
	Data string `json:"data,omitempty"`
}

type WeightingStrategy added in v0.7.0 

type WeightingStrategy string

WeightingStrategy specifies what kind of weighting strategy should be used while querying the matching non-zero dimension values of the query vector with the indexed vectors for sparse vectors. 

const (
	// WeightingStrategyIDF uses inverse document frequencies.
	//
	// It is recommended to use this weighting strategy for
	// BM25 sparse embedding models.
	//
	// It is calculated as
	// ln(((N - n(q) + 0.5) / (n(q) + 0.5)) + 1) where
	// N:    Total number of sparse vectors.
	// n(q): Total number of sparse vectors having non-zero value
	// for that particular dimension.
	// ln:   Natural logarithm
	// The values of N and n(q) are maintained by Upstash as the
	// vectors are indexed.
	WeightingStrategyIDF WeightingStrategy = "IDF"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.