README
¶
go-faiss
Go bindings for Faiss, a library for vector similarity search.
Install
First you will need to build and install Faiss:
git clone https://github.com/blevesearch/faiss.git
cd faiss
cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON .
make -C build
sudo make -C build install
On osX ARM64, the instructions needed to be slightly adjusted based on https://github.com/facebookresearch/faiss/issues/2111:
LDFLAGS="-L/opt/homebrew/opt/llvm/lib" CPPFLAGS="-I/opt/homebrew/opt/llvm/include" CXX=/opt/homebrew/opt/llvm/bin/clang++ CC=/opt/homebrew/opt/llvm/bin/clang cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON .
// set FAISS_ENABLE_PYTHON to OFF in CMakeLists.txt to ignore libpython dylib
make -C build
sudo make -C build install
Building will produce the dynamic library faiss_c.
You will need to install it in a place where your system will find it (e.g. /usr/local/lib on mac or /usr/lib on Linux).
You can do this with:
sudo cp build/c_api/libfaiss_c.so /usr/local/lib
Now you can install the Go module:
go get github.com/blevesearch/go-faiss
Usage
API documentation is available at https://pkg.go.dev/github.com/DataIntelligenceCrew/go-faiss. See the Faiss wiki for more information.
Examples can be found in the _example directory.
Documentation
¶
Overview ¶
Package faiss provides bindings to Faiss, a library for vector similarity search. More detailed documentation can be found at the Faiss wiki: https://github.com/facebookresearch/faiss/wiki.
Index ¶
- Constants
- func NormalizeVector(vector []float32) []float32
- func SetOMPThreads(n uint)
- func WriteIndex(idx Index, filename string) error
- func WriteIndexIntoBuffer(idx Index) ([]byte, error)
- type IDSelector
- type IDSelectorNot
- type Index
- type IndexFlat
- type IndexImpl
- func (idx *IndexImpl) AsFlat() *IndexFlat
- func (idx *IndexImpl) GetNProbe() int32
- func (idx *IndexImpl) GetSubIndex() (*IndexImpl, error)
- func (i *IndexImpl) MergeFrom(other Index, add_id int64) error
- func (idx *IndexImpl) SetDirectMap(mapType int) (err error)
- func (idx *IndexImpl) SetNProbe(nprobe int32)
- type ParameterSpace
- type RangeSearchResult
- type SearchParams
- type Selector
Constants ¶
const ( MetricInnerProduct = C.METRIC_INNER_PRODUCT MetricL2 = C.METRIC_L2 MetricL1 = C.METRIC_L1 MetricLinf = C.METRIC_Linf MetricLp = C.METRIC_Lp MetricCanberra = C.METRIC_Canberra MetricBrayCurtis = C.METRIC_BrayCurtis MetricJensenShannon = C.METRIC_JensenShannon )
Metric type
const ( IOFlagMmap = C.FAISS_IO_FLAG_MMAP IOFlagReadOnly = C.FAISS_IO_FLAG_READ_ONLY IOFlagReadMmap = C.FAISS_IO_FLAG_READ_MMAP | C.FAISS_IO_FLAG_ONDISK_IVF IOFlagSkipPrefetch = C.FAISS_IO_FLAG_SKIP_PREFETCH )
Variables ¶
This section is empty.
Functions ¶
func NormalizeVector ¶ added in v1.0.22
In-place normalization of provided vector (single)
func SetOMPThreads ¶ added in v1.0.17
func SetOMPThreads(n uint)
func WriteIndex ¶
WriteIndex writes an index to a file.
func WriteIndexIntoBuffer ¶ added in v1.0.0
Types ¶
type IDSelector ¶
type IDSelector struct {
// contains filtered or unexported fields
}
IDSelector represents a set of IDs to remove.
func (*IDSelector) Delete ¶
func (s *IDSelector) Delete()
Delete frees the memory associated with s.
func (*IDSelector) Get ¶ added in v1.0.23
func (s *IDSelector) Get() *C.FaissIDSelector
type IDSelectorNot ¶ added in v1.0.22
type IDSelectorNot struct {
// contains filtered or unexported fields
}
func (*IDSelectorNot) Delete ¶ added in v1.0.22
func (s *IDSelectorNot) Delete()
Delete frees the memory associated with s.
func (*IDSelectorNot) Get ¶ added in v1.0.23
func (s *IDSelectorNot) Get() *C.FaissIDSelector
type Index ¶
type Index interface {
// D returns the dimension of the indexed vectors.
D() int
// IsTrained returns true if the index has been trained or does not require
// training.
IsTrained() bool
// Ntotal returns the number of indexed vectors.
Ntotal() int64
// MetricType returns the metric type of the index.
MetricType() int
// Train trains the index on a representative set of vectors.
Train(x []float32) error
// Add adds vectors to the index.
Add(x []float32) error
// AddWithIDs is like Add, but stores xids instead of sequential IDs.
AddWithIDs(x []float32, xids []int64) error
// Returns true if the index is an IVF index.
IsIVFIndex() bool
// Applicable only to IVF indexes: Returns a map where the keys
// are cluster IDs and the values represent the count of input vectors that belong
// to each cluster.
// This method only considers the given vecIDs and does not account for all
// vectors in the index.
// Example:
// If vecIDs = [1, 2, 3, 4, 5], and:
// - Vectors 1 and 2 belong to cluster 1
// - Vectors 3, 4, and 5 belong to cluster 2
// The output will be: map[1:2, 2:3]
ObtainClusterVectorCountsFromIVFIndex(vecIDs []int64) (map[int64]int64, error)
// Applicable only to IVF indexes: Returns the centroid IDs in decreasing order
// of proximity to query 'x' and their distance from 'x'
ObtainClustersWithDistancesFromIVFIndex(x []float32, centroidIDs []int64) (
[]int64, []float32, error)
// Search queries the index with the vectors in x.
// Returns the IDs of the k nearest neighbors for each query vector and the
// corresponding distances.
Search(x []float32, k int64) (distances []float32, labels []int64, err error)
SearchWithoutIDs(x []float32, k int64, exclude []int64, params json.RawMessage) (distances []float32,
labels []int64, err error)
SearchWithIDs(x []float32, k int64, include []int64, params json.RawMessage) (distances []float32,
labels []int64, err error)
// Applicable only to IVF indexes: Search clusters whose IDs are in eligibleCentroidIDs
SearchClustersFromIVFIndex(selector Selector, eligibleCentroidIDs []int64,
minEligibleCentroids int, k int64, x, centroidDis []float32,
params json.RawMessage) ([]float32, []int64, error)
Reconstruct(key int64) ([]float32, error)
ReconstructBatch(keys []int64, recons []float32) ([]float32, error)
MergeFrom(other Index, add_id int64) error
// RangeSearch queries the index with the vectors in x.
// Returns all vectors with distance < radius.
RangeSearch(x []float32, radius float32) (*RangeSearchResult, error)
// Reset removes all vectors from the index.
Reset() error
// RemoveIDs removes the vectors specified by sel from the index.
// Returns the number of elements removed and error.
RemoveIDs(sel *IDSelector) (int, error)
// Close frees the memory used by the index.
Close()
// consults the C++ side to get the size of the index
Size() uint64
// contains filtered or unexported methods
}
Index is a Faiss index.
Note that some index implementations do not support all methods. Check the Faiss wiki to see what operations an index supports.
type IndexFlat ¶
type IndexFlat struct {
Index
}
IndexFlat is an index that stores the full vectors and performs exhaustive search.
func NewIndexFlat ¶
NewIndexFlat creates a new flat index.
func NewIndexFlatIP ¶
NewIndexFlatIP creates a new flat index with the inner product metric type.
func NewIndexFlatL2 ¶
NewIndexFlatL2 creates a new flat index with the L2 metric type.
type IndexImpl ¶
type IndexImpl struct {
Index
}
IndexImpl is an abstract structure for an index.
func IndexFactory ¶
IndexFactory builds a composite index. description is a comma-separated list of components.
func ReadIndexFromBuffer ¶ added in v1.0.0
func (*IndexImpl) AsFlat ¶
AsFlat casts idx to a flat index. AsFlat panics if idx is not a flat index.
func (*IndexImpl) GetSubIndex ¶ added in v1.0.2
func (*IndexImpl) SetDirectMap ¶ added in v1.0.0
type ParameterSpace ¶
type ParameterSpace struct {
// contains filtered or unexported fields
}
func NewParameterSpace ¶
func NewParameterSpace() (*ParameterSpace, error)
NewParameterSpace creates a new ParameterSpace.
func (*ParameterSpace) Delete ¶
func (p *ParameterSpace) Delete()
Delete frees the memory associated with p.
func (*ParameterSpace) SetIndexParameter ¶
func (p *ParameterSpace) SetIndexParameter(idx Index, name string, val float64) error
SetIndexParameter sets one of the parameters.
type RangeSearchResult ¶
type RangeSearchResult struct {
// contains filtered or unexported fields
}
RangeSearchResult is the result of a range search.
func (*RangeSearchResult) Delete ¶
func (r *RangeSearchResult) Delete()
Delete frees the memory associated with r.
func (*RangeSearchResult) Labels ¶
func (r *RangeSearchResult) Labels() (labels []int64, distances []float32)
Labels returns the unsorted IDs and respective distances for each query. The result for query i is labels[lims[i]:lims[i+1]].
func (*RangeSearchResult) Lims ¶
func (r *RangeSearchResult) Lims() []int
Lims returns a slice containing start and end indices for queries in the distances and labels slices returned by Labels.
func (*RangeSearchResult) Nq ¶
func (r *RangeSearchResult) Nq() int
Nq returns the number of queries.
type SearchParams ¶ added in v1.0.20
type SearchParams struct {
// contains filtered or unexported fields
}
func NewSearchParams ¶ added in v1.0.20
func NewSearchParams(idx Index, params json.RawMessage, sel *C.FaissIDSelector, defaultParams *defaultSearchParamsIVF) (*SearchParams, error)
Returns a valid SearchParams object, thus caller must clean up the object by invoking Delete() method.
func (*SearchParams) Delete ¶ added in v1.0.20
func (s *SearchParams) Delete()
Delete frees the memory associated with s.
type Selector ¶ added in v1.0.23
type Selector interface {
Get() *C.FaissIDSelector
Delete()
}
func NewIDSelectorBatch ¶
NewIDSelectorBatch creates a new batch selector.
func NewIDSelectorNot ¶ added in v1.0.5
NewIDSelectorNot creates a new Not selector, wrapped around a batch selector, with the IDs in 'exclude'.
func NewIDSelectorRange ¶
NewIDSelectorRange creates a selector that removes IDs on [imin, imax).
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
_example
|
|
|
flat
Usage example for IndexFlat.
|
Usage example for IndexFlat. |
|
ivfflat
Usage example for IndexIVFFlat.
|
Usage example for IndexIVFFlat. |
|
ivfpq
Usage example for IndexIVFPQ.
|
Usage example for IndexIVFPQ. |
|
misc
Usage example for IndexIVFFlat.
|
Usage example for IndexIVFFlat. |