Documentation
¶
Overview ¶
Package fasttext provides a simple wrapper for working with embedding vectors trained with Facebook's fastText library. It allows fast look-up of word embeddings from persistent data store (SQLite3).
Installation
go get -u github.com/justinfargnoli/go-fasttext
After training a model or downloading a '*.vec' data file from the fastText project, you can initialize the SQLite3 database (in your code):
import (
_ "github.com/mattn/go-sqlite3"
"github.com/justinfargnoli/go-fasttext"
)
...
ft := fasttext.New("/path/to/sqlite3/file")
err := ft.Build("/path/to/word/embedding/.vec/file")
This will create a new file on your disk for the SQLite3 database. Once the above step is finished, you can start looking up word embeddings (in your code):
embeddingVector, err := ft.GetEmb("king")
if err != nil {
fmt.Println(err)
}
fmt.Println(embeddingVector)
Each word's embedding vector is a slice of float64 with length of 300.
Note that you only need to initialize the SQLite3 database once. The next time you use it you can skip the call to BuildDB.
For faster querying during runtime, you can use an in-memory database.
ft := NewFastTextInMem("/path/to/sqlite3/file")
This creates an in-memory SQLite3 database which is a copy of the on-disk one. Using the in-memory version makes query time much faster, but takes a few minutes to load the database.
Index ¶
- Constants
- Variables
- type FastText
- func (ft *FastText) AllEmbeddingVectors() ([][]float64, error)
- func (ft *FastText) Build(wordEmbFile io.Reader) error
- func (ft *FastText) Close() error
- func (ft *FastText) EmbeddingVector(word string) ([]float64, error)
- func (ft *FastText) MostSimilarEmbeddingVector(queryEmbedding []float64) ([]float64, float64, error)
- func (ft *FastText) MultiWordEmbeddingVector(words []string) (vector.Vector, error)
Constants ¶
const ( // TableName used in SQLite3 TableName = "fasttext" // Dim is the number of dimensions in FastText word embedding vectors Dim = 300 )
Variables ¶
var ( // ErrNoEmbFound ... ErrNoEmbFound = errors.New("No embedding found for the given word") // ByteOrder is for the serialization of the embedding vector in // SQLite3 database. ByteOrder = binary.BigEndian )
Functions ¶
This section is empty.
Types ¶
type FastText ¶
type FastText struct {
// contains filtered or unexported fields
}
The FastText session. In multi-thread setting, each thread must have its own copy of FastText session. A single FastText session cannot be shared among multiple threads.
func NewInMemory ¶
NewInMemory creates a new FastText session that uses an in-memory database for faster query time. The on-disk SQLite3 database (given by dbFilename) will be loaded into an in-memory SQLite3 database in this function, which will take a few miniutes to finish.
func (*FastText) AllEmbeddingVectors ¶
AllEmbeddingVectors returns all embedding vectors
func (*FastText) Build ¶
Build initialize the SQLite3 database by importing the word embeddings from the .vec file downloaded from https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md
func (*FastText) EmbeddingVector ¶
EmbeddingVector returns the word embedding of the given word.
func (*FastText) MostSimilarEmbeddingVector ¶
func (ft *FastText) MostSimilarEmbeddingVector(queryEmbedding []float64) ([]float64, float64, error)
MostSimilarEmbeddingVector returns the embedding vector which is most similar to the one passed
Errors from FastText.GetAllEmb() and cosine_similarity.Cosine() will be propogated.
func (*FastText) MultiWordEmbeddingVector ¶
MultiWordEmbeddingVector builds an embedding vector to represent the array of words passed to it.
This is done by averaging the embedding vector of each string in the array.
If a string in words is not in the database, then an error indicator is returned.