fasttext

package module
v0.0.0-...-a12b86a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 15, 2020 License: MIT Imports: 11 Imported by: 0

README

go-fasttext

GoDoc

Package fasttext provides a simple wrapper for working with embedding vectors trained with Facebook's fastText library. It allows fast look-up of word embeddings from persistent data store (SQLite3).

Documentation

Overview

Package fasttext provides a simple wrapper for working with embedding vectors trained with Facebook's fastText library. It allows fast look-up of word embeddings from persistent data store (SQLite3).

Installation

go get -u github.com/justinfargnoli/go-fasttext

After training a model or downloading a '*.vec' data file from the fastText project, you can initialize the SQLite3 database (in your code):

import (
	_ "github.com/mattn/go-sqlite3"
	"github.com/justinfargnoli/go-fasttext"
)
...
ft := fasttext.New("/path/to/sqlite3/file")
err := ft.Build("/path/to/word/embedding/.vec/file")

This will create a new file on your disk for the SQLite3 database. Once the above step is finished, you can start looking up word embeddings (in your code):

embeddingVector, err := ft.GetEmb("king")
if err != nil {
	fmt.Println(err)
}
fmt.Println(embeddingVector)

Each word's embedding vector is a slice of float64 with length of 300.

Note that you only need to initialize the SQLite3 database once. The next time you use it you can skip the call to BuildDB.

For faster querying during runtime, you can use an in-memory database.

ft := NewFastTextInMem("/path/to/sqlite3/file")

This creates an in-memory SQLite3 database which is a copy of the on-disk one. Using the in-memory version makes query time much faster, but takes a few minutes to load the database.

Index

Constants

View Source
const (
	// TableName used in SQLite3
	TableName = "fasttext"
	// Dim is the number of dimensions in FastText word embedding vectors
	Dim = 300
)

Variables

View Source
var (
	// ErrNoEmbFound ...
	ErrNoEmbFound = errors.New("No embedding found for the given word")
	// ByteOrder is for the serialization of the embedding vector in
	// SQLite3 database.
	ByteOrder = binary.BigEndian
)

Functions

This section is empty.

Types

type FastText

type FastText struct {
	// contains filtered or unexported fields
}

The FastText session. In multi-thread setting, each thread must have its own copy of FastText session. A single FastText session cannot be shared among multiple threads.

func New

func New(dbFilename string) *FastText

New starts a new FastText session given the location of the SQLite3 database file.

func NewInMemory

func NewInMemory(dbFilename string) *FastText

NewInMemory creates a new FastText session that uses an in-memory database for faster query time. The on-disk SQLite3 database (given by dbFilename) will be loaded into an in-memory SQLite3 database in this function, which will take a few miniutes to finish.

func (*FastText) AllEmbeddingVectors

func (ft *FastText) AllEmbeddingVectors() ([][]float64, error)

AllEmbeddingVectors returns all embedding vectors

func (*FastText) Build

func (ft *FastText) Build(wordEmbFile io.Reader) error

Build initialize the SQLite3 database by importing the word embeddings from the .vec file downloaded from https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md

func (*FastText) Close

func (ft *FastText) Close() error

Close must be called before finishing using this FastText session.

func (*FastText) EmbeddingVector

func (ft *FastText) EmbeddingVector(word string) ([]float64, error)

EmbeddingVector returns the word embedding of the given word.

func (*FastText) MostSimilarEmbeddingVector

func (ft *FastText) MostSimilarEmbeddingVector(queryEmbedding []float64) ([]float64, float64, error)

MostSimilarEmbeddingVector returns the embedding vector which is most similar to the one passed

Errors from FastText.GetAllEmb() and cosine_similarity.Cosine() will be propogated.

func (*FastText) MultiWordEmbeddingVector

func (ft *FastText) MultiWordEmbeddingVector(words []string) (vector.Vector, error)

MultiWordEmbeddingVector builds an embedding vector to represent the array of words passed to it.

This is done by averaging the embedding vector of each string in the array.

If a string in words is not in the database, then an error indicator is returned.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL