embeddings

Documentation

Index

• func ClearUsedEmbeddings()
• func Close()
• type Config
• type Model
  • func New(config Config) *Model
  • func (m *Model) ClearUsedEmbeddings()
  • func (m *Model) Close()
  • func (m *Model) Count() int
  • func (m *Model) DropAll() error
  • func (m *Model) Encode(words []string) []ag.Node
  • func (m *Model) GetStoredEmbedding(word string) nn.Param
  • func (m *Model) Load(filename string)
  • func (m *Model) SetEmbedding(word string, value *mat.Dense)
  • func (m *Model) SetEmbeddingFromData(word string, data []mat.Float)

Functions

func ClearUsedEmbeddings

func ClearUsedEmbeddings()

ClearUsedEmbeddings clears the cache of the used embeddings of all instantiated embeddings models. Beware of any external references to the values of m.UsedEmbeddings. These are weak references!

func Close

func Close()

Close closes the DBs underlying all instantiated embeddings models. It automatically clears the caches.

Types

type Config

type Config struct {
	// Size of the embedding vectors.
	Size int
	// Whether to return the `ZeroEmbedding` in case the word doesn't exist in the embeddings map.
	// If it is false, nil is returned instead, so the caller has more responsibility but also more control.
	UseZeroEmbedding bool
	// The path to DB on the drive
	DBPath string
	// Whether to use the map in read-only mode (embeddings are not updated during training).
	ReadOnly bool
	// Whether to force the deletion of any existing DB to start with an empty embeddings map.
	ForceNewDB bool
}

Config provides configuration settings for an embeddings Model.

type Model

type Model struct {
	nn.BaseModel
	Config
	Storage        *kvdb.KeyValueDB
	UsedEmbeddings *syncmap.Map `spago:"type:params;scope:model"`
	ZeroEmbedding  nn.Param     `spago:"type:weights"`
}

Model implements an embeddings model.

func New

func New(config Config) *Model

New returns a new embedding model.

func (*Model) ClearUsedEmbeddings

func (m *Model) ClearUsedEmbeddings()

ClearUsedEmbeddings clears the cache of the used embeddings. Beware of any external references to the values of m.UsedEmbeddings. These are weak references!

func (*Model) Close

func (m *Model) Close()

Close closes the DB underlying the model of the embeddings map. It automatically clears the cache.

func (*Model) Count

func (m *Model) Count() int

Count counts how many embeddings are stored in the DB. It invokes log.Fatal in case of reading errors.

func (*Model) DropAll

func (m *Model) DropAll() error

DropAll clears the cache of used embeddings and drops all the data stored in the DB.

func (*Model) Encode

func (m *Model) Encode(words []string) []ag.Node

Encode returns the embeddings associated with the input words. The embeddings are returned as Node(s) already inserted in the graph. To words that have no embeddings, the corresponding nodes are nil or the `ZeroEmbedding`, depending on the configuration.

func (*Model) GetStoredEmbedding

func (m *Model) GetStoredEmbedding(word string) nn.Param

GetStoredEmbedding returns the parameter (the word embedding) associated with the given word. It first looks for the exact correspondence of the word. If there is no match, it tries the word lowercase.

The returned embedding is also cached in m.UsedEmbeddings for two reasons:

• to allow a faster recovery;
• to keep track of used embeddings, should they be optimized.

If no embedding is found, nil is returned. It panics in case of Storage errors.

func (*Model) Load

func (m *Model) Load(filename string)

Load inserts the pre-trained embeddings into the model.

func (*Model) SetEmbedding

func (m *Model) SetEmbedding(word string, value *mat.Dense)

SetEmbedding inserts a new word embedding. If the word is already on the map, it overwrites the existing value with the new one.

func (*Model) SetEmbeddingFromData

func (m *Model) SetEmbeddingFromData(word string, data []mat.Float)

SetEmbeddingFromData inserts a new word embeddings. If the word is already on the map, overwrites the existing value with the new one.