charlm

Documentation

Overview

Package charlm provides an implementation of a character-level language model that uses a recurrent neural network as its backbone. A fully connected softmax layer (a.k.a decoder) is placed on top of each recurrent hidden state to predict the next character.

Index

• Constants
• func CalculatePerplexity(m *Model, text string) mat.Float
• func Convert(modelPath string, flairModelName, configFileName, modelFileName string)
• type Config
  • func LoadConfig(file string) (Config, error)
• type Generator
  • func NewGenerator(model *Model, config GeneratorConfig) *Generator
  • func (m *Generator) GenerateText(prefix string) (text string, logProb mat.Float)
• type GeneratorConfig
• type Model
  • func LoadModel(modelPath string) (*Model, error)
  • func New(config Config) *Model
  • func (m *Model) Forward(in interface{}) interface{}
  • func (m *Model) GetEmbeddings(xs []string) []ag.Node
  • func (m *Model) InitProcessor()
  • func (m *Model) Initialize(rndGen *rand.LockedRand)
  • func (m *Model) UseProjection(xs []ag.Node) []ag.Node
• type Trainer
  • func NewTrainer(config TrainingConfig, corpus corpora.TextCorpusIterator, model *Model) *Trainer
  • func (t *Trainer) Train()
• type TrainingConfig

Constants

const (
	// DefaultSequenceSeparator is the default sequence separator value for the
	// character-level language model.
	DefaultSequenceSeparator = "[SEP]"
	// DefaultUnknownToken is the default unknown token value for the
	// character-level language model.
	DefaultUnknownToken = "[UNK]"
)

Functions

func CalculatePerplexity

func CalculatePerplexity(m *Model, text string) mat.Float

CalculatePerplexity returns the perplexity for the text calculated as Exp(CrossEntropyLoss). The output of the language model is directly compared to the expected targets extracted from the input itself.

func Convert

func Convert(modelPath string, flairModelName, configFileName, modelFileName string)

Convert converts the parameters of a Flair model into spaGO structures.

Types

type Config

type Config struct {
	VocabularySize    int
	EmbeddingSize     int
	HiddenSize        int
	OutputSize        int    // use the projection layer when the output size is > 0
	SequenceSeparator string // empty string is replaced with DefaultSequenceSeparator
	UnknownToken      string // empty string is replaced with DefaultUnknownToken
}

Config provides configuration settings for a Character-level Language Model. TODO: add dropout

func LoadConfig

func LoadConfig(file string) (Config, error)

LoadConfig loads a Config from file.

type Generator

type Generator struct {
	GeneratorConfig
	// contains filtered or unexported fields
}

Generator is a character-level language model text generator.

func NewGenerator

func NewGenerator(model *Model, config GeneratorConfig) *Generator

NewGenerator returns a new Generator.

func (*Generator) GenerateText

func (m *Generator) GenerateText(prefix string) (text string, logProb mat.Float)

GenerateText returns a new sequences of text generated by the learned language model, and its log probability. The output text has a maximum length defined in the generator configuration. The text is incrementally constructed character by character, using all previously sampled characters as input to predict the next one.

type GeneratorConfig

type GeneratorConfig struct {
	MaxCharacters int
	StopAtEOS     bool
	Temperature   mat.Float
}

GeneratorConfig provides configuration settings for a Character-level Language Model Generator.

type Model

type Model struct {
	nn.BaseModel
	Config
	Decoder          *linear.Model
	Projection       *linear.Model
	RNN              *lstm.Model
	Embeddings       []nn.Param `spago:"type:weights;scope:model"`
	Vocabulary       *vocabulary.Vocabulary
	UsedEmbeddings   map[int]ag.Node `spago:"scope:processor"`
	UnknownEmbedding ag.Node         `spago:"scope:processor"`
}

Model implements a Character-level Language Model.

func LoadModel

func LoadModel(modelPath string) (*Model, error)

LoadModel loads a Model model from file.

func New

func New(config Config) *Model

New returns a new character-level language Model, initialized according to the given configuration.

func (*Model) Forward

func (m *Model) Forward(in interface{}) interface{}

Forward performs the forward step for each input and returns the result.

func (*Model) GetEmbeddings

func (m *Model) GetEmbeddings(xs []string) []ag.Node

GetEmbeddings transforms the string sequence xs into a sequence of embeddings nodes.

func (*Model) InitProcessor

func (m *Model) InitProcessor()

InitProcessor initializes embeddings needed by the Forward().

func (*Model) Initialize

func (m *Model) Initialize(rndGen *rand.LockedRand)

Initialize initializes the Model m using the given random generator.

func (*Model) UseProjection

func (m *Model) UseProjection(xs []ag.Node) []ag.Node

UseProjection performs a linear projection with Processor.Projection model, if available, otherwise returns xs unmodified.

type Trainer

type Trainer struct {
	TrainingConfig
	// contains filtered or unexported fields
}

Trainer implements the training process for a Character-level Language Model.

func NewTrainer

func NewTrainer(config TrainingConfig, corpus corpora.TextCorpusIterator, model *Model) *Trainer

NewTrainer returns a new Trainer.

func (*Trainer) Train

func (t *Trainer) Train()

Train executes the training process.

type TrainingConfig

type TrainingConfig struct {
	Seed                  uint64
	BatchSize             int
	BackStep              int
	GradientClipping      mat.Float
	SerializationInterval int
	UpdateMethod          gd.MethodConfig
	ModelPath             string
}

TrainingConfig provides configuration settings for a Character-level Language Trainer. TODO: add dropout