heuristic

Documentation

Index

• func TagTokens(ts []token.Token, d *dict.Dictionary, m *util.Model)

Functions

func TagTokens

func TagTokens(ts []token.Token, d *dict.Dictionary, m *util.Model)

TagTokens is important for both heuristic and Bayes approaches. It analyses tokens and sets up token's indices. Indices determine if a token is a potential unimonial, binomial or trinomial. Then if fills out signfificant number of features pertained to the token.