heuristic

Documentation

Index

• func TagTokens(ts []token.TokenSN, d *dict.Dictionary)

Functions

func TagTokens

func TagTokens(ts []token.TokenSN, d *dict.Dictionary)

TagTokens is important for both heuristic and Bayes approaches. It analyses tokens and sets up token's indices. Indices determine if a token is a potential unimonial, binomial or trinomial. Then if fills out signfificant number of features pertained to the token.