nlp

Documentation

Index

• func NGram(n int, s string) []string
• func RemoveEmptyWord(a []string) []string
• func WeightBinary(a []string) data.Map
• func WeightLogTF(a []string) data.Map
• func WeightTF(a []string) data.Map
• func WordNGram(n int, words []string, sep string) []string

Functions

func NGram

func NGram(n int, s string) []string

NGram returns UTF-8 character n-grams created from the given text. This function assumes that s only contains valid UTF-8 letters. It returns an empty array when n isn't greater than 0.

func RemoveEmptyWord

func RemoveEmptyWord(a []string) []string

RemoveEmptyWord removes an empty string from an array of strings.

func WeightBinary

func WeightBinary(a []string) data.Map

WeightBinary creates a map having weights of each word. The weight is 1 if there's at least one word, or 0 otherwise. Because feature vectors created by this function is sparse, all values in resulting maps are 1. In other words, instead of having 0 as a value, a key doesn't exist for a word that is not in the given array.

func WeightLogTF

func WeightLogTF(a []string) data.Map

WeightLogTF creates a map having a word as a key and its log(1 + tf) as a value. This function is useful when some words appear too much but binary weight isn't sufficient.

func WeightTF

func WeightTF(a []string) data.Map

WeightTF creates a map having a word as a key and its count (i.e. tf) as a value.

func WordNGram

func WordNGram(n int, words []string, sep string) []string

WordNGram creates word n-grams from the given array of words. A separator sep can be any string.It returns an empty array when n isn't greater than 0.