pipeline

Documentation

Index

• func Lower(text string) string
• func Pipeline(text string, pl *Config) []string
• func RemoveSymbols(text string) string
• func Tokenize(text string) []string
• type Config

Functions

func Lower

func Lower(text string) string

Lower converts a string to lowercase

func Pipeline

func Pipeline(text string, pl *Config) []string

Pipeline performs steps to convert a string into a CLF input

func RemoveSymbols

func RemoveSymbols(text string) string

RemoveSymbols removes symbols from string

func Tokenize

func Tokenize(text string) []string

Tokenize splits the text into tokens

Types

type Config

type Config struct {
	RemoveSymbols bool    `mapstructure:"remove_symbols"`
	Lower         bool    `mapstructure:"lower"`
	Threshold     float64 `mapstructure:"threshold"`
}

Config defines a Pipeline configuration