gse

Documentation

Index

• Constants
• type GseTokenizer
  • func NewGseTokenizer(dictPath, stopPath string, alpha bool) (*GseTokenizer, error)
  • func (g *GseTokenizer) Tokenize(text []byte) analysis.TokenStream

Constants

const Name = "gse"

Types

type GseTokenizer

type GseTokenizer struct {
	// contains filtered or unexported fields
}

func NewGseTokenizer

func NewGseTokenizer(dictPath, stopPath string, alpha bool) (*GseTokenizer, error)

NewGseTokenizer create a gse cut tokenizer

func (*GseTokenizer) Tokenize

func (g *GseTokenizer) Tokenize(text []byte) analysis.TokenStream

Tokenize cut the text to bleve token stream