Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type SearchTokenizer ¶
type SearchTokenizer struct {
// contains filtered or unexported fields
}
func NewSearchTokenizer ¶
func NewSearchTokenizer(seg *gse.Segmenter) *SearchTokenizer
func (*SearchTokenizer) Tokenize ¶
func (t *SearchTokenizer) Tokenize(input []byte) analysis.TokenStream
type StandardTokenizer ¶
type StandardTokenizer struct {
// contains filtered or unexported fields
}
func NewStandardTokenizer ¶
func NewStandardTokenizer(seg *gse.Segmenter) *StandardTokenizer
func (*StandardTokenizer) Tokenize ¶
func (t *StandardTokenizer) Tokenize(input []byte) analysis.TokenStream
Source Files
Click to show internal directories.
Click to hide internal directories.