unicode

package
Version: v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 5, 2018 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Index

Constants

View Source
const Name = "unicode"

Variables

This section is empty.

Functions

func UnicodeTokenizerConstructor

func UnicodeTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)

Types

type UnicodeTokenizer

type UnicodeTokenizer struct {
}

func NewUnicodeTokenizer

func NewUnicodeTokenizer() *UnicodeTokenizer

func (*UnicodeTokenizer) Tokenize

func (rt *UnicodeTokenizer) Tokenize(input []byte) analysis.TokenStream

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL