multilang

Documentation

Index

• Constants
• func Register(detector lingua.LanguageDetector, defaultLang string)
• type CharFilter
  • func (c *CharFilter) Filter(input []byte) []byte
• type Tokenizer
  • func (d *Tokenizer) Tokenize(input []byte) analysis.TokenStream

Constants

const (
	// Name used for all components
	Name = "multilang"
	// LangDivider is a special symbol added to the end of the input
	// after that symbol detected lang name is stored
	LangDivider = byte('_')
)

Functions

func Register

func Register(detector lingua.LanguageDetector, defaultLang string)

Register multilang analyzer

Types

type CharFilter

type CharFilter struct {
	// contains filtered or unexported fields
}

CharFilter detects input language and appends it to the input bytes

func (*CharFilter) Filter

func (c *CharFilter) Filter(input []byte) []byte

Filter detects input language and appends it to the end of the input bytes

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer converts input bytes to the token stream using analyzer for specific lang

func (*Tokenizer) Tokenize

func (d *Tokenizer) Tokenize(input []byte) analysis.TokenStream

Tokenize converts input bytes into token stream using analyzer for specific lang