token

Documentation

Overview

Package lowercase implements a TokenFilter which converts tokens to lower case according to unicode rules.

Package stop implements a TokenFilter removing tokens found in a TokenMap.

It constructor takes the following arguments:

"stop_token_map" (string): the name of the token map identifying tokens to remove.

Index

• Constants
• func ApostropheFilter(input analysis.TokenStream) analysis.TokenStream
• func CamelCaseFilter(input analysis.TokenStream) analysis.TokenStream
• func DefaultLowerCaseFilter(input analysis.TokenStream) analysis.TokenStream
• func NewDictionaryCompoundFilter(dict analysis.TokenMap, minWordSize, minSubWordSize, maxSubWordSize int, ...) analysis.TokenFilter
• func NewEdgeNgramFilter(back Side, minLength, maxLength int) analysis.TokenFilter
• func NewElisionFilter(articles analysis.TokenMap) analysis.TokenFilter
• func NewKeyWordMarkerFilter(keyWords analysis.TokenMap) analysis.TokenFilter
• func NewLengthFilter(min, max int) analysis.TokenFilter
• func NewNgramFilter(minLength, maxLength int) analysis.TokenFilter
• func NewShingleFilter(min, max int, outputOriginal bool, sep, fill string) analysis.TokenFilter
• func NewStopTokensFilter(stopTokens analysis.TokenMap) analysis.TokenFilter
• func NewTruncateTokenFilter(length int) analysis.TokenFilter
• func NewUnicodeNormalizeFilter(form norm.Form) analysis.TokenFilter
• func PorterStemmer(input analysis.TokenStream) analysis.TokenStream
• func ReverseFilter(input analysis.TokenStream) analysis.TokenStream
• func UniqueTermFilter(input analysis.TokenStream) analysis.TokenStream
• type DictionaryCompoundFilter
  • func (f *DictionaryCompoundFilter) Filter(input analysis.TokenStream) analysis.TokenStream
• type LengthFilter
• type LowerCaseState
  • func (s *LowerCaseState) Member(sym rune, peek *rune) bool
  • func (s *LowerCaseState) StartSym(sym rune) bool
• type NonAlphaNumericCaseState
  • func (s *NonAlphaNumericCaseState) Member(sym rune, peek *rune) bool
  • func (s *NonAlphaNumericCaseState) StartSym(sym rune) bool
• type NumberCaseState
  • func (s *NumberCaseState) Member(sym rune, peek *rune) bool
  • func (s *NumberCaseState) StartSym(sym rune) bool
• type Parser
  • func NewParser(length, index int) *Parser
  • func (p *Parser) FlushTokens() []*analysis.Token
  • func (p *Parser) NewState(sym rune) State
  • func (p *Parser) Push(sym rune, peek *rune)
• type ShingleFilter
• type Side
• type State
• type UpperCaseState
  • func (s *UpperCaseState) Member(sym rune, peek *rune) bool
  • func (s *UpperCaseState) StartSym(sym rune) bool

Constants

const Apostrophe = '\''

const Apostrophes = string(Apostrophe) + string(RightSingleQuotationMark)

const RightSingleQuotationMark = '’'

Functions

func ApostropheFilter

func ApostropheFilter(input analysis.TokenStream) analysis.TokenStream

func CamelCaseFilter

func CamelCaseFilter(input analysis.TokenStream) analysis.TokenStream

CamelCaseFilter splits a given token into a set of tokens where each resulting token falls into one the following classes:

1. Upper case followed by lower case letters. Terminated by a number, an upper case letter, and a non alpha-numeric symbol.
2. Upper case followed by upper case letters. Terminated by a number, an upper case followed by a lower case letter, and a non alpha-numeric symbol.
3. Lower case followed by lower case letters. Terminated by a number, an upper case letter, and a non alpha-numeric symbol.
4. Number followed by numbers. Terminated by a letter, and a non alpha-numeric symbol.
5. Non alpha-numeric symbol followed by non alpha-numeric symbols. Terminated by a number, and a letter.

It does a one-time sequential pass over an input token, from left to right. The scan is greedy and generates the longest substring that fits into one of the classes.

See the test file for examples of classes and their parsings.

func DefaultLowerCaseFilter

func DefaultLowerCaseFilter(input analysis.TokenStream) analysis.TokenStream

func NewDictionaryCompoundFilter

func NewDictionaryCompoundFilter(dict analysis.TokenMap, minWordSize, minSubWordSize, maxSubWordSize int,
	onlyLongestMatch bool) analysis.TokenFilter

func NewEdgeNgramFilter

func NewEdgeNgramFilter(back Side, minLength, maxLength int) analysis.TokenFilter

func NewElisionFilter

func NewElisionFilter(articles analysis.TokenMap) analysis.TokenFilter

func NewKeyWordMarkerFilter

func NewKeyWordMarkerFilter(keyWords analysis.TokenMap) analysis.TokenFilter

func NewLengthFilter

func NewLengthFilter(min, max int) analysis.TokenFilter

func NewNgramFilter

func NewNgramFilter(minLength, maxLength int) analysis.TokenFilter

func NewShingleFilter

func NewShingleFilter(min, max int, outputOriginal bool, sep, fill string) analysis.TokenFilter

func NewStopTokensFilter

func NewStopTokensFilter(stopTokens analysis.TokenMap) analysis.TokenFilter

func NewTruncateTokenFilter

func NewTruncateTokenFilter(length int) analysis.TokenFilter

func NewUnicodeNormalizeFilter

func NewUnicodeNormalizeFilter(form norm.Form) analysis.TokenFilter

func PorterStemmer

func PorterStemmer(input analysis.TokenStream) analysis.TokenStream

func ReverseFilter

func ReverseFilter(input analysis.TokenStream) analysis.TokenStream

func UniqueTermFilter

func UniqueTermFilter(input analysis.TokenStream) analysis.TokenStream

UniqueTermFilter retains only the tokens which mark the first occurrence of a term. Tokens whose term appears in a preceding token are dropped.

Types

type DictionaryCompoundFilter

type DictionaryCompoundFilter struct {
	// contains filtered or unexported fields
}

func (*DictionaryCompoundFilter) Filter

func (f *DictionaryCompoundFilter) Filter(input analysis.TokenStream) analysis.TokenStream

type LengthFilter

type LengthFilter struct {
	// contains filtered or unexported fields
}

type LowerCaseState

type LowerCaseState struct{}

func (*LowerCaseState) Member

func (s *LowerCaseState) Member(sym rune, peek *rune) bool

func (*LowerCaseState) StartSym

func (s *LowerCaseState) StartSym(sym rune) bool

type NonAlphaNumericCaseState

type NonAlphaNumericCaseState struct{}

func (*NonAlphaNumericCaseState) Member

func (s *NonAlphaNumericCaseState) Member(sym rune, peek *rune) bool

func (*NonAlphaNumericCaseState) StartSym

func (s *NonAlphaNumericCaseState) StartSym(sym rune) bool

type NumberCaseState

type NumberCaseState struct{}

func (*NumberCaseState) Member

func (s *NumberCaseState) Member(sym rune, peek *rune) bool

func (*NumberCaseState) StartSym

func (s *NumberCaseState) StartSym(sym rune) bool

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser accepts a symbol and passes it to the current state (representing a class). The state can accept it (and accumulate it). Otherwise, the parser creates a new state that starts with the pushed symbol.

Parser accumulates a new resulting token every time it switches state. Use FlushTokens() to get the results after the last symbol was pushed.

func NewParser

func NewParser(length, index int) *Parser

func (*Parser) FlushTokens

func (p *Parser) FlushTokens() []*analysis.Token

func (*Parser) NewState

func (p *Parser) NewState(sym rune) State

Note. States have to have different starting symbols.

func (*Parser) Push

func (p *Parser) Push(sym rune, peek *rune)

type ShingleFilter

type ShingleFilter struct {
	// contains filtered or unexported fields
}

type Side

type Side bool

const BACK Side = true

const FRONT Side = false

type State

type State interface {
	// is _sym_ the start character
	StartSym(sym rune) bool

	// is _sym_ a member of a class.
	// peek, the next sym on the tape, can also be used to determine a class.
	Member(sym rune, peek *rune) bool
}

States codify the classes that the parser recognizes.

type UpperCaseState

type UpperCaseState struct {
	// contains filtered or unexported fields
}

func (*UpperCaseState) Member

func (s *UpperCaseState) Member(sym rune, peek *rune) bool

func (*UpperCaseState) StartSym

func (s *UpperCaseState) StartSym(sym rune) bool