exception

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 3, 2020 License: Apache-2.0 Imports: 5 Imported by: 0

Documentation

Overview

package exception implements a Tokenizer which extracts pieces matched by a regular expression from the input data, delegates the rest to another tokenizer, then insert back extracted parts in the token stream. Use it to preserve sequences which a regular tokenizer would alter or remove.

Its constructor takes the following arguments:

"exceptions" ([]string): one or more Go regular expressions matching the sequence to preserve. Multiple expressions are combined with "|".

"tokenizer" (string): the name of the tokenizer processing the data not matched by "exceptions".

Index

Constants

View Source
const Name = "exception"

Variables

This section is empty.

Functions

func ExceptionsTokenizerConstructor

func ExceptionsTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)

Types

type ExceptionsTokenizer

type ExceptionsTokenizer struct {
	// contains filtered or unexported fields
}

func NewExceptionsTokenizer

func NewExceptionsTokenizer(exception *regexp.Regexp, remaining analysis.Tokenizer) *ExceptionsTokenizer

func (*ExceptionsTokenizer) Tokenize

func (t *ExceptionsTokenizer) Tokenize(input []byte) analysis.TokenStream

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL