tokenize

Documentation

Index

• Constants
• func TokenSpans(text string) [][2]int
• func TokenTexts(text string) []string
• type Atom
  • func Atomize(text string) []Atom
  • func (a Atom) Normal() string

Constants

const (
	RU    = "RU"
	LAT   = "LAT"
	INT   = "INT"
	PUNCT = "PUNCT"
	OTHER = "OTHER"
)

Atom kinds mirror upstream razdel.segmenters.tokenize (RU, LAT, INT, PUNCT, OTHER).

const Brackets = openBracket + closeBracket

Brackets is OPEN + CLOSE (upstream BRACKETS).

const Dashes = dashes

Dashes is upstream DASHES.

const Endings = endings

Endings is upstream ENDINGS.

const PunctSet = "\\/!#$%&*+,.:;<=>?@^_`|~№…" + Dashes + Quotes + Brackets

PunctSet is the character class used in upstream ATOM's PUNCT alternative (PUNCTS in tokenize.py: ascii punct + № … + dashes + quotes + brackets).

const Quotes = openQuote + closeQuote + genericQuote

Quotes is OPEN + CLOSE + GENERIC (upstream QUOTES).

Functions

func TokenSpans

func TokenSpans(text string) [][2]int

TokenSpans returns (start, end) byte offsets for each token, matching upstream find_substrings.

func TokenTexts

func TokenTexts(text string) []string

TokenTexts returns final token strings (upstream tokenize() without find_substrings offset pass).

Types

type Atom

type Atom struct {
	Start int
	Stop  int
	Type  string
	Text  string
}

Atom is one lexical atom from upstream Atom (start/stop as byte offsets, half-open).

func Atomize

func Atomize(text string) []Atom

Atomize scans text into atoms using the same classification order as upstream ATOM regex.

func (Atom) Normal

func (a Atom) Normal() string

Normal is lowercase text (upstream atom.normal).