tokenizer

Documentation

Overview

Package tokenizer implements gospacy's rule-based tokenizer. The engine itself is language-independent; per-language data (prefix/suffix/infix patterns + the exception table) is supplied via Rules, typically built by lang/<code>/MakeRules. Algorithm mirrors spacy.Tokenizer (tokenizer.pyx) with the same special-case-first / split-affixes / collect-token order.

Index

• type Rules
  • func NewRules(in RulesInput) (*Rules, error)
  • func (r *Rules) FindInfixes(s string) []Span
  • func (r *Rules) FindPrefix(s string) (string, bool)
  • func (r *Rules) FindSuffix(s string) (string, bool)
  • func (r *Rules) IsTokenMatch(s string) bool
  • func (r *Rules) IsURLMatch(s string) bool
  • func (r *Rules) Special(s string) ([]SpecialPiece, bool)
• type RulesInput
• type Span
• type SpecialPiece
• type Token
• type Tokenizer
  • func New(rules *Rules) *Tokenizer
  • func (t *Tokenizer) ToDoc(v *vocab.Vocab, text string) *doc.Doc
  • func (t *Tokenizer) Tokenize(text string) []Token

Types

type Rules

type Rules struct {
	// contains filtered or unexported fields
}

Rules holds the compiled patterns. Build once, reuse.

func NewRules

func NewRules(in RulesInput) (*Rules, error)

NewRules compiles every pattern in in and returns the immutable Rules.

func (*Rules) FindInfixes

func (r *Rules) FindInfixes(s string) []Span

FindInfixes returns every infix-pattern match span in s, in left-to-right order. Spans from different patterns may overlap.

func (*Rules) FindPrefix

func (r *Rules) FindPrefix(s string) (string, bool)

FindPrefix returns the longest prefix matched by any prefix pattern, or "" + false if nothing matches. regexp2 returns rune (character) offsets, so we slice using []rune.

func (*Rules) FindSuffix

func (r *Rules) FindSuffix(s string) (string, bool)

FindSuffix returns the longest suffix matched, or "" + false. Each suffix pattern is compiled with a "$" anchor (see NewRules), so every match already ends at the string terminus. We pick the match that starts earliest (leftmost), which gives the longest suffix — mirroring spaCy's suffix_search behaviour. regexp2 returns rune (character) offsets, so we slice using []rune.

func (*Rules) IsTokenMatch

func (r *Rules) IsTokenMatch(s string) bool

IsTokenMatch returns true if the entire string matches token_match. regexp2 returns rune offsets, so we compare against rune count.

func (*Rules) IsURLMatch

func (r *Rules) IsURLMatch(s string) bool

IsURLMatch returns true if s matches url_match. regexp2 returns rune offsets, so we compare against rune count.

func (*Rules) Special

func (r *Rules) Special(s string) ([]SpecialPiece, bool)

Special looks up s in the specials map.

type RulesInput

type RulesInput struct {
	Prefixes   []string
	Suffixes   []string
	Infixes    []string
	TokenMatch string
	URLMatch   string
	Specials   map[string][]SpecialPiece
}

RulesInput is the raw, uncompiled tokenizer data; compiled once by NewRules.

type Span

type Span struct{ Start, End int }

Span is a half-open [Start, End) UTF-8 byte range inside a string.

type SpecialPiece

type SpecialPiece struct {
	Orth  string
	Norm  string
	Lemma string
	Tag   string
	POS   string
}

SpecialPiece is one orth-string emitted by a special-case rule. Norm carries the optional NORM-attribute override Python attaches to clitic sub-tokens (e.g. {Orth:"n't", Norm:"not"}); empty string means "no override, fall back to the lexeme's Lower hash". See ToDoc for application.

type Token

type Token struct {
	Orth string
	Idx  int  // Unicode codepoint (rune) offset in the original text, matching spaCy's token.idx
	WS   bool // true if a whitespace char follows in the original text
	// NormOverride is set only when this token came from a special-case rule
	// that supplied a NORM attribute (e.g. "n't"→"not"). Empty string means
	// "no override". ToDoc reads this to bypass the default Lower fallback.
	NormOverride string
}

Token is one output token. Minimal subset of spacy.Token.

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer is the language-independent tokenizer engine.

func New

func New(rules *Rules) *Tokenizer

New returns a Tokenizer driven by rules.

func (*Tokenizer) ToDoc

func (t *Tokenizer) ToDoc(v *vocab.Vocab, text string) *doc.Doc

ToDoc tokenizes text into a *doc.Doc, interning Orth into v.StringStore. Unlike Tokenize (which returns the legacy []Token slice), this is the entry point used by bundle.Pipe — it produces the runtime container the pipeline components mutate.

Per-token Whitespace is reconstructed from consecutive Idx values plus the final tail of text, so Doc.Text() round-trips the input verbatim.

func (*Tokenizer) Tokenize

func (t *Tokenizer) Tokenize(text string) []Token

Tokenize splits text into tokens. Walks whitespace-separated chunks left-to-right; each chunk recursively split by special-case map, prefix, suffix, infix, in that order. A second pass then merges consecutive tokens whose concatenated orths match a special case (mirroring spaCy's _apply_special_cases retokenization step).

Whitespace handling mirrors spaCy: a single ASCII space (' ') immediately after a word token sets WS=true on that token and is consumed silently. Any other whitespace run (leading spaces, tabs, newlines, or 2+ spaces after a word) is emitted as its own token. When a whitespace run starts with a ' ' after a word token, that first space is consumed as WS=true and the rest of the run is emitted as a token.

Idx values are Unicode codepoint (rune) offsets, matching spaCy's token.idx.