tokenizer

package
v0.0.0-...-d3241bd Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Examples

Constants

View Source 
const (
	JiebaCutModeSearch = jieba.CutModeSearch
	JiebaCutModeMix    = jieba.CutModeMix
	JiebaCutModeFull   = jieba.CutModeFull
	JiebaCutModeHMM    = jieba.CutModeHMM
)
View Source 
const (
	// DefaultNGramLength is the pinned minimum and maximum ngram length.
	DefaultNGramLength uint32 = 2
	// MaxNGramLengthDifference is the largest supported Max-Min range.
	MaxNGramLengthDifference uint32 = 1
)
View Source 
const (
	// DefaultStandardMaxTokenLength is the pinned standard-tokenizer default.
	DefaultStandardMaxTokenLength uint32 = 255
	// MinStandardMaxTokenLength is the smallest accepted token length.
	MinStandardMaxTokenLength uint32 = 1
	// MaxStandardMaxTokenLength is the largest accepted token length.
	MaxStandardMaxTokenLength uint32 = 1_048_576
)

Variables

View Source 
var (
	// ErrInvalidJiebaTokenizerOptions identifies invalid construction options.
	ErrInvalidJiebaTokenizerOptions = errors.New("core: invalid jieba tokenizer options")
	// ErrInvalidJiebaUTF8 identifies input that the pinned Jieba decoder
	// cannot decode as one complete sequence.
	ErrInvalidJiebaUTF8 = jieba.ErrInvalidUTF8
)
View Source 
var ErrInvalidFTSAnalyzer = errors.New("core: invalid FTS analyzer")

ErrInvalidFTSAnalyzer identifies a nil or incomplete query/index analysis pipeline.

View Source 
var ErrInvalidNGramTokenizerOptions = errors.New("core: invalid ngram tokenizer options")

ErrInvalidNGramTokenizerOptions identifies invalid ngram construction options.

View Source 
var ErrInvalidStandardTokenizerOptions = errors.New("core: invalid standard tokenizer options")

ErrInvalidStandardTokenizerOptions identifies invalid standard tokenizer construction options.

View Source 
var ErrInvalidStemmerOptions = errors.New("core: invalid stemmer options")

ErrInvalidStemmerOptions identifies an unknown Snowball language or alias.

View Source 
var ErrTokenizerInputTooLarge = errors.New("core: tokenizer input exceeds uint32 offset capacity")

Functions

func DefaultJiebaDictDir 

func DefaultJiebaDictDir() string

DefaultJiebaDictDir returns the process-wide fallback directory.

func SetDefaultJiebaDictDir 

func SetDefaultJiebaDictDir(path string)

SetDefaultJiebaDictDir sets the process-wide fallback dictionary directory. Explicit options and ZVEC_JIEBA_DICT_DIR have higher priority.

func SupportedStemmerLanguages 

func SupportedStemmerLanguages() []string

SupportedStemmerLanguages returns all 115 case-sensitive libstemmer names and aliases in lexical order.

Types

type ASCIIFoldingTokenFilter 

type ASCIIFoldingTokenFilter struct{}

ASCIIFoldingTokenFilter replaces the baseline's supported Unicode codepoints with short ASCII equivalents. It is immutable and safe for concurrent use.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	filter := tokenizer.NewASCIIFoldingTokenFilter()
	tokens, err := filter.Filter(context.Background(), []tokenizer.Token{
		{Text: "café", Offset: 0, Position: 0},
		{Text: "Æsir", Offset: 6, Position: 1},
	})
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%s@%d ", token.Text, token.Offset)
	}
}

Output:
cafe@0 AEsir@6

func NewASCIIFoldingTokenFilter 

func NewASCIIFoldingTokenFilter() *ASCIIFoldingTokenFilter

NewASCIIFoldingTokenFilter constructs a stateless ASCII-folding filter.

func (*ASCIIFoldingTokenFilter) Filter 

func (*ASCIIFoldingTokenFilter) Filter(ctx context.Context, tokens []Token) ([]Token, error)

Filter returns owned, non-empty tokens with folded text. It preserves the order, offsets, and positions of every retained token. Malformed UTF-8 bytes pass through one byte at a time.

func (*ASCIIFoldingTokenFilter) Name 

func (*ASCIIFoldingTokenFilter) Name() string

type FTSAnalyzer 

type FTSAnalyzer interface {
	Analyze(ctx context.Context, text string) ([]Token, error)
}

FTSAnalyzer applies the same tokenization and filtering rules to indexed text and query terms. Implementations must be safe for concurrent calls if they are shared between collections.

type FTSAnalyzerFunc 

type FTSAnalyzerFunc func(context.Context, string) ([]Token, error)

FTSAnalyzerFunc adapts a function to FTSAnalyzer.

func (FTSAnalyzerFunc) Analyze 

func (f FTSAnalyzerFunc) Analyze(ctx context.Context, text string) ([]Token, error)

Analyze invokes f.

type FTSTokenizerPipeline 

type FTSTokenizerPipeline struct {
	// contains filtered or unexported fields
}

FTSTokenizerPipeline applies one tokenizer followed by zero or more filters. The pipeline snapshots the filter list and is safe for concurrent use when its tokenizer and filters are safe for concurrent use.

func NewFTSTokenizerPipeline 

func NewFTSTokenizerPipeline(tokenizer Tokenizer, filters ...TokenFilter) (*FTSTokenizerPipeline, error)

NewFTSTokenizerPipeline constructs a validated analysis pipeline.

func (*FTSTokenizerPipeline) Analyze 

func (p *FTSTokenizerPipeline) Analyze(ctx context.Context, text string) ([]Token, error)

Analyze tokenizes text and applies each filter in declaration order.

func (*FTSTokenizerPipeline) FilterNames 

func (p *FTSTokenizerPipeline) FilterNames() []string

FilterNames returns an owned ordered filter-name slice.

func (*FTSTokenizerPipeline) TokenizerName 

func (p *FTSTokenizerPipeline) TokenizerName() string

TokenizerName returns the configured tokenizer name.

type JiebaCutMode 

type JiebaCutMode = jieba.CutMode

JiebaCutMode selects the baseline Jieba segmentation algorithm.

type JiebaTokenizer 

type JiebaTokenizer struct {
	// contains filtered or unexported fields
}

JiebaTokenizer adapts the vendored Jieba implementation to Tokenizer.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	tokenizer, err := tokenizer.NewJiebaTokenizer(context.Background(), tokenizer.JiebaTokenizerOptions{
		DictDir: "testdata/jieba", CutMode: tokenizer.JiebaCutModeSearch,
	})
	if err != nil {
		panic(err)
	}
	tokens, err := tokenizer.Tokenize(context.Background(), "自然语言处理")
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%s@%d ", token.Text, token.Offset)
	}
}

Output:
自然@0 语言@6 自然语言@0 处理@12

func NewJiebaTokenizer 

func NewJiebaTokenizer(ctx context.Context, options JiebaTokenizerOptions) (*JiebaTokenizer, error)

NewJiebaTokenizer loads and validates the resources required by the chosen mode. Search and mix require both files, full requires only the dictionary, and HMM requires only the model.

func (*JiebaTokenizer) CutMode 

func (t *JiebaTokenizer) CutMode() JiebaCutMode

func (*JiebaTokenizer) DictDir 

func (t *JiebaTokenizer) DictDir() string

func (*JiebaTokenizer) Name 

func (*JiebaTokenizer) Name() string

func (*JiebaTokenizer) Tokenize 

func (t *JiebaTokenizer) Tokenize(ctx context.Context, text string) ([]Token, error)

Tokenize converts Jieba words to FTS tokens with contiguous positions.

type JiebaTokenizerOptions 

type JiebaTokenizerOptions struct {
	DictDir      string
	UserDictPath string
	CutMode      JiebaCutMode
}

JiebaTokenizerOptions configures dictionary resolution and cut mode. An empty CutMode means search. An empty DictDir resolves from ZVEC_JIEBA_DICT_DIR and then DefaultJiebaDictDir.

func DefaultJiebaTokenizerOptions 

func DefaultJiebaTokenizerOptions() JiebaTokenizerOptions

DefaultJiebaTokenizerOptions returns the baseline search-mode defaults.

func (JiebaTokenizerOptions) Validate 

func (o JiebaTokenizerOptions) Validate() error

Validate checks mode and the availability of required resource paths. File contents are validated by NewJiebaTokenizer.

type LowercaseTokenFilter 

type LowercaseTokenFilter struct{}

LowercaseTokenFilter applies the baseline Unicode 17 simple lowercase mapping. It is immutable and safe for concurrent use.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	filter := tokenizer.NewLowercaseTokenFilter()
	tokens, err := filter.Filter(context.Background(), []tokenizer.Token{
		{Text: "Go", Offset: 0, Position: 0},
		{Text: "ÜBER", Offset: 3, Position: 1},
	})
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%s@%d ", token.Text, token.Offset)
	}
}

Output:
go@0 über@3

func NewLowercaseTokenFilter 

func NewLowercaseTokenFilter() *LowercaseTokenFilter

NewLowercaseTokenFilter constructs a stateless lowercase filter.

func (*LowercaseTokenFilter) Filter 

func (*LowercaseTokenFilter) Filter(ctx context.Context, tokens []Token) ([]Token, error)

Filter returns an owned copy of tokens with text lowercased. Malformed UTF-8 bytes are copied one byte at a time, matching utf8proc_iterate fallback.

func (*LowercaseTokenFilter) Name 

func (*LowercaseTokenFilter) Name() string

type NGramTokenChars 

type NGramTokenChars uint32

NGramTokenChars is a mask of Unicode character classes retained by the tokenizer. Zero retains every valid UTF-8 codepoint, matching the baseline. 

const (
	NGramTokenCharLetter NGramTokenChars = 1 << iota
	NGramTokenCharDigit
	NGramTokenCharWhitespace
	NGramTokenCharPunctuation
	NGramTokenCharSymbol
)

type NGramTokenizer 

type NGramTokenizer struct {
	// contains filtered or unexported fields
}

NGramTokenizer emits overlapping UTF-8 codepoint ngrams.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	tokenizer, err := tokenizer.NewNGramTokenizer(tokenizer.NGramTokenizerOptions{
		Min: 2, Max: 2, TokenChars: tokenizer.NGramTokenCharLetter,
	})
	if err != nil {
		panic(err)
	}
	tokens, err := tokenizer.Tokenize(context.Background(), "向量Go")
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%s@%d ", token.Text, token.Offset)
	}
}

Output:
向量@0 量G@3 Go@6

func NewNGramTokenizer 

func NewNGramTokenizer(options NGramTokenizerOptions) (*NGramTokenizer, error)

NewNGramTokenizer constructs a validated tokenizer.

func (*NGramTokenizer) Max 

func (t *NGramTokenizer) Max() uint32

func (*NGramTokenizer) Min 

func (t *NGramTokenizer) Min() uint32

func (*NGramTokenizer) Name 

func (*NGramTokenizer) Name() string

func (*NGramTokenizer) TokenChars 

func (t *NGramTokenizer) TokenChars() NGramTokenChars

func (*NGramTokenizer) Tokenize 

func (t *NGramTokenizer) Tokenize(ctx context.Context, text string) ([]Token, error)

Tokenize emits ngrams ordered first by start codepoint and then by increasing length. Malformed UTF-8 bytes always terminate a segment.

type NGramTokenizerOptions 

type NGramTokenizerOptions struct {
	Min        uint32
	Max        uint32
	TokenChars NGramTokenChars
}

NGramTokenizerOptions configures codepoint ngram lengths and optional Unicode character-class filtering.

func DefaultNGramTokenizerOptions 

func DefaultNGramTokenizerOptions() NGramTokenizerOptions

DefaultNGramTokenizerOptions returns baseline bigram settings and retains all valid UTF-8 codepoints.

func (NGramTokenizerOptions) Validate 

func (o NGramTokenizerOptions) Validate() error

Validate checks the pinned positive uint32 and one-length-span constraints.

type StandardTokenizer 

type StandardTokenizer struct {
	// contains filtered or unexported fields
}

StandardTokenizer implements the Unicode 17 standard tokenizer behavior pinned by zvec commit 58375ff.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	tokenizer, err := tokenizer.NewStandardTokenizer(tokenizer.DefaultStandardTokenizerOptions())
	if err != nil {
		panic(err)
	}
	tokens, err := tokenizer.Tokenize(context.Background(), "Go向量 search 3.14")
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%s@%d ", token.Text, token.Offset)
	}
}

Output:
Go@0 向@2 量@5 search@9 3.14@16

func NewStandardTokenizer 

func NewStandardTokenizer(options StandardTokenizerOptions) (*StandardTokenizer, error)

NewStandardTokenizer constructs a validated tokenizer.

func (*StandardTokenizer) MaxTokenLength 

func (t *StandardTokenizer) MaxTokenLength() uint32

MaxTokenLength returns the configured codepoint limit.

func (*StandardTokenizer) Name 

func (*StandardTokenizer) Name() string

func (*StandardTokenizer) Tokenize 

func (t *StandardTokenizer) Tokenize(ctx context.Context, text string) ([]Token, error)

Tokenize emits owned token text with byte offsets into text and contiguous zero-based positions. MaxTokenLength counts decoded codepoints, not bytes.

type StandardTokenizerOptions 

type StandardTokenizerOptions struct {
	MaxTokenLength uint32
}

StandardTokenizerOptions configures StandardTokenizer. Use DefaultStandardTokenizerOptions when the caller does not supply a value.

func DefaultStandardTokenizerOptions 

func DefaultStandardTokenizerOptions() StandardTokenizerOptions

DefaultStandardTokenizerOptions returns options compatible with zvec 58375ff.

func (StandardTokenizerOptions) Validate 

func (o StandardTokenizerOptions) Validate() error

Validate checks the baseline max_token_length range.

type StemmerTokenFilter 

type StemmerTokenFilter struct {
	// contains filtered or unexported fields
}

StemmerTokenFilter applies one generated Snowball 3.1.1 algorithm. It is immutable and safe for concurrent use.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	filter, err := tokenizer.NewStemmerTokenFilter(tokenizer.StemmerTokenFilterOptions{Language: "english"})
	if err != nil {
		panic(err)
	}
	tokens, err := filter.Filter(context.Background(), []tokenizer.Token{
		{Text: "running", Offset: 0, Position: 0},
		{Text: "connections", Offset: 8, Position: 1},
	})
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%s@%d ", token.Text, token.Offset)
	}
}

Output:
run@0 connect@8

func NewStemmerTokenFilter 

func NewStemmerTokenFilter(options StemmerTokenFilterOptions) (*StemmerTokenFilter, error)

NewStemmerTokenFilter constructs a validated stemmer filter.

func (*StemmerTokenFilter) Filter 

func (f *StemmerTokenFilter) Filter(ctx context.Context, tokens []Token) ([]Token, error)

Filter returns an owned token slice with stemmed text and unchanged order, offsets, and positions. Empty tokens are retained, matching libstemmer.

func (*StemmerTokenFilter) Language 

func (f *StemmerTokenFilter) Language() string

Language returns the configured canonical name or alias.

func (*StemmerTokenFilter) Name 

func (*StemmerTokenFilter) Name() string

type StemmerTokenFilterOptions 

type StemmerTokenFilterOptions struct {
	Language string
}

StemmerTokenFilterOptions configures a Snowball language. An empty language selects the baseline default, english.

func DefaultStemmerTokenFilterOptions 

func DefaultStemmerTokenFilterOptions() StemmerTokenFilterOptions

DefaultStemmerTokenFilterOptions returns the baseline English settings.

func (StemmerTokenFilterOptions) Validate 

func (o StemmerTokenFilterOptions) Validate() error

Validate checks the case-sensitive libstemmer language and alias table.

type Token 

type Token struct {
	Text     string
	Offset   uint32
	Position uint32
}

Token is one owned term emitted by a tokenizer. Offset is the byte offset in the original UTF-8 byte sequence, and Position is the contiguous output sequence number starting at zero.

type TokenFilter 

type TokenFilter interface {
	Name() string
	Filter(ctx context.Context, tokens []Token) ([]Token, error)
}

TokenFilter transforms an ordered token list without changing source offsets or positions.

type Tokenizer 

type Tokenizer interface {
	Name() string
	Tokenize(ctx context.Context, text string) ([]Token, error)
}

Tokenizer converts source text into ordered terms for indexing and query analysis. Implementations must return owned token text and byte offsets.

type WhitespaceTokenizer 

type WhitespaceTokenizer struct{}

WhitespaceTokenizer implements the pinned byte-oriented whitespace split. It recognizes the six ASCII whitespace bytes accepted by C isspace in the C locale and otherwise preserves source bytes verbatim.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/gorse-io/xvec/internal/db/index/column/fts_column/tokenizer"
)

func main() {
	tokenizer := tokenizer.NewWhitespaceTokenizer()
	tokens, err := tokenizer.Tokenize(context.Background(), "  Go\t向量 search")
	if err != nil {
		panic(err)
	}
	for _, token := range tokens {
		fmt.Printf("%d:%d:%s\n", token.Position, token.Offset, token.Text)
	}
}

Output:
0:2:Go
1:5:向量
2:12:search

func NewWhitespaceTokenizer 

func NewWhitespaceTokenizer() *WhitespaceTokenizer

func (*WhitespaceTokenizer) Name 

func (*WhitespaceTokenizer) Name() string

func (*WhitespaceTokenizer) Tokenize 

func (*WhitespaceTokenizer) Tokenize(ctx context.Context, text string) ([]Token, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.