sentencepiece

Documentation

Index

• type Tokenizer
  • func NewFromModelFolder(path string, lowercase bool) (*Tokenizer, error)
  • func (t *Tokenizer) Detokenize(tokens []string) string
  • func (t *Tokenizer) IDsToTokens(ids []int) []string
  • func (t *Tokenizer) Tokenize(text string) []string
  • func (t *Tokenizer) TokensToIDs(tokens []string) []int

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer is a Sentence Piece tokenizer.

func NewFromModelFolder

func NewFromModelFolder(path string, lowercase bool) (*Tokenizer, error)

NewFromModelFolder returns a new Tokenizer.

func (*Tokenizer) Detokenize

func (t *Tokenizer) Detokenize(tokens []string) string

Detokenize flatten and merges a list of tokens into a single string.

func (*Tokenizer) IDsToTokens

func (t *Tokenizer) IDsToTokens(ids []int) []string

IDsToTokens returns a list of string terms from a list of token IDs. It panics if a token is not found in the vocabulary.

func (*Tokenizer) Tokenize

func (t *Tokenizer) Tokenize(text string) []string

Tokenize performs sentence-piece tokenization.

func (*Tokenizer) TokensToIDs

func (t *Tokenizer) TokensToIDs(tokens []string) []int

TokensToIDs returns a list of token IDs from a list of string tokens. It panics if a token is not found in the vocabulary and no unknown token is found.