sentencepiece

Documentation

Overview

Package sentencepiece implements a tokenizers.Tokenizer based on SentencePiece tokenizer.

Index

• func New(config *api.Config, repo *hub.Repo) (api.Tokenizer, error)
• type Tokenizer
  • func (p *Tokenizer) Decode(ids []int) string
  • func (p *Tokenizer) Encode(text string) []int
  • func (p *Tokenizer) SpecialTokenID(token api.SpecialToken) (int, error)

Functions

func New

func New(config *api.Config, repo *hub.Repo) (api.Tokenizer, error)

New creates a SentencePiece tokenizer based on the "tokenizer.model" file, which must be a SentencePiece Model proto (see protos.Model).

It implements a tokenizer.TokenizerConstructor function signature.

Types

type Tokenizer

type Tokenizer struct {
	*esentencepiece.Processor
	Info *esentencepiece.ModelInfo
}

Tokenizer implements tokenizers.Tokenizer interface based on SentencePiece tokenizer by Google.

func (*Tokenizer) Decode

func (p *Tokenizer) Decode(ids []int) string

Decode returns the text from a sequence of ids. It implements sampler.Vocabulary.

func (*Tokenizer) Encode

func (p *Tokenizer) Encode(text string) []int

Encode returns the text encoded into a sequence of ids. It implements sampler.Vocabulary.

func (*Tokenizer) SpecialTokenID

func (p *Tokenizer) SpecialTokenID(token api.SpecialToken) (int, error)

SpecialTokenID returns the token for the given symbol, or an error if not known.