bpe

Documentation

Overview

Package bpe implements Byte Pair Encoding tokenization.

Index

• Constants
• type BPETokenizer
  • func NewEncoderByName(encodingName string) (*BPETokenizer, error)
  • func (tok *BPETokenizer) Decode(tokens []int) string
  • func (tok *BPETokenizer) Encode(text string, allowedSpecial []string, disallowedSpecial []string) ([]int, error)
  • func (tok *BPETokenizer) EncodeOrdinary(text string) []int
• type Definition
• type Encoder
  • func NewEncoder(encoder map[string]int, specialTokensEncoder map[string]int, pattern string) (*Encoder, error)

Constants

const (
	EndOfText   = "<|endoftext|>"
	FIMPrefix   = "<|fim_prefix|>"
	FIMMiddle   = "<|fim_middle|>"
	FIMSuffix   = "<|fim_suffix|>"
	EndOfPrompt = "<|endofprompt|>"
)

Special token constants.

const (
	EncodingO200kBase  = "o200k_base"
	EncodingCL100kBase = "cl100k_base"
	EncodingP50kBase   = "p50k_base"
	EncodingP50kEdit   = "p50k_edit"
	EncodingR50kBase   = "r50k_base"
)

Encoding names.

Types

type BPETokenizer

type BPETokenizer struct {
	// contains filtered or unexported fields
}

BPETokenizer is the main tokenizer that wraps a BPE Encoder.

func NewEncoderByName

func NewEncoderByName(encodingName string) (*BPETokenizer, error)

NewEncoderByName returns a BPETokenizer for the named encoding.

func (*BPETokenizer) Decode

func (tok *BPETokenizer) Decode(tokens []int) string

Decode converts token IDs back to text.

func (*BPETokenizer) Encode

func (tok *BPETokenizer) Encode(text string, allowedSpecial []string, disallowedSpecial []string) ([]int, error)

Encode tokenizes text with optional special token handling. Returns an error if text contains a disallowed special token.

func (*BPETokenizer) EncodeOrdinary

func (tok *BPETokenizer) EncodeOrdinary(text string) []int

EncodeOrdinary tokenizes text without special token handling.

type Definition

type Definition struct {
	Name           string
	PatStr         string
	MergeableRanks map[string]int
	SpecialTokens  map[string]int
	ExplicitNVocab int
}

Definition holds the specification for a BPE encoding scheme.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder is the core BPE encoder/decoder.

func NewEncoder

func NewEncoder(encoder map[string]int, specialTokensEncoder map[string]int, pattern string) (*Encoder, error)

NewEncoder creates a new Encoder from encoder maps and a regex pattern.