Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
- API
  
  Reference documentation for the pkg.go.dev API
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

tiktoken

package module

Go to main page

Versions in this module

v0

v0.2.4

Apr 22, 2025

v0.2.3

Apr 4, 2025

v0.2.2

Apr 4, 2025

v0.2.0

May 20, 2024

Changes in this version

+ const ENDOFPROMPT

+ const ENDOFTEXT

+ const FIM_MIDDLE

+ const FIM_PREFIX

+ const FIM_SUFFIX

+ const MODEL_CL100K_BASE

+ const MODEL_O200K_BASE

+ const MODEL_P50K_BASE

+ const MODEL_P50K_EDIT

+ const MODEL_R50K_BASE

+ var MODEL_PREFIX_TO_ENCODING = map[string]string

+ var MODEL_TO_ENCODING = map[string]string

+ func SetBpeLoader(loader BpeLoader)

+ type BpeLoader interface

+ LoadTiktokenBpe func(tiktokenBpeFile string) (map[string]int, error)

+ func NewDefaultBpeLoader() BpeLoader

+ type CoreBPE struct

+ func NewCoreBPE(encoder map[string]int, specialTokensEncoder map[string]int, pattern string) (*CoreBPE, error)

+ type Encoding struct

+ ExplicitNVocab int

+ MergeableRanks map[string]int

+ Name string

+ PatStr string

+ SpecialTokens map[string]int

+ type Tiktoken struct

+ func EncodingForModel(modelName string) (*Tiktoken, error)

+ func GetEncoding(encodingName string) (*Tiktoken, error)

+ func NewTiktoken(bpe *CoreBPE, encoding *Encoding, specialTokensSet map[string]any) *Tiktoken

+ func (t *Tiktoken) Decode(tokens []int) string

+ func (t *Tiktoken) Encode(text string, allowedSpecial []string, disallowedSpecial []string) []int

+ func (t *Tiktoken) EncodeOrdinary(text string) []int

+ func (t *Tiktoken) EncoderName() string

+ func (t *Tiktoken) SpecialTokenRegex(disallowedSpecialSet map[string]any) *regexp2.Regexp