Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
- API
  
  Reference documentation for the pkg.go.dev API
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

ollamatokenizer

package module

Go to main page

Versions in this module

v0

v0.31.2

Jul 11, 2026

Changes in this version

+ type ChatMessage struct

+ Content string

+ Role string

type Tokenizer

+ func (t *Tokenizer) Close()

v0.20.7

Apr 14, 2026

Changes in this version

+ var ErrNotImplemented = fmt.Errorf("not implemented")

+ type Tokenizer struct

+ func New(name string) (*Tokenizer, error)

+ func (t *Tokenizer) Tokenize(text string, addSpecial, parseSpecial bool) ([]int32, error)

+ func (t *Tokenizer) TokenizeChat(req api.ChatRequest) ([]int32, error)

+ func (t *Tokenizer) TokenizeGenerate(req api.GenerateRequest) ([]int32, error)