ollamatokenizer

README

ollamatokenizer

HTTP server exposing Ollama's internal tokenization as API endpoints.

Build & Run

make build
OLLAMA_MODELS=/var/lib/ollama ollamatokenizer serve

Options: -p, --port (default: 11435)

Docker

docker build -t ollamatokenizer .
docker run -p 11435:11435 -v /var/lib/ollama:/ollama-models:ro ollamatokenizer

Endpoints

GET /health

Returns {"status": "ok"}.

POST /tokenize

Raw tokenization — no chat template or system prompt.

{"model": "llama3.2:3b", "text": "Why is the sky blue?"}

POST /tokenize/generate

Mirrors /api/generate. Applies chat template, system prompt, thinking.

{"model": "llama3.2:3b", "prompt": "Why is the sky blue?", "system": "You are a helpful assistant.", "think": true}

POST /tokenize/chat

Mirrors /api/chat. Applies chat template, tools, thinking.

{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Why is the sky blue?"}], "tools": [], "think": true}

Responses

{"tokens": [1, 2998, 338, 278, 6507, 18561, 29973], "count": 7}

{"error": "description"}

Status	Cause
400	Missing or invalid request body, model not found
501	Unsupported option (suffix, template override, raw mode, context, images)
500	Tokenization error

Documentation

Overview

Package ollamatokenizer provides a simple interface to ollama's internal tokenization functionality.

Index

• Variables
• type Tokenizer
  • func New(name string) (*Tokenizer, error)
  • func (t *Tokenizer) Tokenize(text string, addSpecial, parseSpecial bool) ([]int32, error)
  • func (t *Tokenizer) TokenizeChat(req api.ChatRequest) ([]int32, error)
  • func (t *Tokenizer) TokenizeGenerate(req api.GenerateRequest) ([]int32, error)

Variables

var ErrNotImplemented = fmt.Errorf("not implemented")

ErrNotImplemented is returned when a request uses options that this library does not support.

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer wraps an Ollama tokenizer for a specific model.

func New

func New(name string) (*Tokenizer, error)

New creates a Tokenizer for the given model name (e.g. "llama3.2:3b"). The model must have been pulled via `ollama pull`. https://github.com/ollama/ollama/blob/v0.20.7/llm/server.go#L144-L164

func (*Tokenizer) Tokenize

func (t *Tokenizer) Tokenize(text string, addSpecial, parseSpecial bool) ([]int32, error)

Tokenize encodes text into token IDs without applying any chat template. addSpecial: prepend BOS/append EOS if the model's vocab requires it (AddBOS/AddEOS GGUF metadata). parseSpecial: parse special token strings in text (e.g. <|im_start|>) into token IDs. https://github.com/ollama/ollama/blob/v0.20.7/runner/ollamarunner/runner.go#L246 https://github.com/ollama/ollama/blob/v0.20.7/runner/llamarunner/runner.go#L211

func (*Tokenizer) TokenizeChat

func (t *Tokenizer) TokenizeChat(req api.ChatRequest) ([]int32, error)

TokenizeChat tokenizes messages matching /api/chat. Note: the server's chatPrompt performs context-length truncation that we do not replicate; for prompts within the context window, the rendered output is identical.

https://github.com/ollama/ollama/blob/v0.20.7/server/routes.go#L2276-L2347

func (*Tokenizer) TokenizeGenerate

func (t *Tokenizer) TokenizeGenerate(req api.GenerateRequest) ([]int32, error)

TokenizeGenerate tokenizes a prompt matching /api/generate. Note: the server's chatPrompt performs context-length truncation that we do not replicate; for prompts within the context window, the rendered output is identical.

Unsupported options (return ErrNotImplemented): Suffix, Template, Raw, Context, Images.

https://github.com/ollama/ollama/blob/v0.20.7/server/routes.go#L456-L503