ollamatokenizer

README

ollamatokenizer

HTTP server exposing Ollama's internal tokenization as API endpoints.

Token IDs are byte-identical to a running Ollama server: the service links against Ollama's bundled libllama.so and loads each model's GGUF vocabulary vocab-only (llama_model_load with vocab_only=true).

Build & Run

make build
OLLAMA_MODELS=/var/lib/ollama ./bin/ollamatokenizer serve

Options: -p, --port (default: 11435)

Docker

make docker-build
docker run -p 11435:11435 -v /var/lib/ollama:/ollama-models:ro ollamatokenizer

Endpoints

GET /health

Returns {"status": "ok"}.

POST /tokenize

Raw tokenization - no chat template or system prompt.

{"model": "llama3.2:3b", "text": "Why is the sky blue?"}

POST /tokenize/generate

Mirrors /api/generate. Applies chat template, system prompt, thinking.

{"model": "llama3.2:3b", "prompt": "Why is the sky blue?", "system": "You are a helpful assistant.", "think": true}

POST /tokenize/chat

Mirrors /api/chat. Applies chat template, tools, thinking.

{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Why is the sky blue?"}], "tools": [], "think": true}

Responses

{"tokens": [1, 2998, 338, 278, 6507, 18561, 29973], "count": 7}

{"error": "description"}

Status	Cause
400	Missing or invalid request body, model not found
501	Unsupported option (suffix, template override, raw mode, context, images)
500	Tokenization error

Documentation

Overview

Package ollamatokenizer links ollama's bundled libllama.so and uses llama.cpp's real tokenizer with vocab-only model loading. The llama.cpp version must match the ollama build (ABI: llama_model_params); pin via go.mod → LLAMA_CPP_VERSION. Build inputs (populated by `make fetch-deps`) live under llama-cpp/:

• include/llama.h, ggml/include/*.h (headers)
• lib/lib{llama,ggml,ggml-base}.so (from the ollama install)

Package ollamatokenizer exposes ollama's tokenization without loading model weights, by linking ollama's bundled libllama.so and loading each GGUF vocab-only. Token IDs are byte-identical to a running ollama server.

Index

• Variables
• type ChatMessage
• type Tokenizer
  • func New(name string) (*Tokenizer, error)
  • func (t *Tokenizer) Close()
  • func (t *Tokenizer) Tokenize(text string, addSpecial, parseSpecial bool) ([]int32, error)
  • func (t *Tokenizer) TokenizeChat(req api.ChatRequest) ([]int32, error)
  • func (t *Tokenizer) TokenizeGenerate(req api.GenerateRequest) ([]int32, error)

Variables

var ErrNotImplemented = fmt.Errorf("not implemented")

Types

type ChatMessage

type ChatMessage struct {
	Role    string
	Content string
}

ChatMessage is a {role, content} pair for RenderChatJinja.

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer holds a vocab-only GGUF handle and ollama model metadata.

func New

func New(name string) (*Tokenizer, error)

New loads a model's vocab-only GGUF via cgo and ollama metadata via server.GetModel. https://github.com/ollama/ollama/blob/v0.31.2/server/images.go#L641

func (*Tokenizer) Close

func (t *Tokenizer) Close()

func (*Tokenizer) Tokenize

func (t *Tokenizer) Tokenize(text string, addSpecial, parseSpecial bool) ([]int32, error)

Tokenize encodes text via llama_tokenize. addSpecial applies BOS/EOS per the vocab; parseSpecial parses special-token strings (e.g. <|im_start|>).

func (*Tokenizer) TokenizeChat

func (t *Tokenizer) TokenizeChat(req api.ChatRequest) ([]int32, error)

TokenizeChat mirrors /api/chat's prompt assembly (no context truncation). https://github.com/ollama/ollama/blob/v0.31.2/server/routes.go#L2661-L2705

func (*Tokenizer) TokenizeGenerate

func (t *Tokenizer) TokenizeGenerate(req api.GenerateRequest) ([]int32, error)

TokenizeGenerate mirrors /api/generate's prompt assembly (no context truncation). Unsupported (ErrNotImplemented): Suffix, Template, Raw, Context, Images. https://github.com/ollama/ollama/blob/v0.31.2/server/routes.go#L510-L620