splintr

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 5, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

README

splintr-go

Go bindings for splintr through cgo. Default builds include all 23 bundled tokenizer variants; FromPretrained needs no downloads.

Installation

Requires Go 1.26+ and a C compiler (MinGW-w64 on Windows). Download and extract the matching native archive from Releases, then:

go get github.com/maxdswain/splintr-go@v0.1.0
SPLINTR_LIB=/path/to/extracted/lib
export CGO_LDFLAGS="\"-L$SPLINTR_LIB\" -lsplintr_go $(cat "$SPLINTR_LIB/native-static-libs.txt")"
CGO_ENABLED=1 go build -a ./...

Quick Start

package main

import (
	"fmt"
	"log"

	"github.com/maxdswain/splintr-go"
)

func main() {
	// Load a pretrained vocabulary (OpenAI GPT-4/3.5).
	// Also: llama3, deepseek_v3, qwen3, glm4, gpt-oss.
	tok, err := splintr.FromPretrained("cl100k_base")
	if err != nil {
		log.Fatal(err)
	}
	defer tok.Close()

	// Encode and decode.
	ids, err := tok.Encode("Hello, world!")
	if err != nil {
		log.Fatal(err)
	}
	text, err := tok.Decode(ids)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(ids)
	fmt.Println(text) // Hello, world!

	// Encode multiple texts, one call per text.
	texts := []string{"Hello, world!", "How are you?"}
	batch := make([][]uint32, len(texts))
	for i, text := range texts {
		batch[i], err = tok.Encode(text)
		if err != nil {
			log.Fatal(err)
		}
	}
	fmt.Println(batch)
}
Hugging Face tokenizer.json
tok, err := splintr.FromFile("tokenizer.json") // or splintr.FromBytes(data)
if err != nil {
	log.Fatal(err)
}
defer tok.Close()
Untrusted text
ids, err := tok.EncodeWithSpecialMode("<|im_start|>user", splintr.SpecialOrdinary)
if err != nil {
	log.Fatal(err)
}
fmt.Println(ids)

SpecialOrdinary treats control-token spellings as text. Both encoding modes still apply post-processing. Tokenizers support concurrent calls and explicit cleanup with Close.

Development

Source builds need Rust, Go, Make and a C compiler. License collection and release packaging also need jq and tar.

make build
make test
make fmt-check lint # rustfmt, Clippy and Revive via golangci-lint v2

License

The Go bindings are licensed under Apache 2.0. Bundled tokenizer vocabularies keep their original licenses, listed in LICENSE-OTHERS.

Llama 3 vocabulary attribution (required by Meta’s license): “Built with Llama”.

Documentation

Overview

Package splintr provides Go bindings to the Rust splintr tokenizer.

Index

Constants

View Source
const (
	CodeInvalidArgument = ErrorInvalidArgument
	CodeInvalidUTF8     = ErrorInvalidUTF8
	CodeTokenizer       = ErrorTokenizer
	CodePanic           = ErrorPanic
)

Error-code aliases.

Variables

View Source
var ErrClosed = errors.New("splintr: tokenizer is closed")

ErrClosed is returned when an operation is attempted on a closed, nil, or zero-value Tokenizer.

Functions

This section is empty.

Types

type DecodeSpecialMode

type DecodeSpecialMode int

DecodeSpecialMode controls how declared special token IDs are decoded.

const (
	// DecodeSkipSpecial omits declared special tokens from decoded text.
	DecodeSkipSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_SKIP_SPECIAL)
	// DecodeRenderSpecial renders declared special tokens in decoded text.
	DecodeRenderSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_RENDER_SPECIAL)
)

type EncodeSpecialMode

type EncodeSpecialMode int

EncodeSpecialMode controls recognition of special-token spellings.

const (
	// SpecialOrdinary treats every special-token spelling as ordinary input.
	SpecialOrdinary EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ORDINARY)
	// SpecialAll recognizes every configured special token in the input.
	SpecialAll EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ALL)
)

type Error

type Error struct {
	Code    ErrorCode
	Message string
}

Error is an error reported by the native splintr library.

func (*Error) Error

func (e *Error) Error() string

type ErrorCode

type ErrorCode int

ErrorCode identifies an error status returned by the native library.

const (
	// ErrorInvalidArgument means an argument did not satisfy the ABI contract.
	ErrorInvalidArgument ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_ARGUMENT)
	// ErrorInvalidUTF8 means a text argument was not valid UTF-8.
	ErrorInvalidUTF8 ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_UTF8)
	// ErrorTokenizer means tokenizer parsing or an encode/decode operation failed.
	ErrorTokenizer ErrorCode = ErrorCode(C.SPLINTR_ERROR_TOKENIZER)
	// ErrorPanic means the native library caught a Rust panic.
	ErrorPanic ErrorCode = ErrorCode(C.SPLINTR_ERROR_PANIC)
)

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer is a native tokenizer handle. Its zero value is closed.

Tokenizer values may be copied: copies share ownership and closing one makes every copy closed. Methods may be called concurrently with each other and with Close; Close waits for calls already in progress.

func FromBytes

func FromBytes(data []byte) (*Tokenizer, error)

FromBytes loads a Hugging Face tokenizer.json document. The data must be non-empty. Loading is local and does not retain data.

func FromFile

func FromFile(path string) (*Tokenizer, error)

FromFile reads and loads a Hugging Face tokenizer.json file.

func FromPretrained

func FromPretrained(name string) (*Tokenizer, error)

FromPretrained loads a bundled vocabulary by name or alias, without downloads.

func (*Tokenizer) Close

func (t *Tokenizer) Close() error

Close waits for active calls and releases the shared handle. It is idempotent.

func (*Tokenizer) Decode

func (t *Tokenizer) Decode(ids []uint32) (string, error)

Decode decodes IDs while skipping declared special tokens. It follows splintr's behavior for unknown IDs: declared decoder pipelines may omit them, while other decoding paths return an error.

func (*Tokenizer) DecodeWithSpecialMode

func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)

DecodeWithSpecialMode decodes IDs while either skipping or rendering declared special tokens.

func (*Tokenizer) EOSTokenID

func (t *Tokenizer) EOSTokenID() (uint32, bool, error)

EOSTokenID returns the configured end-of-sequence token ID. The boolean is false when no EOS token is configured.

func (*Tokenizer) Encode

func (t *Tokenizer) Encode(text string) ([]uint32, error)

Encode recognizes all configured special tokens and applies post-processing, which may insert special IDs even for empty text.

func (*Tokenizer) EncodeWithSpecialMode

func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)

EncodeWithSpecialMode encodes text with explicit special-token recognition. The tokenizer.json single-sequence post-processor is applied in both modes.

func (*Tokenizer) Family

func (t *Tokenizer) Family() (string, error)

Family returns the model family: "BPE", "Unigram", "WordPiece", "Spm", or "Unknown".

func (*Tokenizer) SpecialTokenID

func (t *Tokenizer) SpecialTokenID(token string) (uint32, bool, error)

SpecialTokenID looks up a declared special token by its exact content. The boolean is false if token is not declared special.

func (*Tokenizer) VocabSize

func (t *Tokenizer) VocabSize() (int, error)

VocabSize returns the number of entries in the tokenizer vocabulary.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL