splintr

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

README

splintr-go

Go bindings for splintr through cgo. Default builds include 20 bundled tokenizer variants; FromPretrained needs no downloads. Llama vocabularies are not bundled.

Installation

Requires Go 1.26+, a C compiler, curl and tar. On Linux or macOS, run these commands in your application's Go module:

go get github.com/maxdswain/splintr-go@v0.4.0
archive="splintr-go-v0.4.0-$(go env GOOS)-$(go env GOARCH).tar.gz"
curl -fL --create-dirs -o ".splintr-native/$archive" "https://github.com/maxdswain/splintr-go/releases/download/v0.4.0/$archive"
tar -xzf ".splintr-native/$archive" -C .splintr-native

CGO_ENABLED=1 CGO_LDFLAGS="\"-L$PWD/.splintr-native/lib\" -lsplintr_go $(cat .splintr-native/lib/native-static-libs.txt)" go build -a ./...

See release instructions for checksum verification and Windows setup.

Quick Start

package main

import (
	"fmt"
	"log"

	"github.com/maxdswain/splintr-go"
)

func main() {
	// Load a pretrained vocabulary (OpenAI GPT-4/3.5).
	// Also: deepseek_v3, qwen3, glm4, gpt-oss.
	tok, err := splintr.FromPretrained("cl100k_base")
	if err != nil {
		log.Fatal(err)
	}
	defer tok.Close()

	// Encode and decode.
	ids, err := tok.Encode("Hello, world!")
	if err != nil {
		log.Fatal(err)
	}
	text, err := tok.Decode(ids)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(ids)
	fmt.Println(text) // Hello, world!

	// Batch encode and decode, parallelized natively for larger inputs.
	batch, err := tok.EncodeBatch([]string{"Hello, world!", "How are you?"})
	if err != nil {
		log.Fatal(err)
	}
	texts, err := tok.DecodeBatch(batch)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(batch)
	fmt.Println(texts)
}

Batch results preserve input order and follow the same special-token behavior as Encode and Decode. An empty batch returns an empty slice; an error returns no partial results.

Streaming decoding

Using tok and ids from above:

stream, err := tok.StreamingDecoder()
if err != nil {
	log.Fatal(err)
}
defer stream.Close()

for _, id := range ids {
	text, err := stream.AddToken(id)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Print(text)
}
tail, err := stream.Flush()
if err != nil {
	log.Fatal(err)
}
fmt.Print(tail)

The decoder buffers split UTF-8 characters; Flush emits remaining text, replacing incomplete sequences with �. Reset starts a new sequence, while HasPending and PendingBytes inspect buffered text. Use StreamingDecoderWithSpecialMode(DecodeRenderSpecial) to render special tokens, or AddTokenLossy / AddTokensLossy to skip unknown IDs.

Streams survive closing their tokenizer and serialize concurrent calls. Close discards pending text, so flush first. Unsupported decoder pipelines fail at construction; a strict add error retains any accepted prefix for a later flush or reset. Splintr has no streaming encoder; use Encode or EncodeBatch for complete text.

Hugging Face tokenizer.json
tok, err := splintr.FromFile("tokenizer.json") // or splintr.FromBytes(data)
if err != nil {
	log.Fatal(err)
}
defer tok.Close()
Untrusted text
ids, err := tok.EncodeWithSpecialMode("<|im_start|>user", splintr.SpecialOrdinary)
if err != nil {
	log.Fatal(err)
}
fmt.Println(ids)

SpecialOrdinary treats control-token spellings as text. Both encoding modes still apply post-processing. Tokenizers support concurrent calls and explicit cleanup with Close.

Development

Source builds need Rust, Go, Make and a C compiler. License collection and release packaging also need jq and tar.

make build
make test
make fmt-check lint # rustfmt, Clippy and Revive via golangci-lint v2

License

The Go bindings are licensed under Apache 2.0. Bundled tokenizer vocabularies keep their original licenses, listed in LICENSE-OTHERS.

Documentation

Overview

Package splintr provides local text tokenization through the Rust splintr library. It supports whole-text encoding and decoding, parallel batches, and streaming decoding. See installation instructions for cgo and native-library setup.

Start with FromPretrained for a bundled vocabulary or FromFile for a Hugging Face tokenizer.json. Reuse the resulting Tokenizer and call Tokenizer.Close when done.

Getting started

Default native builds include 20 non-Llama vocabulary variants, including cl100k_base, qwen3, and deepseek_v3. Loading a bundled vocabulary needs no downloads:

tok, err := splintr.FromPretrained("cl100k_base")
if err != nil {
	log.Fatal(err)
}
defer tok.Close()

ids, err := tok.Encode("Hello, world!")
if err != nil {
	log.Fatal(err)
}
fmt.Println(ids) // [9906 11 1917 0]

Use Tokenizer.Decode to turn IDs back into text, or FromBytes to load a JSON document already in memory. Token IDs belong to the vocabulary that produced them; use the same tokenizer to decode them. Normalization and special-token handling mean decoding is not always an exact inverse of encoding. See pretrained vocabularies for supported names and model-specific behavior.

Batch processing

Tokenizer.EncodeBatch and Tokenizer.DecodeBatch preserve input order and use the same defaults as their single-input counterparts. Splintr runs larger batches in parallel. Empty batches return empty slices; errors return no partial results.

Streaming decoding

Create a StreamingDecoder with Tokenizer.StreamingDecoder, then feed IDs with StreamingDecoder.AddToken or StreamingDecoder.AddTokens. Concatenate the returned fragments, which may be empty while UTF-8 characters or decoder operations are pending. Call StreamingDecoder.Flush at the end to emit remaining text; incomplete UTF-8 is replaced with U+FFFD. Flush does not close or reset the stream.

Unknown IDs return an error; any prefix accepted by that call stays buffered. StreamingDecoder.AddTokensLossy skips unknown IDs instead. StreamingDecoder.Reset discards pending text and starts a new sequence. Some tokenizer.json decoder pipelines cannot stream and fail at construction. Encoding requires complete input text; there is no streaming encoder.

Special tokens

Tokenizer.Encode recognizes configured special-token spellings using SpecialAll. For untrusted text, use Tokenizer.EncodeWithSpecialMode with SpecialOrdinary. Both modes still apply the tokenizer's post-processor, which may insert special IDs.

Tokenizer.Decode and Tokenizer.StreamingDecoder skip special tokens by default. Use their WithSpecialMode variants with DecodeRenderSpecial to include them. Ordinary decoding may omit unknown IDs in declared decoder pipelines; strict streaming adds reject them. Tokenizer.SpecialTokenID and Tokenizer.EOSTokenID provide special-token metadata.

Ownership and concurrency

Tokenizer methods support concurrent calls. Stream methods serialize access to one cursor; callers must coordinate token order when sharing a stream. Streams own their state and remain usable after their tokenizer closes.

Copies of either handle share ownership, so closing one closes all its copies. Close waits for active calls and is idempotent. StreamingDecoder.Close discards pending text; flush first if it is needed. Returned slices and strings are Go-owned and remain valid after closing the handle.

Errors

Operations on nil, zero-value, or closed handles return ErrClosed; Close remains safe to call. Use errors.Is to check this sentinel and errors.AsType to inspect a native Error and its ErrorCode. FromFile also returns filesystem errors.

Native library

Building requires Go 1.26+, cgo, a C compiler, and a matching native static library. The go get command downloads the Go module, not the native library. See installation instructions for prebuilt archives or source builds for Rust builds. Bundled vocabularies have separate license terms.

Example

This example uses a small JSON vocabulary and needs no bundled vocabularies.

package main

import (
	"fmt"

	"github.com/maxdswain/splintr-go"
)

const tokenizerJSON = `{
	"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
	"pre_tokenizer": {"type": "Whitespace"},
	"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
	"model": {"type": "WordPiece", "unk_token": "[UNK]",
		"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
		"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`

func exampleTokenizer() *splintr.Tokenizer {
	tok, err := splintr.FromBytes([]byte(tokenizerJSON))
	if err != nil {
		panic(err)
	}
	return tok
}

func main() {
	tok := exampleTokenizer()
	defer tok.Close()

	ids, err := tok.Encode("hello world")
	if err != nil {
		panic(err)
	}
	text, err := tok.Decode(ids)
	if err != nil {
		panic(err)
	}
	fmt.Println(ids)
	fmt.Println(text)
}
Output:
[1 2]
hello world

Index

Examples

Constants

View Source
const (
	CodeInvalidArgument = ErrorInvalidArgument
	CodeInvalidUTF8     = ErrorInvalidUTF8
	CodeTokenizer       = ErrorTokenizer
	CodePanic           = ErrorPanic
)

Error-code aliases.

Variables

View Source
var ErrClosed = errors.New("splintr: tokenizer is closed")

ErrClosed is returned when an operation is attempted on a closed, nil, or zero-value Tokenizer or StreamingDecoder.

Functions

This section is empty.

Types

type DecodeSpecialMode

type DecodeSpecialMode int

DecodeSpecialMode controls how declared special token IDs are decoded.

const (
	// DecodeSkipSpecial omits declared special tokens from decoded text.
	DecodeSkipSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_SKIP_SPECIAL)
	// DecodeRenderSpecial renders declared special tokens in decoded text.
	DecodeRenderSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_RENDER_SPECIAL)
)

type EncodeSpecialMode

type EncodeSpecialMode int

EncodeSpecialMode controls recognition of special-token spellings.

const (
	// SpecialOrdinary treats every special-token spelling as ordinary input.
	SpecialOrdinary EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ORDINARY)
	// SpecialAll recognizes every configured special token in the input.
	SpecialAll EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ALL)
)

type Error

type Error struct {
	Code    ErrorCode // Failure category.
	Message string    // Native detail, without the "splintr:" prefix.
}

Error describes a native failure. Use errors.AsType to inspect its Code.

Example
package main

import (
	"errors"
	"fmt"

	"github.com/maxdswain/splintr-go"
)

func main() {
	_, err := splintr.FromBytes(nil)
	if nativeErr, ok := errors.AsType[*splintr.Error](err); ok {
		fmt.Println("invalid argument:", nativeErr.Code == splintr.ErrorInvalidArgument)
	}

	var tok splintr.Tokenizer
	_, err = tok.Encode("hello")
	fmt.Println("closed:", errors.Is(err, splintr.ErrClosed))
}
Output:
invalid argument: true
closed: true

func (*Error) Error

func (e *Error) Error() string

Error formats the native message or error code.

type ErrorCode

type ErrorCode int

ErrorCode identifies an error status returned by the native library.

const (
	// ErrorInvalidArgument means an argument did not satisfy the ABI contract.
	ErrorInvalidArgument ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_ARGUMENT)
	// ErrorInvalidUTF8 means a text argument was not valid UTF-8.
	ErrorInvalidUTF8 ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_UTF8)
	// ErrorTokenizer means tokenizer parsing or an encode/decode operation failed.
	ErrorTokenizer ErrorCode = ErrorCode(C.SPLINTR_ERROR_TOKENIZER)
	// ErrorPanic means the native library caught a Rust panic.
	ErrorPanic ErrorCode = ErrorCode(C.SPLINTR_ERROR_PANIC)
)

type StreamingDecoder added in v0.3.0

type StreamingDecoder struct {
	// contains filtered or unexported fields
}

StreamingDecoder incrementally decodes IDs. Calls are serialized; its zero value is closed. Copies share one cursor and lifetime.

Example
package main

import (
	"fmt"

	"github.com/maxdswain/splintr-go"
)

const tokenizerJSON = `{
	"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
	"pre_tokenizer": {"type": "Whitespace"},
	"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
	"model": {"type": "WordPiece", "unk_token": "[UNK]",
		"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
		"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`

func exampleTokenizer() *splintr.Tokenizer {
	tok, err := splintr.FromBytes([]byte(tokenizerJSON))
	if err != nil {
		panic(err)
	}
	return tok
}

func main() {
	tok := exampleTokenizer()
	defer tok.Close()

	stream, err := tok.StreamingDecoder()
	if err != nil {
		panic(err)
	}
	defer stream.Close()

	for _, id := range []uint32{1, 2} {
		text, err := stream.AddToken(id)
		if err != nil {
			panic(err)
		}
		fmt.Print(text)
	}
	tail, err := stream.Flush()
	if err != nil {
		panic(err)
	}
	fmt.Println(tail)
}
Output:
hello world

func (*StreamingDecoder) AddToken added in v0.3.0

func (d *StreamingDecoder) AddToken(id uint32) (string, error)

AddToken decodes one ID; unknown IDs return an error.

func (*StreamingDecoder) AddTokenLossy added in v0.3.0

func (d *StreamingDecoder) AddTokenLossy(id uint32) (string, error)

AddTokenLossy decodes one ID, skipping unknown IDs.

func (*StreamingDecoder) AddTokens added in v0.3.0

func (d *StreamingDecoder) AddTokens(ids []uint32) (string, error)

AddTokens emits available text, possibly empty. Unknown IDs return an error; any accepted prefix remains buffered.

func (*StreamingDecoder) AddTokensLossy added in v0.3.0

func (d *StreamingDecoder) AddTokensLossy(ids []uint32) (string, error)

AddTokensLossy is StreamingDecoder.AddTokens with unknown IDs skipped.

func (*StreamingDecoder) Close added in v0.3.0

func (d *StreamingDecoder) Close() error

Close waits for active calls and discards pending text. It is idempotent.

func (*StreamingDecoder) Flush added in v0.3.0

func (d *StreamingDecoder) Flush() (string, error)

Flush drains buffered text, replacing incomplete UTF-8 with U+FFFD. It neither resets nor closes the decoder; use StreamingDecoder.Reset for a new sequence.

func (*StreamingDecoder) HasPending added in v0.3.0

func (d *StreamingDecoder) HasPending() (bool, error)

HasPending reports whether any buffered text remains.

func (*StreamingDecoder) PendingBytes added in v0.3.0

func (d *StreamingDecoder) PendingBytes() (int, error)

PendingBytes counts bytes awaiting emission, including post-processing buffers.

func (*StreamingDecoder) Reset added in v0.3.0

func (d *StreamingDecoder) Reset() error

Reset discards pending text and starts a new sequence.

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer is a native tokenizer handle. Its zero value is closed.

Tokenizer values may be copied: copies share ownership and closing one makes every copy closed. Methods may be called concurrently with each other and with Tokenizer.Close; Close waits for calls already in progress.

func FromBytes

func FromBytes(data []byte) (*Tokenizer, error)

FromBytes loads a Hugging Face tokenizer.json document. The data must be non-empty. Loading is local and does not retain data.

func FromFile

func FromFile(path string) (*Tokenizer, error)

FromFile reads a local tokenizer.json file and loads it with FromBytes.

func FromPretrained

func FromPretrained(name string) (*Tokenizer, error)

FromPretrained loads a bundled vocabulary by name or alias, without downloads. Available vocabularies depend on the native library's compiled features.

func (*Tokenizer) Close

func (t *Tokenizer) Close() error

Close waits for active calls and releases the shared handle. It is idempotent.

func (*Tokenizer) Decode

func (t *Tokenizer) Decode(ids []uint32) (string, error)

Decode decodes IDs with DecodeSkipSpecial. It follows splintr's behavior for unknown IDs: declared decoder pipelines may omit them, while other decoding paths return an error.

func (*Tokenizer) DecodeBatch added in v0.2.0

func (t *Tokenizer) DecodeBatch(tokenLists [][]uint32) ([]string, error)

DecodeBatch decodes token lists in input order using splintr's native batch processing. Special tokens and unknown IDs behave as in Tokenizer.Decode. Empty batches return an empty slice; any error returns nil rather than partial results.

func (*Tokenizer) DecodeWithSpecialMode

func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)

DecodeWithSpecialMode decodes IDs while either skipping or rendering declared special tokens.

func (*Tokenizer) EOSTokenID

func (t *Tokenizer) EOSTokenID() (uint32, bool, error)

EOSTokenID returns the configured end-of-sequence token ID. The boolean is false when no EOS token is configured.

func (*Tokenizer) Encode

func (t *Tokenizer) Encode(text string) ([]uint32, error)

Encode encodes text with SpecialAll and applies post-processing, which may insert special IDs even for empty text.

func (*Tokenizer) EncodeBatch added in v0.2.0

func (t *Tokenizer) EncodeBatch(texts []string) ([][]uint32, error)

EncodeBatch encodes texts in input order using splintr's native batch processing. It has the same special-token and post-processing behavior as Tokenizer.Encode. Empty batches return an empty slice; any error returns nil rather than partial results.

Example
package main

import (
	"fmt"

	"github.com/maxdswain/splintr-go"
)

const tokenizerJSON = `{
	"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
	"pre_tokenizer": {"type": "Whitespace"},
	"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
	"model": {"type": "WordPiece", "unk_token": "[UNK]",
		"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
		"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`

func exampleTokenizer() *splintr.Tokenizer {
	tok, err := splintr.FromBytes([]byte(tokenizerJSON))
	if err != nil {
		panic(err)
	}
	return tok
}

func main() {
	tok := exampleTokenizer()
	defer tok.Close()

	batch, err := tok.EncodeBatch([]string{"hello world", "world", ""})
	if err != nil {
		panic(err)
	}
	texts, err := tok.DecodeBatch(batch)
	if err != nil {
		panic(err)
	}
	fmt.Println(batch)
	fmt.Printf("%q\n", texts)
}
Output:
[[1 2] [2] []]
["hello world" "world" ""]

func (*Tokenizer) EncodeWithSpecialMode

func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)

EncodeWithSpecialMode controls special-token recognition; use SpecialOrdinary for untrusted text. Both modes apply the tokenizer.json single-sequence post-processor.

Example
package main

import (
	"fmt"
	"slices"

	"github.com/maxdswain/splintr-go"
)

const tokenizerJSON = `{
	"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
	"pre_tokenizer": {"type": "Whitespace"},
	"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
	"model": {"type": "WordPiece", "unk_token": "[UNK]",
		"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
		"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`

func exampleTokenizer() *splintr.Tokenizer {
	tok, err := splintr.FromBytes([]byte(tokenizerJSON))
	if err != nil {
		panic(err)
	}
	return tok
}

func main() {
	tok := exampleTokenizer()
	defer tok.Close()

	id, found, err := tok.SpecialTokenID("[SEP]")
	if err != nil {
		panic(err)
	}
	if !found {
		panic("missing special token")
	}
	special, err := tok.EncodeWithSpecialMode("hello [SEP]", splintr.SpecialAll)
	if err != nil {
		panic(err)
	}
	ordinary, err := tok.EncodeWithSpecialMode("hello [SEP]", splintr.SpecialOrdinary)
	if err != nil {
		panic(err)
	}
	fmt.Println("recognized:", slices.Contains(special, id))
	fmt.Println("ordinary:", slices.Contains(ordinary, id))

	text, err := tok.DecodeWithSpecialMode(special, splintr.DecodeRenderSpecial)
	if err != nil {
		panic(err)
	}
	fmt.Println(text)
}
Output:
recognized: true
ordinary: false
hello [SEP]

func (*Tokenizer) Family

func (t *Tokenizer) Family() (string, error)

Family returns the model family: "BPE", "Unigram", "WordPiece", "Spm", or "Unknown".

func (*Tokenizer) SpecialTokenID

func (t *Tokenizer) SpecialTokenID(token string) (uint32, bool, error)

SpecialTokenID looks up a declared special token by its exact content. The boolean is false if token is not declared special.

func (*Tokenizer) StreamingDecoder added in v0.3.0

func (t *Tokenizer) StreamingDecoder() (*StreamingDecoder, error)

StreamingDecoder creates a decoder with DecodeSkipSpecial that outlives its tokenizer.

func (*Tokenizer) StreamingDecoderWithSpecialMode added in v0.3.0

func (t *Tokenizer) StreamingDecoderWithSpecialMode(mode DecodeSpecialMode) (*StreamingDecoder, error)

StreamingDecoderWithSpecialMode sets special-token handling in a new decoder. Unsupported pipelines return an error.

func (*Tokenizer) VocabSize

func (t *Tokenizer) VocabSize() (int, error)

VocabSize returns the number of entries in the tokenizer vocabulary.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL