splintr

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

README

splintr-go

Go bindings for splintr through cgo. Default builds include all 23 bundled tokenizer variants; FromPretrained needs no downloads.

Installation

Requires Go 1.26+, a C compiler, curl and tar. On Linux or macOS, run these commands in your application's Go module:

go get github.com/maxdswain/splintr-go@v0.3.0
archive="splintr-go-v0.3.0-$(go env GOOS)-$(go env GOARCH).tar.gz"
curl -fL --create-dirs -o ".splintr-native/$archive" "https://github.com/maxdswain/splintr-go/releases/download/v0.3.0/$archive"
tar -xzf ".splintr-native/$archive" -C .splintr-native

CGO_ENABLED=1 CGO_LDFLAGS="\"-L$PWD/.splintr-native/lib\" -lsplintr_go $(cat .splintr-native/lib/native-static-libs.txt)" go build -a ./...

See release instructions for checksum verification and Windows setup.

Quick Start

package main

import (
	"fmt"
	"log"

	"github.com/maxdswain/splintr-go"
)

func main() {
	// Load a pretrained vocabulary (OpenAI GPT-4/3.5).
	// Also: llama3, deepseek_v3, qwen3, glm4, gpt-oss.
	tok, err := splintr.FromPretrained("cl100k_base")
	if err != nil {
		log.Fatal(err)
	}
	defer tok.Close()

	// Encode and decode.
	ids, err := tok.Encode("Hello, world!")
	if err != nil {
		log.Fatal(err)
	}
	text, err := tok.Decode(ids)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(ids)
	fmt.Println(text) // Hello, world!

	// Batch encode and decode, parallelized natively for larger inputs.
	batch, err := tok.EncodeBatch([]string{"Hello, world!", "How are you?"})
	if err != nil {
		log.Fatal(err)
	}
	texts, err := tok.DecodeBatch(batch)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(batch)
	fmt.Println(texts)
}

Batch results preserve input order and follow the same special-token behavior as Encode and Decode. An empty batch returns an empty slice; an error returns no partial results.

Streaming decoding

Using tok and ids from above:

stream, err := tok.StreamingDecoder()
if err != nil {
	log.Fatal(err)
}
defer stream.Close()

for _, id := range ids {
	text, err := stream.AddToken(id)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Print(text)
}
tail, err := stream.Flush()
if err != nil {
	log.Fatal(err)
}
fmt.Print(tail)

The decoder buffers split UTF-8 characters; Flush emits remaining text, replacing incomplete sequences with �. Reset starts a new sequence, while HasPending and PendingBytes inspect buffered text. Use StreamingDecoderWithSpecialMode(DecodeRenderSpecial) to render special tokens, or AddTokenLossy / AddTokensLossy to skip unknown IDs.

Streams survive closing their tokenizer and serialize concurrent calls. Close discards pending text, so flush first. Unsupported decoder pipelines fail at construction; a strict add error retains any accepted prefix for a later flush or reset. Splintr has no streaming encoder; use Encode or EncodeBatch for complete text.

Hugging Face tokenizer.json
tok, err := splintr.FromFile("tokenizer.json") // or splintr.FromBytes(data)
if err != nil {
	log.Fatal(err)
}
defer tok.Close()
Untrusted text
ids, err := tok.EncodeWithSpecialMode("<|im_start|>user", splintr.SpecialOrdinary)
if err != nil {
	log.Fatal(err)
}
fmt.Println(ids)

SpecialOrdinary treats control-token spellings as text. Both encoding modes still apply post-processing. Tokenizers support concurrent calls and explicit cleanup with Close.

Development

Source builds need Rust, Go, Make and a C compiler. License collection and release packaging also need jq and tar.

make build
make test
make fmt-check lint # rustfmt, Clippy and Revive via golangci-lint v2

License

The Go bindings are licensed under Apache 2.0. Bundled tokenizer vocabularies keep their original licenses, listed in LICENSE-OTHERS.

Llama 3 vocabulary attribution (required by Meta’s license): “Built with Llama”.

Documentation

Overview

Package splintr provides Go bindings to the Rust splintr tokenizer.

Index

Constants

View Source
const (
	CodeInvalidArgument = ErrorInvalidArgument
	CodeInvalidUTF8     = ErrorInvalidUTF8
	CodeTokenizer       = ErrorTokenizer
	CodePanic           = ErrorPanic
)

Error-code aliases.

Variables

View Source
var ErrClosed = errors.New("splintr: tokenizer is closed")

ErrClosed is returned when an operation is attempted on a closed, nil, or zero-value Tokenizer or StreamingDecoder.

Functions

This section is empty.

Types

type DecodeSpecialMode

type DecodeSpecialMode int

DecodeSpecialMode controls how declared special token IDs are decoded.

const (
	// DecodeSkipSpecial omits declared special tokens from decoded text.
	DecodeSkipSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_SKIP_SPECIAL)
	// DecodeRenderSpecial renders declared special tokens in decoded text.
	DecodeRenderSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_RENDER_SPECIAL)
)

type EncodeSpecialMode

type EncodeSpecialMode int

EncodeSpecialMode controls recognition of special-token spellings.

const (
	// SpecialOrdinary treats every special-token spelling as ordinary input.
	SpecialOrdinary EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ORDINARY)
	// SpecialAll recognizes every configured special token in the input.
	SpecialAll EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ALL)
)

type Error

type Error struct {
	Code    ErrorCode
	Message string
}

Error is an error reported by the native splintr library.

func (*Error) Error

func (e *Error) Error() string

type ErrorCode

type ErrorCode int

ErrorCode identifies an error status returned by the native library.

const (
	// ErrorInvalidArgument means an argument did not satisfy the ABI contract.
	ErrorInvalidArgument ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_ARGUMENT)
	// ErrorInvalidUTF8 means a text argument was not valid UTF-8.
	ErrorInvalidUTF8 ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_UTF8)
	// ErrorTokenizer means tokenizer parsing or an encode/decode operation failed.
	ErrorTokenizer ErrorCode = ErrorCode(C.SPLINTR_ERROR_TOKENIZER)
	// ErrorPanic means the native library caught a Rust panic.
	ErrorPanic ErrorCode = ErrorCode(C.SPLINTR_ERROR_PANIC)
)

type StreamingDecoder added in v0.3.0

type StreamingDecoder struct {
	// contains filtered or unexported fields
}

StreamingDecoder incrementally decodes IDs. Calls are serialized; its zero value is closed. Copies share one cursor and lifetime.

func (*StreamingDecoder) AddToken added in v0.3.0

func (d *StreamingDecoder) AddToken(id uint32) (string, error)

AddToken decodes one ID; unknown IDs return an error.

func (*StreamingDecoder) AddTokenLossy added in v0.3.0

func (d *StreamingDecoder) AddTokenLossy(id uint32) (string, error)

AddTokenLossy decodes one ID, skipping unknown IDs.

func (*StreamingDecoder) AddTokens added in v0.3.0

func (d *StreamingDecoder) AddTokens(ids []uint32) (string, error)

AddTokens emits available text, possibly empty. Unknown IDs return an error; any accepted prefix remains buffered.

func (*StreamingDecoder) AddTokensLossy added in v0.3.0

func (d *StreamingDecoder) AddTokensLossy(ids []uint32) (string, error)

AddTokensLossy is AddTokens with unknown IDs skipped.

func (*StreamingDecoder) Close added in v0.3.0

func (d *StreamingDecoder) Close() error

Close waits for active calls and discards pending text. It is idempotent.

func (*StreamingDecoder) Flush added in v0.3.0

func (d *StreamingDecoder) Flush() (string, error)

Flush drains buffered text, replacing incomplete UTF-8. It neither resets nor closes the decoder.

func (*StreamingDecoder) HasPending added in v0.3.0

func (d *StreamingDecoder) HasPending() (bool, error)

HasPending reports whether any buffered text remains.

func (*StreamingDecoder) PendingBytes added in v0.3.0

func (d *StreamingDecoder) PendingBytes() (int, error)

PendingBytes counts bytes awaiting emission, including post-processing buffers.

func (*StreamingDecoder) Reset added in v0.3.0

func (d *StreamingDecoder) Reset() error

Reset discards pending text and starts a new sequence.

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer is a native tokenizer handle. Its zero value is closed.

Tokenizer values may be copied: copies share ownership and closing one makes every copy closed. Methods may be called concurrently with each other and with Close; Close waits for calls already in progress.

func FromBytes

func FromBytes(data []byte) (*Tokenizer, error)

FromBytes loads a Hugging Face tokenizer.json document. The data must be non-empty. Loading is local and does not retain data.

func FromFile

func FromFile(path string) (*Tokenizer, error)

FromFile reads and loads a Hugging Face tokenizer.json file.

func FromPretrained

func FromPretrained(name string) (*Tokenizer, error)

FromPretrained loads a bundled vocabulary by name or alias, without downloads.

func (*Tokenizer) Close

func (t *Tokenizer) Close() error

Close waits for active calls and releases the shared handle. It is idempotent.

func (*Tokenizer) Decode

func (t *Tokenizer) Decode(ids []uint32) (string, error)

Decode decodes IDs while skipping declared special tokens. It follows splintr's behavior for unknown IDs: declared decoder pipelines may omit them, while other decoding paths return an error.

func (*Tokenizer) DecodeBatch added in v0.2.0

func (t *Tokenizer) DecodeBatch(tokenLists [][]uint32) ([]string, error)

DecodeBatch decodes token lists in input order using splintr's native batch processing. Special tokens and unknown IDs behave as in Decode. Empty batches return an empty slice; any error returns nil rather than partial results.

func (*Tokenizer) DecodeWithSpecialMode

func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)

DecodeWithSpecialMode decodes IDs while either skipping or rendering declared special tokens.

func (*Tokenizer) EOSTokenID

func (t *Tokenizer) EOSTokenID() (uint32, bool, error)

EOSTokenID returns the configured end-of-sequence token ID. The boolean is false when no EOS token is configured.

func (*Tokenizer) Encode

func (t *Tokenizer) Encode(text string) ([]uint32, error)

Encode recognizes all configured special tokens and applies post-processing, which may insert special IDs even for empty text.

func (*Tokenizer) EncodeBatch added in v0.2.0

func (t *Tokenizer) EncodeBatch(texts []string) ([][]uint32, error)

EncodeBatch encodes texts in input order using splintr's native batch processing. It has the same special-token and post-processing behavior as Encode. Empty batches return an empty slice; any error returns nil rather than partial results.

func (*Tokenizer) EncodeWithSpecialMode

func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)

EncodeWithSpecialMode encodes text with explicit special-token recognition. The tokenizer.json single-sequence post-processor is applied in both modes.

func (*Tokenizer) Family

func (t *Tokenizer) Family() (string, error)

Family returns the model family: "BPE", "Unigram", "WordPiece", "Spm", or "Unknown".

func (*Tokenizer) SpecialTokenID

func (t *Tokenizer) SpecialTokenID(token string) (uint32, bool, error)

SpecialTokenID looks up a declared special token by its exact content. The boolean is false if token is not declared special.

func (*Tokenizer) StreamingDecoder added in v0.3.0

func (t *Tokenizer) StreamingDecoder() (*StreamingDecoder, error)

StreamingDecoder creates a decoder that skips specials and outlives its tokenizer.

func (*Tokenizer) StreamingDecoderWithSpecialMode added in v0.3.0

func (t *Tokenizer) StreamingDecoderWithSpecialMode(mode DecodeSpecialMode) (*StreamingDecoder, error)

StreamingDecoderWithSpecialMode sets special-token handling in a new decoder. Unsupported pipelines return an error.

func (*Tokenizer) VocabSize

func (t *Tokenizer) VocabSize() (int, error)

VocabSize returns the number of entries in the tokenizer vocabulary.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL