Documentation
¶
Overview ¶
Package splintr provides local text tokenization through the Rust splintr library. It supports whole-text encoding and decoding, parallel batches, and streaming decoding. See installation instructions for cgo and native-library setup.
Start with FromPretrained for a bundled vocabulary or FromFile for a Hugging Face tokenizer.json. Reuse the resulting Tokenizer and call Tokenizer.Close when done.
Getting started ¶
Default native builds include 23 vocabulary variants, including cl100k_base, llama3, qwen3, and deepseek_v3. Loading a bundled vocabulary needs no downloads:
tok, err := splintr.FromPretrained("cl100k_base")
if err != nil {
log.Fatal(err)
}
defer tok.Close()
ids, err := tok.Encode("Hello, world!")
if err != nil {
log.Fatal(err)
}
fmt.Println(ids) // [9906 11 1917 0]
Use Tokenizer.Decode to turn IDs back into text, or FromBytes to load a JSON document already in memory. Token IDs belong to the vocabulary that produced them; use the same tokenizer to decode them. Normalization and special-token handling mean decoding is not always an exact inverse of encoding. See pretrained vocabularies for supported names and model-specific behavior.
Batch processing ¶
Tokenizer.EncodeBatch and Tokenizer.DecodeBatch preserve input order and use the same defaults as their single-input counterparts. Splintr runs larger batches in parallel. Empty batches return empty slices; errors return no partial results.
Streaming decoding ¶
Create a StreamingDecoder with Tokenizer.StreamingDecoder, then feed IDs with StreamingDecoder.AddToken or StreamingDecoder.AddTokens. Concatenate the returned fragments, which may be empty while UTF-8 characters or decoder operations are pending. Call StreamingDecoder.Flush at the end to emit remaining text; incomplete UTF-8 is replaced with U+FFFD. Flush does not close or reset the stream.
Unknown IDs return an error; any prefix accepted by that call stays buffered. StreamingDecoder.AddTokensLossy skips unknown IDs instead. StreamingDecoder.Reset discards pending text and starts a new sequence. Some tokenizer.json decoder pipelines cannot stream and fail at construction. Encoding requires complete input text; there is no streaming encoder.
Special tokens ¶
Tokenizer.Encode recognizes configured special-token spellings using SpecialAll. For untrusted text, use Tokenizer.EncodeWithSpecialMode with SpecialOrdinary. Both modes still apply the tokenizer's post-processor, which may insert special IDs.
Tokenizer.Decode and Tokenizer.StreamingDecoder skip special tokens by default. Use their WithSpecialMode variants with DecodeRenderSpecial to include them. Ordinary decoding may omit unknown IDs in declared decoder pipelines; strict streaming adds reject them. Tokenizer.SpecialTokenID and Tokenizer.EOSTokenID provide special-token metadata.
Ownership and concurrency ¶
Tokenizer methods support concurrent calls. Stream methods serialize access to one cursor; callers must coordinate token order when sharing a stream. Streams own their state and remain usable after their tokenizer closes.
Copies of either handle share ownership, so closing one closes all its copies. Close waits for active calls and is idempotent. StreamingDecoder.Close discards pending text; flush first if it is needed. Returned slices and strings are Go-owned and remain valid after closing the handle.
Errors ¶
Operations on nil, zero-value, or closed handles return ErrClosed; Close remains safe to call. Use errors.Is to check this sentinel and errors.AsType to inspect a native Error and its ErrorCode. FromFile also returns filesystem errors.
Native library ¶
Building requires Go 1.26+, cgo, a C compiler, and a matching native static library. The go get command downloads the Go module, not the native library. See installation instructions for prebuilt archives or source builds for Rust builds. Bundled vocabularies have separate license terms.
Example ¶
This example uses a small JSON vocabulary and needs no bundled vocabularies.
package main
import (
"fmt"
"github.com/maxdswain/splintr-go"
)
const tokenizerJSON = `{
"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
"pre_tokenizer": {"type": "Whitespace"},
"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
"model": {"type": "WordPiece", "unk_token": "[UNK]",
"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`
func exampleTokenizer() *splintr.Tokenizer {
tok, err := splintr.FromBytes([]byte(tokenizerJSON))
if err != nil {
panic(err)
}
return tok
}
func main() {
tok := exampleTokenizer()
defer tok.Close()
ids, err := tok.Encode("hello world")
if err != nil {
panic(err)
}
text, err := tok.Decode(ids)
if err != nil {
panic(err)
}
fmt.Println(ids)
fmt.Println(text)
}
Output: [1 2] hello world
Index ¶
- Constants
- Variables
- type DecodeSpecialMode
- type EncodeSpecialMode
- type Error
- type ErrorCode
- type StreamingDecoder
- func (d *StreamingDecoder) AddToken(id uint32) (string, error)
- func (d *StreamingDecoder) AddTokenLossy(id uint32) (string, error)
- func (d *StreamingDecoder) AddTokens(ids []uint32) (string, error)
- func (d *StreamingDecoder) AddTokensLossy(ids []uint32) (string, error)
- func (d *StreamingDecoder) Close() error
- func (d *StreamingDecoder) Flush() (string, error)
- func (d *StreamingDecoder) HasPending() (bool, error)
- func (d *StreamingDecoder) PendingBytes() (int, error)
- func (d *StreamingDecoder) Reset() error
- type Tokenizer
- func (t *Tokenizer) Close() error
- func (t *Tokenizer) Decode(ids []uint32) (string, error)
- func (t *Tokenizer) DecodeBatch(tokenLists [][]uint32) ([]string, error)
- func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)
- func (t *Tokenizer) EOSTokenID() (uint32, bool, error)
- func (t *Tokenizer) Encode(text string) ([]uint32, error)
- func (t *Tokenizer) EncodeBatch(texts []string) ([][]uint32, error)
- func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)
- func (t *Tokenizer) Family() (string, error)
- func (t *Tokenizer) SpecialTokenID(token string) (uint32, bool, error)
- func (t *Tokenizer) StreamingDecoder() (*StreamingDecoder, error)
- func (t *Tokenizer) StreamingDecoderWithSpecialMode(mode DecodeSpecialMode) (*StreamingDecoder, error)
- func (t *Tokenizer) VocabSize() (int, error)
Examples ¶
Constants ¶
const ( CodeInvalidArgument = ErrorInvalidArgument CodeInvalidUTF8 = ErrorInvalidUTF8 CodeTokenizer = ErrorTokenizer CodePanic = ErrorPanic )
Error-code aliases.
Variables ¶
var ErrClosed = errors.New("splintr: tokenizer is closed")
ErrClosed is returned when an operation is attempted on a closed, nil, or zero-value Tokenizer or StreamingDecoder.
Functions ¶
This section is empty.
Types ¶
type DecodeSpecialMode ¶
type DecodeSpecialMode int
DecodeSpecialMode controls how declared special token IDs are decoded.
const ( // DecodeSkipSpecial omits declared special tokens from decoded text. DecodeSkipSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_SKIP_SPECIAL) // DecodeRenderSpecial renders declared special tokens in decoded text. DecodeRenderSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_RENDER_SPECIAL) )
type EncodeSpecialMode ¶
type EncodeSpecialMode int
EncodeSpecialMode controls recognition of special-token spellings.
const ( // SpecialOrdinary treats every special-token spelling as ordinary input. SpecialOrdinary EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ORDINARY) // SpecialAll recognizes every configured special token in the input. SpecialAll EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ALL) )
type Error ¶
type Error struct {
Code ErrorCode // Failure category.
Message string // Native detail, without the "splintr:" prefix.
}
Error describes a native failure. Use errors.AsType to inspect its Code.
Example ¶
package main
import (
"errors"
"fmt"
"github.com/maxdswain/splintr-go"
)
func main() {
_, err := splintr.FromBytes(nil)
if nativeErr, ok := errors.AsType[*splintr.Error](err); ok {
fmt.Println("invalid argument:", nativeErr.Code == splintr.ErrorInvalidArgument)
}
var tok splintr.Tokenizer
_, err = tok.Encode("hello")
fmt.Println("closed:", errors.Is(err, splintr.ErrClosed))
}
Output: invalid argument: true closed: true
type ErrorCode ¶
type ErrorCode int
ErrorCode identifies an error status returned by the native library.
const ( // ErrorInvalidArgument means an argument did not satisfy the ABI contract. ErrorInvalidArgument ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_ARGUMENT) // ErrorInvalidUTF8 means a text argument was not valid UTF-8. ErrorInvalidUTF8 ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_UTF8) // ErrorTokenizer means tokenizer parsing or an encode/decode operation failed. ErrorTokenizer ErrorCode = ErrorCode(C.SPLINTR_ERROR_TOKENIZER) // ErrorPanic means the native library caught a Rust panic. ErrorPanic ErrorCode = ErrorCode(C.SPLINTR_ERROR_PANIC) )
type StreamingDecoder ¶ added in v0.3.0
type StreamingDecoder struct {
// contains filtered or unexported fields
}
StreamingDecoder incrementally decodes IDs. Calls are serialized; its zero value is closed. Copies share one cursor and lifetime.
Example ¶
package main
import (
"fmt"
"github.com/maxdswain/splintr-go"
)
const tokenizerJSON = `{
"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
"pre_tokenizer": {"type": "Whitespace"},
"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
"model": {"type": "WordPiece", "unk_token": "[UNK]",
"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`
func exampleTokenizer() *splintr.Tokenizer {
tok, err := splintr.FromBytes([]byte(tokenizerJSON))
if err != nil {
panic(err)
}
return tok
}
func main() {
tok := exampleTokenizer()
defer tok.Close()
stream, err := tok.StreamingDecoder()
if err != nil {
panic(err)
}
defer stream.Close()
for _, id := range []uint32{1, 2} {
text, err := stream.AddToken(id)
if err != nil {
panic(err)
}
fmt.Print(text)
}
tail, err := stream.Flush()
if err != nil {
panic(err)
}
fmt.Println(tail)
}
Output: hello world
func (*StreamingDecoder) AddToken ¶ added in v0.3.0
func (d *StreamingDecoder) AddToken(id uint32) (string, error)
AddToken decodes one ID; unknown IDs return an error.
func (*StreamingDecoder) AddTokenLossy ¶ added in v0.3.0
func (d *StreamingDecoder) AddTokenLossy(id uint32) (string, error)
AddTokenLossy decodes one ID, skipping unknown IDs.
func (*StreamingDecoder) AddTokens ¶ added in v0.3.0
func (d *StreamingDecoder) AddTokens(ids []uint32) (string, error)
AddTokens emits available text, possibly empty. Unknown IDs return an error; any accepted prefix remains buffered.
func (*StreamingDecoder) AddTokensLossy ¶ added in v0.3.0
func (d *StreamingDecoder) AddTokensLossy(ids []uint32) (string, error)
AddTokensLossy is StreamingDecoder.AddTokens with unknown IDs skipped.
func (*StreamingDecoder) Close ¶ added in v0.3.0
func (d *StreamingDecoder) Close() error
Close waits for active calls and discards pending text. It is idempotent.
func (*StreamingDecoder) Flush ¶ added in v0.3.0
func (d *StreamingDecoder) Flush() (string, error)
Flush drains buffered text, replacing incomplete UTF-8 with U+FFFD. It neither resets nor closes the decoder; use StreamingDecoder.Reset for a new sequence.
func (*StreamingDecoder) HasPending ¶ added in v0.3.0
func (d *StreamingDecoder) HasPending() (bool, error)
HasPending reports whether any buffered text remains.
func (*StreamingDecoder) PendingBytes ¶ added in v0.3.0
func (d *StreamingDecoder) PendingBytes() (int, error)
PendingBytes counts bytes awaiting emission, including post-processing buffers.
func (*StreamingDecoder) Reset ¶ added in v0.3.0
func (d *StreamingDecoder) Reset() error
Reset discards pending text and starts a new sequence.
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer is a native tokenizer handle. Its zero value is closed.
Tokenizer values may be copied: copies share ownership and closing one makes every copy closed. Methods may be called concurrently with each other and with Tokenizer.Close; Close waits for calls already in progress.
func FromBytes ¶
FromBytes loads a Hugging Face tokenizer.json document. The data must be non-empty. Loading is local and does not retain data.
func FromPretrained ¶
FromPretrained loads a bundled vocabulary by name or alias, without downloads. Available vocabularies depend on the native library's compiled features.
func (*Tokenizer) Close ¶
Close waits for active calls and releases the shared handle. It is idempotent.
func (*Tokenizer) Decode ¶
Decode decodes IDs with DecodeSkipSpecial. It follows splintr's behavior for unknown IDs: declared decoder pipelines may omit them, while other decoding paths return an error.
func (*Tokenizer) DecodeBatch ¶ added in v0.2.0
DecodeBatch decodes token lists in input order using splintr's native batch processing. Special tokens and unknown IDs behave as in Tokenizer.Decode. Empty batches return an empty slice; any error returns nil rather than partial results.
func (*Tokenizer) DecodeWithSpecialMode ¶
func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)
DecodeWithSpecialMode decodes IDs while either skipping or rendering declared special tokens.
func (*Tokenizer) EOSTokenID ¶
EOSTokenID returns the configured end-of-sequence token ID. The boolean is false when no EOS token is configured.
func (*Tokenizer) Encode ¶
Encode encodes text with SpecialAll and applies post-processing, which may insert special IDs even for empty text.
func (*Tokenizer) EncodeBatch ¶ added in v0.2.0
EncodeBatch encodes texts in input order using splintr's native batch processing. It has the same special-token and post-processing behavior as Tokenizer.Encode. Empty batches return an empty slice; any error returns nil rather than partial results.
Example ¶
package main
import (
"fmt"
"github.com/maxdswain/splintr-go"
)
const tokenizerJSON = `{
"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
"pre_tokenizer": {"type": "Whitespace"},
"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
"model": {"type": "WordPiece", "unk_token": "[UNK]",
"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`
func exampleTokenizer() *splintr.Tokenizer {
tok, err := splintr.FromBytes([]byte(tokenizerJSON))
if err != nil {
panic(err)
}
return tok
}
func main() {
tok := exampleTokenizer()
defer tok.Close()
batch, err := tok.EncodeBatch([]string{"hello world", "world", ""})
if err != nil {
panic(err)
}
texts, err := tok.DecodeBatch(batch)
if err != nil {
panic(err)
}
fmt.Println(batch)
fmt.Printf("%q\n", texts)
}
Output: [[1 2] [2] []] ["hello world" "world" ""]
func (*Tokenizer) EncodeWithSpecialMode ¶
func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)
EncodeWithSpecialMode controls special-token recognition; use SpecialOrdinary for untrusted text. Both modes apply the tokenizer.json single-sequence post-processor.
Example ¶
package main
import (
"fmt"
"slices"
"github.com/maxdswain/splintr-go"
)
const tokenizerJSON = `{
"added_tokens": [{"id": 3, "content": "[SEP]", "special": true}],
"pre_tokenizer": {"type": "Whitespace"},
"decoder": {"type": "WordPiece", "prefix": "##", "cleanup": true},
"model": {"type": "WordPiece", "unk_token": "[UNK]",
"continuing_subword_prefix": "##", "max_input_chars_per_word": 100,
"vocab": {"[UNK]": 0, "hello": 1, "world": 2, "[SEP]": 3}}
}`
func exampleTokenizer() *splintr.Tokenizer {
tok, err := splintr.FromBytes([]byte(tokenizerJSON))
if err != nil {
panic(err)
}
return tok
}
func main() {
tok := exampleTokenizer()
defer tok.Close()
id, found, err := tok.SpecialTokenID("[SEP]")
if err != nil {
panic(err)
}
if !found {
panic("missing special token")
}
special, err := tok.EncodeWithSpecialMode("hello [SEP]", splintr.SpecialAll)
if err != nil {
panic(err)
}
ordinary, err := tok.EncodeWithSpecialMode("hello [SEP]", splintr.SpecialOrdinary)
if err != nil {
panic(err)
}
fmt.Println("recognized:", slices.Contains(special, id))
fmt.Println("ordinary:", slices.Contains(ordinary, id))
text, err := tok.DecodeWithSpecialMode(special, splintr.DecodeRenderSpecial)
if err != nil {
panic(err)
}
fmt.Println(text)
}
Output: recognized: true ordinary: false hello [SEP]
func (*Tokenizer) Family ¶
Family returns the model family: "BPE", "Unigram", "WordPiece", "Spm", or "Unknown".
func (*Tokenizer) SpecialTokenID ¶
SpecialTokenID looks up a declared special token by its exact content. The boolean is false if token is not declared special.
func (*Tokenizer) StreamingDecoder ¶ added in v0.3.0
func (t *Tokenizer) StreamingDecoder() (*StreamingDecoder, error)
StreamingDecoder creates a decoder with DecodeSkipSpecial that outlives its tokenizer.
func (*Tokenizer) StreamingDecoderWithSpecialMode ¶ added in v0.3.0
func (t *Tokenizer) StreamingDecoderWithSpecialMode(mode DecodeSpecialMode) (*StreamingDecoder, error)
StreamingDecoderWithSpecialMode sets special-token handling in a new decoder. Unsupported pipelines return an error.