Documentation
¶
Overview ¶
Package splintr provides Go bindings to the Rust splintr tokenizer.
Index ¶
- Constants
- Variables
- type DecodeSpecialMode
- type EncodeSpecialMode
- type Error
- type ErrorCode
- type StreamingDecoder
- func (d *StreamingDecoder) AddToken(id uint32) (string, error)
- func (d *StreamingDecoder) AddTokenLossy(id uint32) (string, error)
- func (d *StreamingDecoder) AddTokens(ids []uint32) (string, error)
- func (d *StreamingDecoder) AddTokensLossy(ids []uint32) (string, error)
- func (d *StreamingDecoder) Close() error
- func (d *StreamingDecoder) Flush() (string, error)
- func (d *StreamingDecoder) HasPending() (bool, error)
- func (d *StreamingDecoder) PendingBytes() (int, error)
- func (d *StreamingDecoder) Reset() error
- type Tokenizer
- func (t *Tokenizer) Close() error
- func (t *Tokenizer) Decode(ids []uint32) (string, error)
- func (t *Tokenizer) DecodeBatch(tokenLists [][]uint32) ([]string, error)
- func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)
- func (t *Tokenizer) EOSTokenID() (uint32, bool, error)
- func (t *Tokenizer) Encode(text string) ([]uint32, error)
- func (t *Tokenizer) EncodeBatch(texts []string) ([][]uint32, error)
- func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)
- func (t *Tokenizer) Family() (string, error)
- func (t *Tokenizer) SpecialTokenID(token string) (uint32, bool, error)
- func (t *Tokenizer) StreamingDecoder() (*StreamingDecoder, error)
- func (t *Tokenizer) StreamingDecoderWithSpecialMode(mode DecodeSpecialMode) (*StreamingDecoder, error)
- func (t *Tokenizer) VocabSize() (int, error)
Constants ¶
const ( CodeInvalidArgument = ErrorInvalidArgument CodeInvalidUTF8 = ErrorInvalidUTF8 CodeTokenizer = ErrorTokenizer CodePanic = ErrorPanic )
Error-code aliases.
Variables ¶
var ErrClosed = errors.New("splintr: tokenizer is closed")
ErrClosed is returned when an operation is attempted on a closed, nil, or zero-value Tokenizer or StreamingDecoder.
Functions ¶
This section is empty.
Types ¶
type DecodeSpecialMode ¶
type DecodeSpecialMode int
DecodeSpecialMode controls how declared special token IDs are decoded.
const ( // DecodeSkipSpecial omits declared special tokens from decoded text. DecodeSkipSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_SKIP_SPECIAL) // DecodeRenderSpecial renders declared special tokens in decoded text. DecodeRenderSpecial DecodeSpecialMode = DecodeSpecialMode(C.SPLINTR_DECODE_RENDER_SPECIAL) )
type EncodeSpecialMode ¶
type EncodeSpecialMode int
EncodeSpecialMode controls recognition of special-token spellings.
const ( // SpecialOrdinary treats every special-token spelling as ordinary input. SpecialOrdinary EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ORDINARY) // SpecialAll recognizes every configured special token in the input. SpecialAll EncodeSpecialMode = EncodeSpecialMode(C.SPLINTR_SPECIAL_ALL) )
type ErrorCode ¶
type ErrorCode int
ErrorCode identifies an error status returned by the native library.
const ( // ErrorInvalidArgument means an argument did not satisfy the ABI contract. ErrorInvalidArgument ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_ARGUMENT) // ErrorInvalidUTF8 means a text argument was not valid UTF-8. ErrorInvalidUTF8 ErrorCode = ErrorCode(C.SPLINTR_ERROR_INVALID_UTF8) // ErrorTokenizer means tokenizer parsing or an encode/decode operation failed. ErrorTokenizer ErrorCode = ErrorCode(C.SPLINTR_ERROR_TOKENIZER) // ErrorPanic means the native library caught a Rust panic. ErrorPanic ErrorCode = ErrorCode(C.SPLINTR_ERROR_PANIC) )
type StreamingDecoder ¶ added in v0.3.0
type StreamingDecoder struct {
// contains filtered or unexported fields
}
StreamingDecoder incrementally decodes IDs. Calls are serialized; its zero value is closed. Copies share one cursor and lifetime.
func (*StreamingDecoder) AddToken ¶ added in v0.3.0
func (d *StreamingDecoder) AddToken(id uint32) (string, error)
AddToken decodes one ID; unknown IDs return an error.
func (*StreamingDecoder) AddTokenLossy ¶ added in v0.3.0
func (d *StreamingDecoder) AddTokenLossy(id uint32) (string, error)
AddTokenLossy decodes one ID, skipping unknown IDs.
func (*StreamingDecoder) AddTokens ¶ added in v0.3.0
func (d *StreamingDecoder) AddTokens(ids []uint32) (string, error)
AddTokens emits available text, possibly empty. Unknown IDs return an error; any accepted prefix remains buffered.
func (*StreamingDecoder) AddTokensLossy ¶ added in v0.3.0
func (d *StreamingDecoder) AddTokensLossy(ids []uint32) (string, error)
AddTokensLossy is AddTokens with unknown IDs skipped.
func (*StreamingDecoder) Close ¶ added in v0.3.0
func (d *StreamingDecoder) Close() error
Close waits for active calls and discards pending text. It is idempotent.
func (*StreamingDecoder) Flush ¶ added in v0.3.0
func (d *StreamingDecoder) Flush() (string, error)
Flush drains buffered text, replacing incomplete UTF-8. It neither resets nor closes the decoder.
func (*StreamingDecoder) HasPending ¶ added in v0.3.0
func (d *StreamingDecoder) HasPending() (bool, error)
HasPending reports whether any buffered text remains.
func (*StreamingDecoder) PendingBytes ¶ added in v0.3.0
func (d *StreamingDecoder) PendingBytes() (int, error)
PendingBytes counts bytes awaiting emission, including post-processing buffers.
func (*StreamingDecoder) Reset ¶ added in v0.3.0
func (d *StreamingDecoder) Reset() error
Reset discards pending text and starts a new sequence.
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer is a native tokenizer handle. Its zero value is closed.
Tokenizer values may be copied: copies share ownership and closing one makes every copy closed. Methods may be called concurrently with each other and with Close; Close waits for calls already in progress.
func FromBytes ¶
FromBytes loads a Hugging Face tokenizer.json document. The data must be non-empty. Loading is local and does not retain data.
func FromPretrained ¶
FromPretrained loads a bundled vocabulary by name or alias, without downloads.
func (*Tokenizer) Close ¶
Close waits for active calls and releases the shared handle. It is idempotent.
func (*Tokenizer) Decode ¶
Decode decodes IDs while skipping declared special tokens. It follows splintr's behavior for unknown IDs: declared decoder pipelines may omit them, while other decoding paths return an error.
func (*Tokenizer) DecodeBatch ¶ added in v0.2.0
DecodeBatch decodes token lists in input order using splintr's native batch processing. Special tokens and unknown IDs behave as in Decode. Empty batches return an empty slice; any error returns nil rather than partial results.
func (*Tokenizer) DecodeWithSpecialMode ¶
func (t *Tokenizer) DecodeWithSpecialMode(ids []uint32, mode DecodeSpecialMode) (string, error)
DecodeWithSpecialMode decodes IDs while either skipping or rendering declared special tokens.
func (*Tokenizer) EOSTokenID ¶
EOSTokenID returns the configured end-of-sequence token ID. The boolean is false when no EOS token is configured.
func (*Tokenizer) Encode ¶
Encode recognizes all configured special tokens and applies post-processing, which may insert special IDs even for empty text.
func (*Tokenizer) EncodeBatch ¶ added in v0.2.0
EncodeBatch encodes texts in input order using splintr's native batch processing. It has the same special-token and post-processing behavior as Encode. Empty batches return an empty slice; any error returns nil rather than partial results.
func (*Tokenizer) EncodeWithSpecialMode ¶
func (t *Tokenizer) EncodeWithSpecialMode(text string, mode EncodeSpecialMode) ([]uint32, error)
EncodeWithSpecialMode encodes text with explicit special-token recognition. The tokenizer.json single-sequence post-processor is applied in both modes.
func (*Tokenizer) Family ¶
Family returns the model family: "BPE", "Unigram", "WordPiece", "Spm", or "Unknown".
func (*Tokenizer) SpecialTokenID ¶
SpecialTokenID looks up a declared special token by its exact content. The boolean is false if token is not declared special.
func (*Tokenizer) StreamingDecoder ¶ added in v0.3.0
func (t *Tokenizer) StreamingDecoder() (*StreamingDecoder, error)
StreamingDecoder creates a decoder that skips specials and outlives its tokenizer.
func (*Tokenizer) StreamingDecoderWithSpecialMode ¶ added in v0.3.0
func (t *Tokenizer) StreamingDecoderWithSpecialMode(mode DecodeSpecialMode) (*StreamingDecoder, error)
StreamingDecoderWithSpecialMode sets special-token handling in a new decoder. Unsupported pipelines return an error.