gguf

Documentation

Overview

Package gguf extracts a BPE tokenizer from GGUF file metadata.

It defines a Metadata interface so callers can provide any GGUF parser implementation without depending on a specific one.

Index

• func ExtractTokenizer(m Metadata) (*ztoken.BPETokenizer, error)
• type Metadata

Functions

func ExtractTokenizer

func ExtractTokenizer(m Metadata) (*ztoken.BPETokenizer, error)

ExtractTokenizer builds a BPETokenizer from GGUF metadata. GGUF files store tokenizer data under the "tokenizer.ggml.*" metadata keys.

Types

type Metadata

type Metadata interface {
	GetString(key string) (string, bool)
	GetStringArray(key string) ([]string, bool)
	GetUint32(key string) (uint32, bool)
	GetInt32Array(key string) ([]int32, bool)
}

Metadata provides access to GGUF key-value metadata.