llama

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 20, 2026 License: MIT Imports: 9 Imported by: 0

README

Go Reference go-llama.cpp

LLama.cpp golang bindings.

The go-llama.cpp bindings are high level, as such most of the work is kept into the C/C++ code to avoid any extra computational cost, be more performant and lastly ease out maintenance, while keeping the usage as simple as possible.

Check out this and this write-ups which summarize the impact of a low-level interface which calls C functions from Go.

If you are looking for an high-level OpenAI compatible API, check out here.

Attention!

Since https://github.com/dyammarcano/go-llama.cpp/pull/180 is merged, now go-llama.cpp is not anymore compatible with ggml format, but it works ONLY with the new gguf file format. See also the upstream PR: https://github.com/ggerganov/llama.cpp/pull/2398.

If you need to use the ggml format, use the https://github.com/dyammarcano/go-llama.cpp/releases/tag/pre-gguf tag.

Usage

Note: This repository uses git submodules to keep track of LLama.cpp.

Clone the repository locally:

git clone --recurse-submodules https://github.com/dyammarcano/go-llama.cpp

Modernized fork: this fork tracks current llama.cpp (pure C llama.h API, llama_sampler chain, chat templates), builds with Go 1.25 and a Task file (the old make libbinding.a flow is gone), and has build targets for CPU, CUDA, and Vulkan backends. Only the CPU target is verified against the current pin — CUDA and Vulkan have not been built here (their build directories are empty).

Build with Task (task --list shows all targets):

cd go-llama.cpp
task deps          # init the llama.cpp submodule
task build:cpu     # MinGW static (default)
task build:cuda    # MSVC shared DLLs, GGML_CUDA=ON (needs VS2022 + CUDA toolkit)
task build:vulkan  # MinGW static, GGML_VULKAN=ON (needs the Vulkan SDK)

Then run the example:

go run ./examples -m "/model/path/here" -t 14          # CPU
go run -tags cuda ./examples -m "/model/path/here"      # CUDA (ship the build-cuda DLLs on PATH)

Other targets: task test, task fmt, task fix, task lint, task clean.

Reading GGUF metadata (no model load)

The gguf subpackage reads model metadata and tensor info in pure Go — no cgo, no llama.cpp, no GPU:

import "github.com/dyammarcano/go-llama.cpp/gguf"

info, err := gguf.Stat("model.gguf")
// info.Architecture, info.ContextLength, info.BlockCount,
// info.Quantization, info.ChatTemplate, info.NumTensors, ...

For lower-level access (per-tensor shapes/types, raw key-values), use gguf.Open and the *gguf.File accessors.

Estimating GPU layers (no model load)

gguf.EstimateLayers computes how many transformer layers fit in a VRAM budget — pure Go, no cgo, no GPU calls (the caller supplies the budget):

import "github.com/dyammarcano/go-llama.cpp/gguf"

est, err := gguf.EstimateLayers("model.gguf", gguf.EstimateOptions{
    NumCtx:   4096,
    FreeVRAM: 3500 << 20, // ~3.5 GiB
})
// est.Layers (n_gpu_layers), est.FullyOffloaded, est.Weights/KVCache/Graph

It faithfully ports Ollama's Llama-family memory model (single-GPU). Non-Llama dense architectures get an approximate estimate (est.Approximate == true); recurrent/SSM architectures are not yet supported.

Streaming output filter (stop sequences + UTF-8)

streamfilter.Filter filters a stream of decoded token pieces — pure Go, no cgo: it holds back text that might be part of a stop sequence or an incomplete multibyte UTF-8 character, and reports when a stop is hit:

import "github.com/dyammarcano/go-llama.cpp/streamfilter"

f := streamfilter.New([]string{"</s>", "User:"})
emit, stop := f.Push(piece) // forward emit to your callback; halt when stop
// ... at end of generation:
emit = f.Flush()

It is a faithful port of Ollama's stop-sequence handling, and it is now wired into the in-process binding: Predict, PredictResult, and SpeculativeSampling route every decoded piece through it, so stop sequences are trimmed (even when split across token pieces) and multibyte UTF-8 is never broken. See docs/streamfilter-smoke-test.md for the manual model smoke test.

Sampler smoke test (manual, requires a built cgo binary + a GGUF model)
  1. Greedy (SetTemperature(0)) output is identical to before this change.
  2. SetMinP(0.05) with SetTemperature(0.8) produces coherent text.
  3. SetMirostat(2) produces coherent text.
  4. SetLogitBias("<id>:-100") for a token that otherwise appears makes that token never appear in the output.

Acceleration (legacy — pre-Task, non-functional)

These blocks do not run. They invoke make libbinding.a, but this fork has no Makefile — the Make flow was replaced by Task (see the note near the top of this README). They are retained only as a record of the upstream flags each backend needs; port them to scripts/llamacpp.sh before use.

OpenBLAS

To build and run with OpenBLAS, for example:

BUILD_TYPE=openblas make libbinding.a
CGO_LDFLAGS="-lopenblas" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run -tags openblas ./examples -m "/model/path/here" -t 14
CuBLAS

To build with CuBLAS:

BUILD_TYPE=cublas make libbinding.a
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14
ROCM

To build with ROCM (HIPBLAS):

BUILD_TYPE=hipblas make libbinding.a
CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ CGO_LDFLAGS="-O3 --hip-link --rtlib=compiler-rt -unwindlib=libgcc -lrocblas -lhipblas" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -ngl 64 -t 32
OpenCL
BUILD_TYPE=clblas CLBLAS_DIR=... make libbinding.a
CGO_LDFLAGS="-lOpenCL -lclblast -L/usr/local/lib64/" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14

You should see something like this from the output when using the GPU:

ggml_opencl: selecting platform: 'Intel(R) OpenCL HD Graphics'
ggml_opencl: selecting device: 'Intel(R) Graphics [0x46a6]'
ggml_opencl: device FP16 support: true

GPU offloading (legacy — pre-Task, non-functional)

Same caveat as Acceleration above: make libbinding.a no longer exists.

Metal (Apple Silicon)
BUILD_TYPE=metal make libbinding.a
CGO_LDFLAGS="-framework Foundation -framework Metal -framework MetalKit -framework MetalPerformanceShaders" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go build ./examples/main.go
cp build/bin/ggml-metal.metal .
./main -m "/model/path/here" -t 1 -ngl 1

Enjoy!

The documentation is available here and the full example code is here.

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrModelLoad is returned when a model file cannot be loaded.
	ErrModelLoad = errors.New("llama: failed to load model")

	// ErrStateLoad is returned when a saved context state cannot be restored.
	ErrStateLoad = errors.New("llama: failed to load state")

	// ErrInference is returned when a decode/generation call fails.
	ErrInference = errors.New("llama: inference failed")

	// ErrOutOfMemory is returned when a generation buffer cannot be allocated.
	ErrOutOfMemory = errors.New("llama: out of memory allocating result buffer")

	// ErrEmbeddingsDisabled is returned by embedding calls on a model that was
	// loaded without EnableEmbeddings.
	ErrEmbeddingsDisabled = errors.New("llama: model loaded without embeddings")

	// ErrNotImplemented is returned by API surface that is exported but not yet
	// backed by a real implementation (embeddings, state save/load, speculative
	// sampling). It exists so these calls fail loudly instead of silently
	// returning empty results.
	ErrNotImplemented = errors.New("llama: not implemented")
)

Sentinel errors returned by the binding. Callers can match them with errors.Is so failure handling does not depend on message text.

Functions

This section is empty.

Types

type LLama 

type LLama struct {
	// contains filtered or unexported fields
}

LLama is a handle to a loaded llama.cpp model and its context. It is created with New and must be released with Free when no longer needed. A LLama is not safe for concurrent use by multiple goroutines.

func New 

func New(model string, opts ...ModelOption) (*LLama, error)

New loads the GGUF model at the given path and returns a ready LLama handle. Configure loading with ModelOption values (context size, mmap, GPU layers, LoRA, and so on). The returned handle must be released with Free. It returns ErrModelLoad if the model cannot be loaded.

func (*LLama) ApplyChatTemplate 

func (l *LLama) ApplyChatTemplate(system, user string) (string, error)

ApplyChatTemplate formats a (system, user) pair using the model's embedded GGUF chat template. It returns ("", nil) when the model has no template, so callers can fall back to raw concatenation.

func (*LLama) Embeddings 

func (l *LLama) Embeddings(text string, opts ...PredictOption) ([]float32, error)

Embeddings returns the embedding vector for the given text.

Not yet implemented: it returns ErrNotImplemented (or ErrEmbeddingsDisabled if the model was loaded without EnableEmbeddings). The underlying binding is a stub, so this fails loudly instead of returning an empty vector.

func (*LLama) Eval 

func (l *LLama) Eval(text string, opts ...PredictOption) error

Eval runs a forward pass over the given text without returning generated output. It is used to warm the context / KV cache. Configure it with PredictOption values. It returns ErrInference if the forward pass fails.

func (*LLama) Free 

func (l *LLama) Free()

Free releases the model and its context. It must be called exactly once when the handle is no longer needed; the handle must not be used afterwards.

func (*LLama) LoadState 

func (l *LLama) LoadState(state string) error

LoadState restores a previously saved context state from the given file.

Not yet implemented: it returns ErrNotImplemented. The underlying binding is a stub, so this call is a loud no-op rather than a silent one.

func (*LLama) Predict 

func (l *LLama) Predict(text string, opts ...PredictOption) (string, error)

Predict generates text from the given prompt and returns the completed output. Configure sampling and stopping with PredictOption values; use SetTokenCallback to stream tokens as they are produced. It returns ErrOutOfMemory if the result buffer cannot be allocated, or ErrInference if generation fails.

func (*LLama) PredictResult 

func (l *LLama) PredictResult(text string, opts ...PredictOption) (string, int, error)

PredictResult generates a completion and returns the full generated text plus the number of tokens generated. The text is assembled in Go from the filtering sink (stop sequences trimmed, UTF-8 made whole), so it is not capped by any C buffer size. n is the number of tokens C produced (including any whose text the filter trimmed as part of a stop sequence).

func (*LLama) SaveState 

func (l *LLama) SaveState(dst string) error

SaveState writes the current context state to the given file.

Not yet implemented: it returns ErrNotImplemented. The underlying binding is a stub, so this call is a loud no-op rather than a silent one.

func (*LLama) SetTokenCallback 

func (l *LLama) SetTokenCallback(callback func(token string) bool)

SetTokenCallback registers a callback for the individual tokens created when running Predict. It will be called once for each token. The callback shall return true as long as the model should continue predicting the next token. When the callback returns false the predictor will return. The tokens are just converted into Go strings, they are not trimmed or otherwise changed. Also the tokens may not be valid UTF-8. Pass in nil to remove a callback.

It is save to call this method while a prediction is running.

func (*LLama) SpeculativeSampling 

func (l *LLama) SpeculativeSampling(ll *LLama, text string, opts ...PredictOption) (string, error)

SpeculativeSampling generates text from the target model (the receiver) using the given draft model to propose tokens.

Not yet implemented: it returns ErrNotImplemented. The underlying binding is a stub, so this fails loudly instead of returning an empty string.

func (*LLama) TokenEmbeddings 

func (l *LLama) TokenEmbeddings(tokens []int, opts ...PredictOption) ([]float32, error)

TokenEmbeddings returns the embedding vector for the given token IDs.

Not yet implemented: it returns ErrNotImplemented (or ErrEmbeddingsDisabled if the model was loaded without EnableEmbeddings). The underlying binding is a stub, so this fails loudly instead of returning an empty vector.

func (*LLama) TokenizeString 

func (l *LLama) TokenizeString(text string, opts ...PredictOption) (int32, []int32, error)

tokenize has an interesting return property: negative lengths (potentially) have meaning. Therefore, return the length seperate from the slice and error - all three can be used together

type ModelOption 

type ModelOption func(p *ModelOptions)
var EnabelLowVRAM ModelOption = func(p *ModelOptions) {
	p.LowVRAM = true
}
var EnableEmbeddings ModelOption = func(p *ModelOptions) {
	p.Embeddings = true
}
var EnableF16Memory ModelOption = func(p *ModelOptions) {
	p.F16Memory = true
}
var EnableMLock ModelOption = func(p *ModelOptions) {
	p.MLock = true
}
var EnableNUMA ModelOption = func(p *ModelOptions) {
	p.NUMA = true
}

func SetContext 

func SetContext(c int) ModelOption

SetContext sets the context size.

func SetGPULayers 

func SetGPULayers(n int) ModelOption

SetGPULayers sets the number of GPU layers to use to offload computation

func SetLoraAdapter 

func SetLoraAdapter(s string) ModelOption

func SetLoraBase 

func SetLoraBase(s string) ModelOption

func SetMMap 

func SetMMap(b bool) ModelOption

SetMMap enables or disables memory-mapping of the model file.

func SetMainGPU 

func SetMainGPU(maingpu string) ModelOption

SetMainGPU sets the main_gpu

func SetModelSeed 

func SetModelSeed(c int) ModelOption

func SetMulMatQ 

func SetMulMatQ(b bool) ModelOption

func SetNBatch 

func SetNBatch(n_batch int) ModelOption

SetNBatch sets the n_Batch

func SetPerplexity 

func SetPerplexity(b bool) ModelOption

func SetTensorSplit 

func SetTensorSplit(maingpu string) ModelOption

Set sets the tensor split for the GPU

func WithRopeFreqBase 

func WithRopeFreqBase(f float32) ModelOption

func WithRopeFreqScale 

func WithRopeFreqScale(f float32) ModelOption

type ModelOptions 

type ModelOptions struct {
	ContextSize   int
	Seed          int
	NBatch        int
	F16Memory     bool
	MLock         bool
	MMap          bool
	LowVRAM       bool
	Embeddings    bool
	NUMA          bool
	NGPULayers    int
	MainGPU       string
	TensorSplit   string
	FreqRopeBase  float32
	FreqRopeScale float32
	MulMatQ       *bool
	LoraBase      string
	LoraAdapter   string
	Perplexity    bool
}
var DefaultModelOptions ModelOptions = ModelOptions{
	ContextSize:   512,
	Seed:          0,
	F16Memory:     false,
	MLock:         false,
	Embeddings:    false,
	MMap:          true,
	LowVRAM:       false,
	NBatch:        512,
	FreqRopeBase:  10000,
	FreqRopeScale: 1.0,
}

func NewModelOptions 

func NewModelOptions(opts ...ModelOption) ModelOptions

Create a new PredictOptions object with the given options.

type PredictOption 

type PredictOption func(p *PredictOptions)
var Debug PredictOption = func(p *PredictOptions) {
	p.DebugMode = true
}
var EnableF16KV PredictOption = func(p *PredictOptions) {
	p.F16KV = true
}
var EnablePromptCacheAll PredictOption = func(p *PredictOptions) {
	p.PromptCacheAll = true
}
var EnablePromptCacheRO PredictOption = func(p *PredictOptions) {
	p.PromptCacheRO = true
}
var IgnoreEOS PredictOption = func(p *PredictOptions) {
	p.IgnoreEOS = true
}

func SetBatch 

func SetBatch(size int) PredictOption

SetBatch sets the batch size.

func SetFrequencyPenalty 

func SetFrequencyPenalty(fp float32) PredictOption

SetFrequencyPenalty sets the frequency penalty parameter, freq_penalty.

func SetLogitBias 

func SetLogitBias(lb string) PredictOption

SetLogitBias sets the logit bias parameter.

func SetMemoryMap 

func SetMemoryMap(b bool) PredictOption

SetMemoryMap sets memory mapping.

func SetMinP 

func SetMinP(minp float32) PredictOption

SetMinP sets the min-p sampling cutoff: tokens below this fraction of the top token's probability are dropped. 0 (the default) disables it.

func SetMirostat 

func SetMirostat(m int) PredictOption

SetMirostat sets the mirostat parameter.

func SetMirostatETA 

func SetMirostatETA(me float32) PredictOption

SetMirostatETA sets the mirostat ETA parameter.

func SetMirostatTAU 

func SetMirostatTAU(mt float32) PredictOption

SetMirostatTAU sets the mirostat TAU parameter.

func SetMlock 

func SetMlock(b bool) PredictOption

SetMlock sets the memory lock.

func SetNDraft 

func SetNDraft(nd int) PredictOption

func SetNKeep 

func SetNKeep(n int) PredictOption

SetKeep sets the number of tokens from initial prompt to keep.

func SetNegativePrompt 

func SetNegativePrompt(np string) PredictOption

func SetNegativePromptScale 

func SetNegativePromptScale(nps float32) PredictOption

func SetPathPromptCache 

func SetPathPromptCache(f string) PredictOption

SetPathPromptCache sets the session file to store the prompt cache.

func SetPenalizeNL 

func SetPenalizeNL(pnl bool) PredictOption

SetPenalizeNL is a no-op: newline penalization was folded into the unified penalties sampler upstream and is no longer wired as a standalone knob.

func SetPenalty 

func SetPenalty(penalty float32) PredictOption

SetPenalty sets the repetition penalty for text generation.

func SetPredictionMainGPU 

func SetPredictionMainGPU(maingpu string) PredictOption

SetPredictionMainGPU sets the main_gpu

func SetPredictionTensorSplit 

func SetPredictionTensorSplit(maingpu string) PredictOption

SetPredictionTensorSplit sets the tensor split for the GPU

func SetPresencePenalty 

func SetPresencePenalty(pp float32) PredictOption

SetPresencePenalty sets the presence penalty parameter, presence_penalty.

func SetRepeat 

func SetRepeat(repeat int) PredictOption

SetRepeat sets the number of times to repeat text generation.

func SetRopeFreqBase 

func SetRopeFreqBase(rfb float32) PredictOption

Rope and negative prompt parameters

func SetRopeFreqScale 

func SetRopeFreqScale(rfs float32) PredictOption

func SetSeed 

func SetSeed(seed int) PredictOption

SetSeed sets the random seed for sampling text generation.

func SetStopWords 

func SetStopWords(stop ...string) PredictOption

SetStopWords sets the prompts that will stop predictions.

func SetTailFreeSamplingZ 

func SetTailFreeSamplingZ(tfz float32) PredictOption

SetTailFreeSamplingZ is a no-op: tail-free sampling was removed from upstream llama.cpp's sampler API and is no longer wired into the chain.

func SetTemperature 

func SetTemperature(temp float32) PredictOption

SetTemperature sets the temperature value for text generation.

func SetThreads 

func SetThreads(threads int) PredictOption

SetThreads sets the number of threads to use for text generation.

func SetTokenCallback 

func SetTokenCallback(fn func(string) bool) PredictOption

SetTokenCallback sets the prompts that will stop predictions.

func SetTokens 

func SetTokens(tokens int) PredictOption

SetTokens sets the number of tokens to generate.

func SetTopK 

func SetTopK(topk int) PredictOption

SetTopK sets the value for top-K sampling.

func SetTopP 

func SetTopP(topp float32) PredictOption

SetTopP sets the value for nucleus sampling.

func SetTypicalP 

func SetTypicalP(tp float32) PredictOption

SetTypicalP sets the typicality parameter, p_typical.

func WithGrammar 

func WithGrammar(s string) PredictOption

WithGrammar sets the grammar to constrain the output of the LLM response

type PredictOptions 

type PredictOptions struct {
	Seed, Threads, Tokens, TopK, Repeat, Batch, NKeep int
	TopP, Temperature, Penalty                        float32
	NDraft                                            int
	F16KV                                             bool
	DebugMode                                         bool
	StopPrompts                                       []string
	IgnoreEOS                                         bool

	TailFreeSamplingZ float32
	TypicalP          float32
	MinP              float32
	FrequencyPenalty  float32
	PresencePenalty   float32
	Mirostat          int
	MirostatETA       float32
	MirostatTAU       float32
	PenalizeNL        bool
	LogitBias         string
	TokenCallback     func(string) bool

	PathPromptCache             string
	MLock, MMap, PromptCacheAll bool
	PromptCacheRO               bool
	Grammar                     string
	MainGPU                     string
	TensorSplit                 string

	// Rope parameters
	RopeFreqBase  float32
	RopeFreqScale float32

	// Negative prompt parameters
	NegativePromptScale float32
	NegativePrompt      string
}
var DefaultOptions PredictOptions = PredictOptions{
	Seed:              -1,
	Threads:           4,
	Tokens:            128,
	Penalty:           1.1,
	Repeat:            64,
	Batch:             512,
	NKeep:             64,
	TopK:              40,
	TopP:              0.95,
	TailFreeSamplingZ: 1.0,
	TypicalP:          1.0,
	Temperature:       0.8,
	FrequencyPenalty:  0.0,
	PresencePenalty:   0.0,
	Mirostat:          0,
	MirostatTAU:       5.0,
	MirostatETA:       0.1,
	MMap:              true,
	RopeFreqBase:      10000,
	RopeFreqScale:     1.0,
}

func NewPredictOptions 

func NewPredictOptions(opts ...PredictOption) PredictOptions

Create a new PredictOptions object with the given options.

Source Files

View all Source files

Directories

Path Synopsis
sampler-smoke command
smoketest command
Package gguf reads and parses GGUF model-file headers without loading tensor data.
 Package gguf reads and parses GGUF model-file headers without loading tensor data.
Package logitbias parses the user-facing logit-bias specification string ("<tokenID>:<bias>[,<tokenID>:<bias>...]") into token/bias pairs.
 Package logitbias parses the user-facing logit-bias specification string ("<tokenID>:<bias>[,<tokenID>:<bias>...]") into token/bias pairs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.