modelnexus

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 17, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package modelnexus runs a local LLM inside your own process. 

chat, err := modelnexus.Open("model.gguf")
defer chat.Close()

resp, err := chat.Infer(modelnexus.Request{
    Messages: []modelnexus.Message{{Role: "user", Content: "hello"}},
})
fmt.Println(resp.Text)

No server, no subprocess, no port -- and no cgo: the native bridge is loaded at runtime with purego, so CGO_ENABLED=0 and cross-compilation both keep working.

Index

Constants

View Source 
const BridgeVersion = "0.2.1"

BridgeVersion is the modelnexus C ABI this binding speaks.

It is part of the cache path, and that is not cosmetic. The natives release is keyed on the llama.cpp tag (ADR-0004), but the BRIDGE moves independently: 0.2.0 added entry points against the same llama.cpp b9371. Keyed on LlamaTag alone, a user who fetched 0.1.0 natives would keep a cache that looks valid forever and would never re-fetch — a new binding silently loading an old library.

View Source 
const FinishCancelled = "cancelled"

FinishCancelled is the FinishReason of a generation a consumer stopped.

A cancelled generation is a RESULT, not an error: the response carries the text produced so far and honest usage counts, because those tokens were really generated. Nothing here turns it into a Go error -- the caller decides.

View Source 
const LlamaTag = "b9371"

LlamaTag is the llama.cpp release this binding's native library is built against.

It must match core/build.sh's pin: the natives are published in a GitHub release keyed by this tag, and a mismatch means Fetch downloads a library the bindings were never tested with.

Variables

This section is empty.

Functions

func CacheDir 

func CacheDir() (string, error)

CacheDir reports where Fetch stores downloaded natives.

Honours MODELNEXUS_CACHE, else the OS user cache directory.

func Fetch 

func Fetch() (string, error)

Fetch downloads this platform's native library into the cache, if it is not already there, and returns the directory holding it.

Go is the one binding that does not receive the library from its package manager: a Go module is a source tree, and embedding five platforms' binaries would make every `go get` pull ~70 MB of libraries the user will not use (ADR-0007). So the library is fetched once, at the user's explicit request, rather than smuggled into the module.

The returned directory is also searched automatically on the next Open, so the usual shape is simply: 

if _, err := modelnexus.Fetch(); err != nil { ... }
chat, err := modelnexus.Open("model.gguf")

You do not have to pass the path anywhere.

Fetch is a convenience, not a requirement. Setting MODELNEXUS_LIB to a directory you already have — an air-gapped machine, a vendored copy, a build from core/build.sh — works and skips this entirely.

func PlatformKey 

func PlatformKey() string

PlatformKey is the os-arch key used for the staged native directory layout.

func ReuseCacheOff added in v0.2.0 

func ReuseCacheOff() *bool

ReuseCacheOff is a convenience for Request.ReuseCache, which needs an address.

func ReuseCacheOn added in v0.2.0 

func ReuseCacheOn() *bool

ReuseCacheOn is a convenience for Request.ReuseCache. It states the core's default explicitly, which is worth doing in a test that is about the flag.

func SetLogHandler 

func SetLogHandler(fn func(level LogLevel, text string)) error

SetLogHandler routes engine log output to fn instead of stderr. Passing nil restores stderr.

func SetLogLevel 

func SetLogLevel(level LogLevel) error

SetLogLevel sets how much the engine logs.

Call it before loading a model: llama.cpp starts logging during load, so afterwards is too late to silence it.

func Version 

func Version() (string, error)

Version reports the bridge version and the llama.cpp tag it was linked against.

Types

type Adapter 

type Adapter struct {
	ID    int     `json:"id"`
	Path  string  `json:"path"`
	Scale float64 `json:"scale"`
}

Adapter is one LoRA adapter applied to a Chat's context.

type CacheState added in v0.2.0 

type CacheState struct {
	// Tokens is how much of the cache is resident, AFTER the operation that reported
	// it -- so a clear always reports 0, and a caller can assert rather than assume.
	Tokens int `json:"tokens"`
	// NCtx is the engine's context window, so a caller can compare the two.
	NCtx int `json:"n_ctx"`
}

CacheState is what the engine's KV cache holds, and the window it holds it in.

type Chat 

type Chat struct {
	// contains filtered or unexported fields
}

Chat is a loaded model and its inference context. Close it when done.

func Open 

func Open(ggufPath string, opts ...Option) (*Chat, error)

Open loads a GGUF model and creates an inference engine.

Models whose chat template cannot do tool calling are rejected here rather than silently degraded -- a deliberate contract inherited from the core.

func (*Chat) CacheStatus added in v0.2.0 

func (c *Chat) CacheStatus() (CacheState, error)

CacheStatus reports what the engine's KV cache currently holds. It changes nothing.

func (*Chat) ClearCache added in v0.2.0 

func (c *Chat) ClearCache() (CacheState, error)

ClearCache drops the KV cache, freeing its memory and forgetting the sequence, and returns the state afterwards -- which is always zero tokens, so a caller can assert the release happened rather than trust that it did.

Prefix reuse is right for a conversation that appends and wrong when a Chat moves to unrelated work: the old conversation keeps occupying context memory, and two tenants sharing a handle would share a cache. Passing ReuseCacheOff() on the next inference also clears, but only as a side effect of doing work -- no help when the point is to release memory now, or to prove the cache is empty before handing the handle on.

func (*Chat) ClearLoRAs 

func (c *Chat) ClearLoRAs() error

ClearLoRAs unloads every adapter, returning the model to its base behaviour.

func (*Chat) Close 

func (c *Chat) Close() error

Close releases the model and its context. Safe to call more than once.

func (*Chat) CountTokens added in v0.2.0 

func (c *Chat) CountTokens(req Request) (TokenCount, error)

CountTokens reports what a request's messages will cost, without generating.

It applies the model's chat template and tokenizes; it creates no context, decodes nothing, and does not disturb the KV cache, so it is safe to call between inferences. Every generation parameter on req is ignored.

This is an ABI call rather than a Go helper because counting needs both the model's vocabulary and its parsed chat template, and a binding holds neither.

func (*Chat) Infer 

func (c *Chat) Infer(req Request) (*Response, error)

Infer runs one turn and returns the response.

func (*Chat) InferContext added in v0.2.0 

func (c *Chat) InferContext(ctx context.Context, req Request) (*Response, error)

InferContext is Infer, stopped by cancelling ctx.

The ABI's only cancellation mechanism is the token callback's return value, so ctx is checked once per decoded token: cancellation takes effect before the NEXT token, not mid-token. A cancelled run returns a normal response with FinishCancelled and a nil error -- the tokens were really generated and the usage counts are real, and throwing that away to return ctx.Err() would lose the one thing the caller was billed for. Check Response.Cancelled, or ctx.Err(), to tell the two apart.

func (*Chat) InferStream 

func (c *Chat) InferStream(req Request, onToken func(piece string) bool) (*Response, error)

InferStream runs one turn, calling onToken with each decoded piece as it is produced. The complete response is still returned when generation finishes.

onToken returns whether to KEEP GOING: return false to stop generation, which ends the turn with FinishCancelled and a complete response carrying the text produced so far. Stopping is a result, not an error.

func (*Chat) InferStreamContext added in v0.2.0 

func (c *Chat) InferStreamContext(ctx context.Context, req Request, onToken func(piece string) bool) (*Response, error)

InferStreamContext is InferStream, additionally stopped by cancelling ctx. Either the callback returning false or ctx being cancelled ends generation.

func (*Chat) LoRAs 

func (c *Chat) LoRAs() ([]Adapter, error)

LoRAs reports the adapters currently applied, in order.

func (*Chat) LoadLoRA 

func (c *Chat) LoadLoRA(path string, scale float64) (int, error)

LoadLoRA loads a LoRA adapter and applies it, returning its id.

Several adapters can be active at once; they apply in load order. Adapters change *behaviour* -- output format, tone, tool-call reliability -- not knowledge. For facts, retrieve.

func (*Chat) RemoveLoRA 

func (c *Chat) RemoveLoRA(id int) error

RemoveLoRA unloads one adapter and reapplies the rest.

func (*Chat) SetLoRAScale 

func (c *Chat) SetLoRAScale(id int, scale float64) error

SetLoRAScale changes an adapter's scale. It takes effect on the next inference.

type EmbedOptions 

type EmbedOptions struct {
	// Pooling strategy. Empty means the model's own default. Use PoolingRank for a
	// reranker -- Rerank refuses to run without it.
	Pooling Pooling
	// NCtx is the context size; 0 means the model default.
	NCtx int
	// NBatch caps how many tokens one input may have; 0 means the core's default
	// (llb_embed_create in core/include/llamabridge.h states it).
	NBatch int
	// OnEvent receives lifecycle events during load.
	OnEvent func(string)
}

EmbedOptions configures OpenEmbedder.

type Embedder 

type Embedder struct {
	// contains filtered or unexported fields
}

Embedder is a model loaded for embedding or reranking.

Separate from Chat on purpose: embedding needs a context created with embeddings enabled and a pooling type fixed up front, and reranking needs PoolingRank specifically -- none of which can be switched on a generation context afterwards.

func OpenEmbedder 

func OpenEmbedder(ggufPath string, opts *EmbedOptions) (*Embedder, error)

OpenEmbedder loads a model for embedding or reranking.

func (*Embedder) Close 

func (e *Embedder) Close() error

Close releases the model and its context. Safe to call more than once.

func (*Embedder) Embed 

func (e *Embedder) Embed(texts []string) ([][]float32, error)

Embed returns one vector per input, in input order.

Vectors are L2-normalized, which is what makes a dot product a cosine similarity.

func (*Embedder) EmbedRaw 

func (e *Embedder) EmbedRaw(texts []string) ([][]float32, error)

EmbedRaw is Embed without normalization.

func (*Embedder) Rerank 

func (e *Embedder) Rerank(query string, documents []string, topN int) ([]RerankHit, error)

Rerank scores documents against a query, best first.

topN <= 0 returns every document. Requires a reranker model opened with PoolingRank; otherwise this fails with POOLING_NOT_RANK rather than returning numbers that look like scores and are not.

type ErrNativeLibraryNotFound 

type ErrNativeLibraryNotFound struct {
	Searched []string
	Cause    error
}

ErrNativeLibraryNotFound reports that the native bridge could not be located or loaded.

Go is the one binding that does not receive the shared library from a package manager -- purego resolves it at runtime -- so this failure mode is unique to Go and is surfaced as a distinct typed error rather than a panic. See ADR-0002 and the model-core spec.

func (*ErrNativeLibraryNotFound) Error 

func (e *ErrNativeLibraryNotFound) Error() string

func (*ErrNativeLibraryNotFound) Unwrap 

func (e *ErrNativeLibraryNotFound) Unwrap() error

type Error 

type Error struct {
	Code    string `json:"code"`
	Message string `json:"message"`
}

Error is a failure reported by the core.

Code is stable and identical across every language binding; Message is for humans.

func (*Error) Error 

func (e *Error) Error() string

type LogLevel 

type LogLevel int32

LogLevel is how much the inference engine is allowed to say.

The bridge defaults to LogWarn rather than llama.cpp's own default: a library embedded in someone else's process should be quiet unless asked. 

const (
	// LogNone silences the engine entirely.
	LogNone  LogLevel = 0
	LogDebug LogLevel = 1
	LogInfo  LogLevel = 2
	// LogWarn is the default.
	LogWarn  LogLevel = 3
	LogError LogLevel = 4
)

type Message 

type Message struct {
	Role       string     `json:"role"`
	Content    string     `json:"content"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
}

Message is one turn of a conversation.

type ModelInfo 

type ModelInfo struct {
	SupportsTools      bool   `json:"supports_tools"`
	SupportsToolCalls  bool   `json:"supports_tool_calls"`
	HasToolUseTemplate bool   `json:"has_tool_use_template"`
	ChatFormat         string `json:"chat_format"`
	Error              string `json:"error"`
}

ModelInfo reports a model's tool-calling capability.

func Info 

func Info(ggufPath string) (ModelInfo, error)

Info inspects a GGUF's tool-calling capability without loading an engine.

Cheap enough to call before committing to a multi-gigabyte load, which is why the core exposes it separately from Open.

type Option 

type Option func(*openConfig)

Option configures Open.

func WithBatchSize added in v0.2.0 

func WithBatchSize(n int) Option

WithBatchSize sets the logical batch size. Unset means the core's default.

func WithContextSize added in v0.2.0 

func WithContextSize(n int) Option

WithContextSize sets the engine's context window in tokens. Unset means the core's default (llb_chat_create in core/include/llamabridge.h states it).

This is create-time: it is the size of the KV cache, and it cannot be changed on a live engine.

func WithEventHandler 

func WithEventHandler(fn func(string)) Option

WithEventHandler receives the core's progress events during load and inference.

func WithMaxSequences added in v0.2.0 

func WithMaxSequences(n int) Option

WithMaxSequences reserves room for n concurrent sequences. Unset means the core's default.

It has no observable effect today. It is accepted now because it is a create-time parameter -- the one kind that cannot be added later through request JSON -- so reserving it is what lets multi-sequence slots arrive without an ABI break (ADR-0008 D6).

type Pooling 

type Pooling string

Pooling is how token vectors are reduced to one vector per input. 

const (
	// PoolingMean averages token vectors -- the usual choice for sentence embeddings.
	PoolingMean Pooling = "mean"
	// PoolingCLS takes the first token's vector, used by BERT-style encoders.
	PoolingCLS Pooling = "cls"
	// PoolingLast takes the final token's vector, used by decoder-style embedders.
	PoolingLast Pooling = "last"
	// PoolingRank attaches the model's classification head. Required for Rerank,
	// and useless for anything else.
	PoolingRank Pooling = "rank"
	// PoolingNone returns no pooled vector at all.
	PoolingNone Pooling = "none"
)

type Request 

type Request struct {
	Messages   []Message `json:"messages"`
	Tools      []Tool    `json:"tools,omitempty"`
	ToolChoice string    `json:"tool_choice,omitempty"`

	Temperature   *float64 `json:"temperature,omitempty"`
	TopK          *int     `json:"top_k,omitempty"`
	TopP          *float64 `json:"top_p,omitempty"`
	MinP          *float64 `json:"min_p,omitempty"`
	MaxTokens     *int     `json:"max_tokens,omitempty"`
	RepeatPenalty *float64 `json:"repeat_penalty,omitempty"`
	Seed          *uint32  `json:"seed,omitempty"`
	Stop          []string `json:"stop,omitempty"`

	// JSONSchema constrains the output to a JSON Schema. The returned Text is
	// guaranteed to parse as JSON: the grammar llama.cpp derives from a schema
	// deliberately permits a “`json fence, and the core strips it before returning.
	//
	// Any value that marshals to a JSON Schema object works -- but PREFER
	// json.RawMessage, and read the next paragraph before reaching for a map.
	//
	// encoding/json SORTS map keys, so a schema built as map[string]any reaches the
	// model with its properties in alphabetical order. Under grammar-constrained
	// decoding property order is load-bearing: the model must emit the fields in the
	// order the grammar allows, so it commits to a "rating" before it has reasoned out
	// a "sentiment" it would otherwise have written first. This is not theoretical --
	// the same prompt at the same seed returned rating 2 from Go and rating 3 from
	// Python and JS, purely because Go had reordered the two properties. The output
	// still parses, still validates, and is quietly a different answer.
	//
	// json.RawMessage preserves the order you wrote:
	//
	//	JSONSchema: json.RawMessage(`{
	//	    "type": "object",
	//	    "properties": {
	//	        "sentiment": {"type": "string"},
	//	        "rating":    {"type": "integer"}
	//	    },
	//	    "required": ["sentiment", "rating"]
	//	}`)
	//
	// A struct with ordered fields works too; a map does not, and cannot be made to.
	JSONSchema any `json:"json_schema,omitempty"`

	// Grammar constrains the output to a raw GBNF grammar.
	//
	// Setting this together with JSONSchema is rejected by the core with
	// INVALID_REQUEST rather than resolved by precedence: a silent winner between
	// two output constraints is a debugging session nobody should have.
	Grammar string `json:"grammar,omitempty"`

	// ReuseCache controls KV prefix reuse across calls on the same Chat. The core's
	// default is true, which is why this is a pointer -- leaving it nil means "the
	// core decides", and a plain bool would silently opt every caller out.
	//
	// Reuse is purely a latency property; output is identical either way. Set it to
	// false when calls must be provably independent (a determinism harness, or
	// tenants sharing one handle).
	ReuseCache *bool `json:"reuse_cache,omitempty"`
}

Request is one inference call.

Generation parameters are pointers so that "unset" is distinguishable from the zero value -- a Temperature of 0 is a legitimate, and very different, request from "use the default".

type RerankHit 

type RerankHit struct {
	Index int     `json:"index"`
	Score float64 `json:"score"`
}

RerankHit is one scored document.

Index is the document's position in the ORIGINAL slice, because results come back reordered. Score is a raw model logit: comparable within one call, not across models, and not a probability.

type Response 

type Response struct {
	Type         string     `json:"type"`
	Text         string     `json:"text"`
	ToolCalls    []ToolCall `json:"tool_calls"`
	FinishReason string     `json:"finish_reason"`
	Usage        Usage      `json:"usage"`
}

Response is the result of one inference call.

func (*Response) Cancelled added in v0.2.0 

func (r *Response) Cancelled() bool

Cancelled reports whether generation was stopped early by the token callback or a cancelled context.

type TokenCount added in v0.2.0 

type TokenCount struct {
	// Tokens is the prompt length after the model's chat template is applied.
	Tokens int `json:"tokens"`
	// NCtx is the engine's context window, so a caller can compare the two.
	NCtx int `json:"n_ctx"`
}

TokenCount is what a message list will cost, without running inference.

type Tool 

type Tool struct {
	Type     string       `json:"type"`
	Function ToolFunction `json:"function"`
}

Tool is a function the model may call, in OpenAI's schema shape.

type ToolCall 

type ToolCall struct {
	ID        string `json:"id"`
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCall is a call the model proposes. It is never executed here -- modelnexus emits OpenAI-shaped tool calls and stops. Executing them is the caller's job (ADR-0003).

type ToolFunction 

type ToolFunction struct {
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	Parameters  any    `json:"parameters,omitempty"`
}

ToolFunction describes one callable.

type Usage 

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

Usage is the token accounting for one call.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.