llamacpp

README

llamacpp

This project lets you use Go for hardware accelerated local inference with llama.cpp directly integrated into your applications. It provides a high level API based on the yzma module.

Copyright 2025 Ardan Labs
hello@ardanlabs.com

My Information

Name:    Bill Kennedy
Company: Ardan Labs
Title:   Managing Partner
Email:   bill@ardanlabs.com
Twitter: goinggodotnet

Examples

You can find examples in the ArdanLabs AI training repo at example13:

https://github.com/ardanlabs/ai-training/tree/main/cmd/examples/example13

Documentation

Overview

Package llamacpp provides support for working with models using llamacpp.

Index

• func InstallLlama(libPath string, processor download.Processor, allowUpgrade bool) error
• func InstallModel(modelURL string, modelPath string) (string, error)
• func WithProjection(projFile string) func(m *model) error
• type ChatMessage
• type ChatResponse
• type Config
• type Llama
  • func New(concurrency int, libPath string, modelFile string, cfg Config, ...) (*Llama, error)
  • func (llm *Llama) ChatCompletions(ctx context.Context, messages []ChatMessage, params Params) (<-chan ChatResponse, error)
  • func (llm *Llama) ChatVision(ctx context.Context, message ChatMessage, imageFile string, params Params) (<-chan ChatResponse, error)
  • func (llm *Llama) Embed(ctx context.Context, text string) ([]float32, error)
  • func (llm *Llama) ModelInfo(ctx context.Context) (ModelInfo, error)
  • func (llm *Llama) ModelName() string
  • func (llm *Llama) Rerank(rankingDocs []RankingDocument) ([]Ranking, error)
  • func (llm *Llama) Unload()
• type LogType
• type ModelInfo
• type Params
• type Ranking
• type RankingDocument

Functions

func InstallLlama

func InstallLlama(libPath string, processor download.Processor, allowUpgrade bool) error

func InstallModel

func InstallModel(modelURL string, modelPath string) (string, error)

func WithProjection

func WithProjection(projFile string) func(m *model) error

Types

type ChatMessage

type ChatMessage struct {
	Role    string
	Content string
}

ChatMessage represent input for chat and vision models.

type ChatResponse

type ChatResponse struct {
	Response string
	Err      error
}

ChatResponse represents output for chat and vision models.

type Config

type Config struct {
	LogSet        LogType
	ContextWindow uint32
	Embeddings    bool
}

Config represents model level configuration.

type Llama

type Llama struct {
	// contains filtered or unexported fields
}

Llama represents a concurrency group of a specified model.

func New

func New(concurrency int, libPath string, modelFile string, cfg Config, options ...func(llg *model) error) (*Llama, error)

New provides the ability to use models in a concurrently safe way.

func (*Llama) ChatCompletions

func (llm *Llama) ChatCompletions(ctx context.Context, messages []ChatMessage, params Params) (<-chan ChatResponse, error)

ChatCompletions provides support to interact with an inference model. It will block until a model becomes available or the context times out.

func (*Llama) ChatVision

func (llm *Llama) ChatVision(ctx context.Context, message ChatMessage, imageFile string, params Params) (<-chan ChatResponse, error)

ChatVision provides support to interact with a vision language model. It will block until a model becomes available or the context times out.

func (*Llama) Embed

func (llm *Llama) Embed(ctx context.Context, text string) ([]float32, error)

Embed provides support to interact with an embedding model. It will block until a model becomes available or the context times out.

func (*Llama) ModelInfo

func (llm *Llama) ModelInfo(ctx context.Context) (ModelInfo, error)

ModelInfo provides support to extract the model card information.

func (*Llama) ModelName

func (llm *Llama) ModelName() string

ModelName returns the model name.

func (*Llama) Rerank

func (llm *Llama) Rerank(rankingDocs []RankingDocument) ([]Ranking, error)

Rerank provides support to rerank a set of embeddings.

func (*Llama) Unload

func (llm *Llama) Unload()

Unload will close down all loaded models. You should call this only when you are completely done using the group.

type LogType

type LogType int

const (
	LogSilent LogType = iota + 1
	LogNormal
)

type ModelInfo

type ModelInfo struct {
	Desc        string
	Size        uint64
	HasEncoder  bool
	HasDecoder  bool
	IsRecurrent bool
	IsHybrid    bool
	Metadata    map[string]string
}

ModelInfo represents the model's card information.

type Params

type Params struct {
	TopK int32
	TopP float32
	Temp float32
}

Params represents the different sample options when using a model.

type Ranking

type Ranking struct {
	Document string
	Score    float64
}

Ranking represents output for reranking.

type RankingDocument

type RankingDocument struct {
	Document  string
	Embedding []float64
}

RankingDocument represents input for reranking.