litertlmgo

README

litertlm-go

A high-performance, cgo-free, purego-backed Go binding for Google's LiteRT-LM runtime, designed for local on-device LLM inference.

[!NOTE] Inspired by Hybridgroup's Yzma.

📖 Full documentation: vladimirvivien.github.io/litertlm-go

---

🔥 Features

• 💬 Stateful Chat & Conversations — Multi-turn chat orchestration with system prompts, message history, and token tracking.
• 🖼️ Multimodal Inputs — Process text, images, and audio inputs in any order using a unified Go interface.
• 🛠️ Automated Tool Calling — Register standard Go functions as tools that the model can call automatically and in parallel.
• 🎯 Structured JSON Output — Extract model outputs directly into Go structs with type-safe generic helpers.
• ⚡ High Performance & GPU Acceleration — Run on CPU or GPU (WebGPU/Direct3D 12) with built-in memory safety.
• 🔍 Low-Level Control — Access the full LiteRT-LM C-API when you need custom inference loops, scoring, or custom memory management.

---

⚙️ Install

go get github.com/vladimirvivien/litertlm-go@latest

---

🚀 Quickstart

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/vladimirvivien/litertlm-go/pkg/litertlm"
)

func main() {
	ctx := context.Background()
	
	// Initialize client using compiled libraries and model
	client, err := litertlm.New(ctx,
		litertlm.WithLib(os.Getenv("LITERTLM_LIB")),
		litertlm.WithModel(os.Getenv("LITERTLM_MODEL")),
	)
	if err != nil {
		fmt.Printf("Initialization failed: %v\n", err)
		os.Exit(1)
	}
	defer client.Close()

	// Generate text
	text, err := client.Generate(ctx, "Write a haiku about the sea.")
	if err != nil {
		fmt.Printf("Generation failed: %v\n", err)
		os.Exit(1)
	}
	fmt.Println(text)
}

Run with your compiled libraries and model paths:

LITERTLM_LIB=/abs/path/to/dist/lib \
LITERTLM_MODEL=/abs/path/to/gemma-4-E2B-it.litertlm \
    go run main.go

[!TIP] To run on the GPU backend, configure your initialization with litertlm.WithBackend("gpu") (defaults to "cpu").

---

🏗️ Building the C Library

LiteRT-LM does not ship prebuilt C-API shared libraries. Build the staging libraries (litertlm_c.dll / litertlm_c_cpu.dll) for your platform using the guides below:

• Linux / macOS — LITERTLM-BUILD.md
• Windows — LITERTLM-BUILD-WINDOWS.md

[!IMPORTANT] Minimum supported upstream: LiteRT-LM v0.13.1. Stateful token tracking via Chat.TokenCount() binds to litert_lm_conversation_get_token_count, which was introduced in v0.13.1.

---

🤖 Examples

The repository includes a variety of examples showcasing advanced features. For example, ./examples/bot demonstrates the OCR capabilities of the Gemma 4 family, extracting structured information from an image of a form:

See examples/README.md for a full list of available examples.

---

📖 Documentation

• Getting Started — Installation and library setup
• Client Guide — New, Generate, and unary/multimodal configurations
• Chat Guide — Multi-turn history, system prompts, and tool calling
• Structured Output — JSON extraction via GenerateData[T]
• Low-Level API — Custom inference, scoring, and memory management
• Supported Models — Supported model families and template processor mappings
• Troubleshooting — Debugging load, memory, and acceleration issues

---

📄 License

Apache-2.0, same as LiteRT-LM itself.

Documentation

Overview

Package litertlmgo is the module root for litertlm-go.

Consumers import the actual Go API from the subpackages:

import "github.com/vladimirvivien/litertlm-go/pkg/litertlm"
import "github.com/vladimirvivien/litertlm-go/pkg/loader"

See README.md for the full usage guide and LITERTLM-BUILD.md for the native-library build recipe.