litertlmgo

README

litertlm-go

A Go binding for Google's LiteRT-LM for high-performance local on-device LLM inference.

Inspired by Hybridgroup's Yzma.

📖 Full documentation: vladimirvivien.github.io/litertlm-go

Features

• Client — single-call Generate, range-over-func GenerateStream, rich GenerateResponse. Functional options for every engine setting. Context-driven cancellation. → Client guide
• Chat — multi-turn conversations with system prompts, tool declarations, and structured tool_calls parsing. → Chat guide
• GenerateData[T] — generic helper that returns *T populated from the model's JSON output, with retry and tolerant parsing. → Structured output
• Low-level API — every C-API symbol exposed as a Go method. Useful when you need explicit prefill→decode, scoring, token introspection, or deterministic resource lifetimes. → Low-level guide

Install

go get github.com/vladimirvivien/litertlm-go@latest

Quickstart

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/vladimirvivien/litertlm-go/pkg/litertlm"
)

func main() {
    ctx := context.Background()
    client, err := litertlm.New(ctx,
        litertlm.WithLib(os.Getenv("LITERTLM_LIB")),
        litertlm.WithModel(os.Getenv("LITERTLM_MODEL")),
    )
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
    defer client.Close()

    text, err := client.Generate(ctx, "Write a haiku about the sea.")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
    fmt.Println(text)
}

LITERTLM_LIB=/abs/path/to/dist/lib \
LITERTLM_MODEL=/abs/path/to/gemma-4-E2B-it.litertlm \
    go run main.go

Full walkthrough → docs/getting-started.

Building the C library

Currently, LiteRT-LM doesn't ship a prebuilt C API shared libraries. The followings walk you through how to build them:

• Linux / macOS — LITERTLM-BUILD.md
• Windows — LITERTLM-BUILD-WINDOWS.md

Examples

Path	What it shows
examples/hello/	Minimal Generate
examples/stream/	GenerateStream with range-over-func
examples/chat/	Multi-turn Chat with a system prompt
examples/conversation/	Chat + tools + structured tool_calls
examples/structured/	GenerateData[T] (typed JSON output via reflection)
examples/cancel/	Cancelling a streaming generation via context.WithCancel
examples/prefill-decode/	Explicit two-phase generation (low-level)
examples/score/	ScoreTexts + Score / TokenLength (low-level)
examples/tokenize/	Engine.Tokenize / Detokenize + start/stop tokens (low-level)
examples/gpu/	GPU-backed generation + benchmark metrics

Documentation

• Getting started
• Client — New, Generate, GenerateStream, GenerateResponse
• Chat — multi-turn, system prompts, tool calling
• Structured output — GenerateData[T]
• Low-level API — when to drop down
• Troubleshooting

License

Apache-2.0, same as LiteRT-LM itself.

Documentation

Overview

Package litertlmgo is the module root for litertlm-go.

Consumers import the actual Go API from the subpackages:

import "github.com/vladimirvivien/litertlm-go/pkg/litertlm"
import "github.com/vladimirvivien/litertlm-go/pkg/loader"

See README.md for the full usage guide and LITERTLM-BUILD.md for the native-library build recipe.