ortgenai

README

ortgenai

Go bindings for the ONNX Runtime GenAI C API.

This package provides a thin, idiomatic Go wrapper around the ONNX Runtime GenAI shared library, exposing:

• Session and model creation from local model folders
• Tokenization and chat templating
• Batched, streaming text generation with per-token deltas
• Runtime statistics (tokens/sec, prefill timings)
• Provider selection and advanced provider options
• Multimodal input support (text + images)

Note: The current implementation loads the GenAI shared library via dlopen, and targets Linux (ELF .so).

Note: This implementation is still alpha so the API may change in future releases. You might want to rely on hugot for a higher-level interface to ONNX Runtime GenAI in Go.

Contents

• Requirements
• Installation
• Quick start
• Advanced usage
• Running tests
• Docker and containerized tests
• Troubleshooting
• License

Requirements

• Go 1.19+
• Linux with glibc (uses dlfcn.h and .so loading)
• ONNX Runtime GenAI shared library and dependencies available at runtime:
  • libonnxruntime-genai.so
  • libonnxruntime.so (must be available in the same directory as libonnxruntime-genai.so)
• A local model directory compatible with ONNX Runtime GenAI (e.g., a converted Phi-3.5 model folder)

Installation

go get github.com/knights-analytics/ortgenai

At runtime, the wrapper needs to dlopen the ONNX Runtime GenAI shared library. You can:

1. Place libonnxruntime-genai.so next to your application binary (with libonnxruntime.so in the same folder), or
2. Call genai.SetSharedLibraryPath("/path/to/libonnxruntime-genai.so") before initialization.

Quick start

package main

import (
    "context"
    "fmt"
    "time"

    genai "github.com/knights-analytics/ortgenai"
)

func main() {
    // Optional: set explicit path if the .so isn't on the default loader path
    // Note: ensure libonnxruntime.so is colocated with libonnxruntime-genai.so
    genai.SetSharedLibraryPath("/usr/lib/libonnxruntime-genai.so")

    if err := genai.InitializeEnvironment(); err != nil {
        panic(fmt.Errorf("init failed: %w", err))
    }
    defer func() {
        err = genai.DestroyEnvironment()
        if err != nil {
            panic(fmt.Errorf("destroy environment: %w", err))
        }
    }()

    // Point to a local model folder compatible with ONNX Runtime GenAI
    session, err := genai.CreateSession("./models/phi3.5")
    if err != nil {
        panic(fmt.Errorf("create session: %w", err))
    }
    defer session.Destroy()

    // Prepare one or more conversations (batched)
    conv1 := []genai.Message{
        {Role: "system", Content: "You are a helpful assistant."},
        {Role: "user", Content: "What is the capital of France?"},
    }

    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()

    deltas, errs, err := session.Generate(ctx, [][]genai.Message{conv1})
    if err != nil {
        panic(fmt.Errorf("generate: %w", err))
    }

    // Stream tokens as they arrive
    for {
        select {
        case d, ok := <-deltas:
            if !ok { // stream completed
                stats := session.GetStatistics()
                fmt.Printf("\nTokens/sec: %.2f\n", stats.TokensPerSecond)
                return
            }
            fmt.Print(d.Tokens) // append to your buffer
        case e := <-errs:
            if e != nil { panic(e) }
        }
    }
}

Advanced usage

Explicit shared library path

genai.SetSharedLibraryPath("/opt/onnxruntime/lib/libonnxruntime-genai.so")
if err := genai.InitializeEnvironment(); err != nil { /* handle */ }

If not set, the code tries libonnxruntime-genai.so relative to the loader’s search path. Ensure libonnxruntime.so is colocated with the GenAI .so.

Provider selection and options

If you have a JSON config path and custom providers to use (e.g., CUDA, CPU) you can create a session with advanced settings:

providers := []string{"cpu"} // or e.g., []string{"cuda", "cpu"}
providerOptions := map[string]map[string]string{
    "cpu": {"intra_op_num_threads": "4"},
}

session, err := genai.CreateSessionWithOptions(
    "./path/to/config.json", // model/session config from GenAI tooling
    providers,
    providerOptions,
)

Batched generation

Session.Generate accepts multiple conversations in one call: [][]Message. The returned channel carries SequenceDelta items, each labeled with the Sequence index so you can route output per-conversation.

Statistics

After generation, inspect session.GetStatistics() for fields such as TokensPerSecond, cumulative token counts, and prefill timings.

Running tests

Local tests require the GenAI shared libraries and a local model directory. The provided unit test expects:

• libonnxruntime-genai.so available (by default at /usr/lib/libonnxruntime-genai.so in the test; adjust via SetSharedLibraryPath), and
• a model directory at ./_models/phi3.5 (update the path as needed).

Run:

go test ./...

Docker and containerized tests

Two Dockerfiles are provided:

• Dockerfile — base image with dependencies
• test.Dockerfile — image to run unit tests

Helper scripts are available in scripts/:

• scripts/run-unit-tests-container.sh — build the test image and run tests in a container
• scripts/run-unit-test.sh — run tests directly (expects environment to be prepared)

You may also use compose-test.yaml to orchestrate test runs.

Troubleshooting

• error loading GenAI shared library: Ensure the path to libonnxruntime-genai.so is correct and readable by the process. Set it explicitly with SetSharedLibraryPath.
• missing Oga... symbols or "missing Oga..." errors: The GenAI .so must export required symbols (e.g., OgaCreateModel). Make sure versions of libonnxruntime-genai.so and libonnxruntime.so are compatible and colocated.
• segmentation fault on load: Verify that your system’s CUDA/CPU provider dependencies match the .so build (driver/runtime versions).
• no output / stuck: Ensure your model folder is valid for ONNX Runtime GenAI and accessible; increase timeouts during first-run warmup.

License

This project is licensed under the terms of the MIT License. See LICENSE for details.

Documentation

Index

• Variables
• func DestroyEnvironment() error
• func InitializeEnvironment() error
• func InitializeGenAiLibrary() error
• func IsInitialized() bool
• func OgaResultToError(result *C.OgaResult) error
• func SetSharedLibraryPath(path string)
• type GenerationOptions
• type Guidance
• type GuidanceType
• type Images
  • func LoadImage(imagePath string) (*Images, error)
  • func LoadImageFromBuffer(imageData []byte) (*Images, error)
  • func LoadImages(imagePaths []string) (*Images, error)
  • func LoadImagesFromBuffers(imageBuffers [][]byte) (*Images, error)
  • func (i *Images) Destroy()
• type MaxLengthReachedError
  • func (e MaxLengthReachedError) Error() string
• type Message
• type NamedTensors
  • func (nt *NamedTensors) Destroy()
• type SequenceDelta
• type Session
  • func CreateSession(modelPath string) (*Session, error)
  • func CreateSessionWithOptions(configDirectoryPath string, providers []string, ...) (*Session, error)
  • func (s *Session) Destroy()
  • func (s *Session) Generate(ctx context.Context, messages [][]Message, tools []string, ...) (<-chan SequenceDelta, <-chan error, error)
  • func (s *Session) GenerateWithImages(ctx context.Context, messages [][]Message, images *Images, tools []string, ...) (<-chan SequenceDelta, <-chan error, error)
  • func (s *Session) GetStatistics() *Statistics
• type Statistics

Variables

var ErrNotInitialized = fmt.Errorf("InitializeEnvironment() has either " +
	"not yet been called, or did not return successfully")

Functions

func DestroyEnvironment

func DestroyEnvironment() error

DestroyEnvironment Call this function to clean up the internal onnxruntime environment when it is no longer required.

func InitializeEnvironment

func InitializeEnvironment() error

func InitializeGenAiLibrary

func InitializeGenAiLibrary() error

InitializeGenAiLibrary loads the ONNX Runtime GenAI shared library specified by onnxGenaiSharedLibraryPath (or a default) so its exported symbols become available. The assumption is that libonnxruntime.so is available in the same folder where the libonnxruntime-genai.so is located.

func IsInitialized

func IsInitialized() bool

func OgaResultToError

func OgaResultToError(result *C.OgaResult) error

func SetSharedLibraryPath

func SetSharedLibraryPath(path string)

Types

type GenerationOptions

type GenerationOptions struct {
	MaxLength   int
	BatchSize   int
	Temperature *float64
	TopP        *float64
	Seed        *int
	Guidance    *Guidance
}

type Guidance

type Guidance struct {
	Type GuidanceType
	Data string
	// EnableFFTokens speeds up generation by force-forwarding tokens that satisfy the grammar
	// without calling the model. Only valid when BatchSize=1 and beam_size=1.
	EnableFFTokens bool
}

Guidance configures constrained (guided) generation. Requires a recent OGA runtime.

type GuidanceType

type GuidanceType string

GuidanceType specifies the constrained-generation strategy passed to OgaGeneratorParamsSetGuidance.

const (
	GuidanceTypeJSONSchema  GuidanceType = "json_schema"
	GuidanceTypeRegex       GuidanceType = "regex"
	GuidanceTypeLarkGrammar GuidanceType = "lark_grammar"
)

type Images

type Images struct {
	// contains filtered or unexported fields
}

Images represents a collection of loaded images for multimodal processing.

func LoadImage

func LoadImage(imagePath string) (*Images, error)

LoadImage loads a single image from a file path or data URI.

func LoadImageFromBuffer

func LoadImageFromBuffer(imageData []byte) (*Images, error)

LoadImageFromBuffer loads a single image from a byte buffer.

func LoadImages

func LoadImages(imagePaths []string) (*Images, error)

LoadImages loads multiple images from file paths or data URIs.

func LoadImagesFromBuffers

func LoadImagesFromBuffers(imageBuffers [][]byte) (*Images, error)

LoadImagesFromBuffers loads multiple images from byte buffers.

func (*Images) Destroy

func (i *Images) Destroy()

Destroy releases the images resources.

type MaxLengthReachedError

type MaxLengthReachedError struct{}

func (MaxLengthReachedError) Error

func (e MaxLengthReachedError) Error() string

type Message

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

type NamedTensors

type NamedTensors struct {
	// contains filtered or unexported fields
}

NamedTensors represents a collection of named tensor inputs.

func (*NamedTensors) Destroy

func (nt *NamedTensors) Destroy()

Destroy releases the named tensors resources.

type SequenceDelta

type SequenceDelta struct {
	Sequence   int
	Token      string
	EOSReached bool
}

type Session

type Session struct {
	// contains filtered or unexported fields
}

func CreateSession

func CreateSession(modelPath string) (*Session, error)

func CreateSessionWithOptions

func CreateSessionWithOptions(configDirectoryPath string, providers []string, providerOptions map[string]map[string]string) (*Session, error)

CreateSessionWithOptions builds a GenAI config from a config directory, applies execution providers and options, creates the model and tokenizer, and returns a Session. providers: list of EP names in priority order (e.g., ["cuda"], ["NvTensorRtRtx"], ["OpenVINO"]). providerOptions: map of EP name -> map of key/value options.

func (*Session) Destroy

func (s *Session) Destroy()

func (*Session) Generate

func (s *Session) Generate(ctx context.Context, messages [][]Message, tools []string, generationOptions *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

func (*Session) GenerateWithImages

func (s *Session) GenerateWithImages(ctx context.Context, messages [][]Message, images *Images, tools []string, generationOptions *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

GenerateWithImages generates text using pre-processed named tensors (for multimodal inputs). Currently only supports a single prompt.

func (*Session) GetStatistics

func (s *Session) GetStatistics() *Statistics

GetStatistics returns the last computed statistics for the session.

type Statistics

type Statistics struct {
	AvgPrefillSeconds float64
	TokensPerSecond   float64
	// cumulative
	CumulativePrefillSum           float64
	CumulativePrefillCount         int
	CumulativeTokens               int
	CumulativeTokenDurationSeconds float64
}

Statistics captures generation performance metrics.