ortgenai

README

ortgenai

Go bindings for the ONNX Runtime GenAI C API.

This package provides a thin, idiomatic Go wrapper around the ONNX Runtime GenAI shared library, exposing:

• Session and model creation from local model folders
• Tokenization and chat templating
• Batched, streaming text generation with per-token deltas
• Runtime statistics (tokens/sec, prefill timings)
• Provider selection and advanced provider options

Note: The current implementation loads the GenAI shared library via dlopen, and targets Linux (ELF .so).

Note: This implementation is still alpha so the API may change in future releases. You might want to rely on hugot for a higher-level interface to ONNX Runtime GenAI in Go.

Contents

• Requirements
• Installation
• Quick start
• Advanced usage
• Running tests
• Docker and containerized tests
• Troubleshooting
• License

Requirements

• Go 1.19+
• Linux with glibc (uses dlfcn.h and .so loading)
• ONNX Runtime GenAI shared library and dependencies available at runtime:
  • libonnxruntime-genai.so
  • libonnxruntime.so (must be available in the same directory as libonnxruntime-genai.so)
• A local model directory compatible with ONNX Runtime GenAI (e.g., a converted Phi-3.5 model folder)

Installation

go get github.com/knights-analytics/ortgenai

At runtime, the wrapper needs to dlopen the ONNX Runtime GenAI shared library. You can:

1. Place libonnxruntime-genai.so next to your application binary (with libonnxruntime.so in the same folder), or
2. Call genai.SetSharedLibraryPath("/path/to/libonnxruntime-genai.so") before initialization.

Quick start

package main

import (
    "context"
    "fmt"
    "time"

    genai "github.com/knights-analytics/ortgenai"
)

func main() {
    // Optional: set explicit path if the .so isn't on the default loader path
    // Note: ensure libonnxruntime.so is colocated with libonnxruntime-genai.so
    genai.SetSharedLibraryPath("/usr/lib/libonnxruntime-genai.so")

    if err := genai.InitializeEnvironment(); err != nil {
        panic(fmt.Errorf("init failed: %w", err))
    }
    defer func() {
        err = genai.DestroyEnvironment()
        if err != nil {
            panic(fmt.Errorf("destroy environment: %w", err))
        }
    }()

    // Point to a local model folder compatible with ONNX Runtime GenAI
    session, err := genai.CreateGenerativeSession("./_models/phi3.5")
    if err != nil {
        panic(fmt.Errorf("create session: %w", err))
    }
    defer session.Destroy()

    // Prepare one or more conversations (batched)
    conv1 := []genai.Message{
        {Role: "system", Content: "You are a helpful assistant."},
        {Role: "user", Content: "What is the capital of France?"},
    }

    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()

    deltas, errs, err := session.Generate(ctx, [][]genai.Message{conv1})
    if err != nil {
        panic(fmt.Errorf("generate: %w", err))
    }

    // Stream tokens as they arrive
    for {
        select {
        case d, ok := <-deltas:
            if !ok { // stream completed
                stats := session.GetStatistics()
                fmt.Printf("\nTokens/sec: %.2f\n", stats.TokensPerSecond)
                return
            }
            fmt.Print(d.Tokens) // append to your buffer
        case e := <-errs:
            if e != nil { panic(e) }
        }
    }
}

Advanced usage

Explicit shared library path

genai.SetSharedLibraryPath("/opt/onnxruntime/lib/libonnxruntime-genai.so")
if err := genai.InitializeEnvironment(); err != nil { /* handle */ }

If not set, the code tries libonnxruntime-genai.so relative to the loader’s search path. Ensure libonnxruntime.so is colocated with the GenAI .so.

Provider selection and options

If you have a JSON config path and custom providers to use (e.g., CUDA, CPU) you can create a session with advanced settings:

providers := []string{"cpu"} // or e.g., []string{"cuda", "cpu"}
providerOptions := map[string]map[string]string{
    "cpu": {"intra_op_num_threads": "4"},
}

session, err := genai.CreateGenerativeSessionAdvanced(
    "./path/to/config.json", // model/session config from GenAI tooling
    providers,
    providerOptions,
)

Batched generation

Session.Generate accepts multiple conversations in one call: [][]Message. The returned channel carries SequenceDelta items, each labeled with the Sequence index so you can route output per-conversation.

Statistics

After generation, inspect session.GetStatistics() for fields such as TokensPerSecond, cumulative token counts, and prefill timings.

Running tests

Local tests require the GenAI shared libraries and a local model directory. The provided unit test expects:

• libonnxruntime-genai.so available (by default at /usr/lib/libonnxruntime-genai.so in the test; adjust via SetSharedLibraryPath), and
• a model directory at ./_models/phi3.5 (update the path as needed).

Run:

go test ./...

Docker and containerized tests

Two Dockerfiles are provided:

• Dockerfile — base image with dependencies
• test.Dockerfile — image to run unit tests

Helper scripts are available in scripts/:

• scripts/run-unit-tests-container.sh — build the test image and run tests in a container
• scripts/run-unit-test.sh — run tests directly (expects environment to be prepared)

You may also use compose-test.yaml to orchestrate test runs.

Troubleshooting

• error loading GenAI shared library: Ensure the path to libonnxruntime-genai.so is correct and readable by the process. Set it explicitly with SetSharedLibraryPath.
• missing Oga... symbols or "missing Oga..." errors: The GenAI .so must export required symbols (e.g., OgaCreateModel). Make sure versions of libonnxruntime-genai.so and libonnxruntime.so are compatible and colocated.
• segmentation fault on load: Verify that your system’s CUDA/CPU provider dependencies match the .so build (driver/runtime versions).
• no output / stuck: Ensure your model folder is valid for ONNX Runtime GenAI and accessible; increase timeouts during first-run warmup.

License

This project is licensed under the terms of the MIT License. See LICENSE for details.

Documentation

Index

• Variables
• func DestroyEnvironment() error
• func InitializeEnvironment() error
• func InitializeGenAiLibrary() error
• func IsInitialized() bool
• func OgaResultToError(result *C.OgaResult) error
• func SetSharedLibraryPath(path string)
• type GenerationOptions
• type Message
• type SequenceDelta
• type Session
  • func CreateGenerativeSession(modelPath string) (*Session, error)
  • func CreateGenerativeSessionAdvanced(configDirectoryPath string, providers []string, ...) (*Session, error)
  • func (s *Session) Destroy()
  • func (s *Session) Generate(ctx context.Context, messages [][]Message, ...) (<-chan SequenceDelta, <-chan error, error)
  • func (s *Session) GetStatistics() Statistics
• type Statistics

Variables

var ErrNotInitialized = fmt.Errorf("InitializeEnvironment() has either " +
	"not yet been called, or did not return successfully")

Functions

func DestroyEnvironment

func DestroyEnvironment() error

DestroyEnvironment Call this function to clean up the internal onnxruntime environment when it is no longer required.

func InitializeEnvironment

func InitializeEnvironment() error

func InitializeGenAiLibrary

func InitializeGenAiLibrary() error

InitializeGenAiLibrary loads the ONNX Runtime GenAI shared library specified by onnxGenaiSharedLibraryPath (or a default) so its exported symbols become available. The assumption is that libonnxruntime.so is available in the same folder where the libonnxruntime-genai.so is located.

func IsInitialized

func IsInitialized() bool

func OgaResultToError

func OgaResultToError(result *C.OgaResult) error

func SetSharedLibraryPath

func SetSharedLibraryPath(path string)

Types

type GenerationOptions

type GenerationOptions struct {
	MaxLength int
	BatchSize int
}

type Message

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

type SequenceDelta

type SequenceDelta struct {
	Sequence int
	Tokens   string
}

type Session

type Session struct {
	// contains filtered or unexported fields
}

func CreateGenerativeSession

func CreateGenerativeSession(modelPath string) (*Session, error)

func CreateGenerativeSessionAdvanced

func CreateGenerativeSessionAdvanced(configDirectoryPath string, providers []string, providerOptions map[string]map[string]string) (*Session, error)

CreateGenerativeSessionAdvanced builds a GenAI config from a config directory, applies execution providers and options, creates the model and tokenizer, and returns a Session. providers: list of EP names in priority order (e.g., ["cuda"], ["NvTensorRtRtx"], ["OpenVINO"]). providerOptions: map of EP name -> map of key/value options.

func (*Session) Destroy

func (s *Session) Destroy()

func (*Session) Generate

func (s *Session) Generate(ctx context.Context, messages [][]Message, generationOptions *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

func (*Session) GetStatistics

func (s *Session) GetStatistics() Statistics

GetStatistics returns the last computed statistics for the session.

type Statistics

type Statistics struct {
	AvgPrefillSeconds float64
	TokensPerSecond   float64
	// contains filtered or unexported fields
}

Statistics captures generation performance metrics.