ortgenai

package module
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 4, 2026 License: MIT Imports: 13 Imported by: 0

README

ortgenai

Go bindings for the ONNX Runtime GenAI C API.

This package provides a thin, idiomatic Go wrapper around the ONNX Runtime GenAI shared library, exposing:

  • Session and model creation from local model folders
  • Tokenization and chat templating
  • Batched, streaming text generation with per-token deltas
  • Runtime statistics (tokens/sec, prefill timings)
  • Provider selection and advanced provider options
  • Multimodal input support (text + images)

Note: The current implementation loads the GenAI shared library via dlopen, and targets Linux (ELF .so).

Note: This implementation is still alpha so the API may change in future releases. You might want to rely on hugot for a higher-level interface to ONNX Runtime GenAI in Go.

Contents

  • Requirements
  • Installation
  • Quick start
  • Advanced usage
  • Running tests
  • Docker and containerized tests
  • Troubleshooting
  • License

Requirements

  • Go 1.19+
  • Linux with glibc (uses dlfcn.h and .so loading)
  • ONNX Runtime GenAI shared library and dependencies available at runtime:
    • libonnxruntime-genai.so
    • libonnxruntime.so (must be available in the same directory as libonnxruntime-genai.so)
  • A local model directory compatible with ONNX Runtime GenAI (e.g., a converted Phi-3.5 model folder)

Installation

go get github.com/knights-analytics/ortgenai

At runtime, the wrapper needs to dlopen the ONNX Runtime GenAI shared library. You can:

  1. Place libonnxruntime-genai.so next to your application binary (with libonnxruntime.so in the same folder), or
  2. Call genai.SetSharedLibraryPath("/path/to/libonnxruntime-genai.so") before initialization.

Quick start

package main

import (
    "context"
    "fmt"
    "time"

    genai "github.com/knights-analytics/ortgenai"
)

func main() {
    // Optional: set explicit path if the .so isn't on the default loader path
    // Note: ensure libonnxruntime.so is colocated with libonnxruntime-genai.so
    genai.SetSharedLibraryPath("/usr/lib/libonnxruntime-genai.so")

    if err := genai.InitializeEnvironment(); err != nil {
        panic(fmt.Errorf("init failed: %w", err))
    }
    defer func() {
        err = genai.DestroyEnvironment()
        if err != nil {
            panic(fmt.Errorf("destroy environment: %w", err))
        }
    }()

    // Point to a local model folder compatible with ONNX Runtime GenAI
    session, err := genai.CreateSession("./models/phi3.5")
    if err != nil {
        panic(fmt.Errorf("create session: %w", err))
    }
    defer session.Destroy()

    // Prepare one or more conversations (batched)
    conv1 := []genai.Message{
        {Role: "system", Content: "You are a helpful assistant."},
        {Role: "user", Content: "What is the capital of France?"},
    }

    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()

    deltas, errs, err := session.Generate(ctx, [][]genai.Message{conv1})
    if err != nil {
        panic(fmt.Errorf("generate: %w", err))
    }

    // Stream tokens as they arrive
    for {
        select {
        case d, ok := <-deltas:
            if !ok { // stream completed
                stats := session.GetStatistics()
                fmt.Printf("\nTokens/sec: %.2f\n", stats.TokensPerSecond)
                return
            }
            fmt.Print(d.Tokens) // append to your buffer
        case e := <-errs:
            if e != nil { panic(e) }
        }
    }
}

Advanced usage

Explicit shared library path
genai.SetSharedLibraryPath("/opt/onnxruntime/lib/libonnxruntime-genai.so")
if err := genai.InitializeEnvironment(); err != nil { /* handle */ }

If not set, the code tries libonnxruntime-genai.so relative to the loader’s search path. Ensure libonnxruntime.so is colocated with the GenAI .so.

Provider selection and options

If you have a JSON config path and custom providers to use (e.g., CUDA, CPU) you can create a session with advanced settings:

providers := []string{"cpu"} // or e.g., []string{"cuda", "cpu"}
providerOptions := map[string]map[string]string{
    "cpu": {"intra_op_num_threads": "4"},
}

session, err := genai.CreateSessionWithOptions(
    "./path/to/config.json", // model/session config from GenAI tooling
    providers,
    providerOptions,
)
Batched generation

Session.Generate accepts multiple conversations in one call: [][]Message. The returned channel carries SequenceDelta items, each labeled with the Sequence index so you can route output per-conversation.

Statistics

After generation, inspect session.GetStatistics() for fields such as TokensPerSecond, cumulative token counts, and prefill timings.

Running tests

Local tests require the GenAI shared libraries and a local model directory. The provided unit test expects:

  • libonnxruntime-genai.so available (by default at /usr/lib/libonnxruntime-genai.so in the test; adjust via SetSharedLibraryPath), and
  • a model directory at ./_models/phi3.5 (update the path as needed).

Run:

go test ./...

Docker and containerized tests

Two Dockerfiles are provided:

  • Dockerfile — base image with dependencies
  • test.Dockerfile — image to run unit tests

Helper scripts are available in scripts/:

  • scripts/run-unit-tests-container.sh — build the test image and run tests in a container
  • scripts/run-unit-test.sh — run tests directly (expects environment to be prepared)

You may also use compose-test.yaml to orchestrate test runs.

Troubleshooting

  • error loading GenAI shared library: Ensure the path to libonnxruntime-genai.so is correct and readable by the process. Set it explicitly with SetSharedLibraryPath.
  • missing Oga... symbols or "missing Oga..." errors: The GenAI .so must export required symbols (e.g., OgaCreateModel). Make sure versions of libonnxruntime-genai.so and libonnxruntime.so are compatible and colocated.
  • segmentation fault on load: Verify that your system’s CUDA/CPU provider dependencies match the .so build (driver/runtime versions).
  • no output / stuck: Ensure your model folder is valid for ONNX Runtime GenAI and accessible; increase timeouts during first-run warmup.

License

This project is licensed under the terms of the MIT License. See LICENSE for details.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrNotInitialized = fmt.Errorf("InitializeEnvironment() has either " +
	"not yet been called, or did not return successfully")

Functions

func DestroyEnvironment 

func DestroyEnvironment() error

DestroyEnvironment Call this function to clean up the internal onnxruntime environment when it is no longer required.

func InitializeEnvironment 

func InitializeEnvironment() error

func InitializeGenAiLibrary 

func InitializeGenAiLibrary() error

InitializeGenAiLibrary loads the ONNX Runtime GenAI shared library specified by onnxGenaiSharedLibraryPath (or a default) so its exported symbols become available. The assumption is that libonnxruntime.so is available in the same folder where the libonnxruntime-genai.so is located.

func IsInitialized 

func IsInitialized() bool

func OgaResultToError 

func OgaResultToError(result *C.OgaResult) error

func SetSharedLibraryPath 

func SetSharedLibraryPath(path string)

func SetTelemetryEnabled added in v0.3.2 

func SetTelemetryEnabled(enabled bool)

SetTelemetryEnabled enables or disables telemetry collection in ORT GenAI.

Types

type Engine added in v0.3.0 

type Engine struct {
	// contains filtered or unexported fields
}

Engine provides continuous batching via the OgaEngine C API. Multiple goroutines may call Submit concurrently; the engine batches their requests for efficient inference.

func CreateEngine added in v0.3.0 

func CreateEngine(modelPath string) (*Engine, error)

CreateEngine creates a new Engine from the given model path. The engine starts a background loop that processes submitted requests.

func CreateEngineWithOptions added in v0.3.0 

func CreateEngineWithOptions(configDirectoryPath string, providers []string, providerOptions map[string]map[string]string) (*Engine, error)

CreateEngineWithOptions creates a new Engine with explicit execution provider configuration.

func (*Engine) Destroy added in v0.3.0 

func (e *Engine) Destroy()

Destroy stops the engine and releases all resources. It must not be called concurrently with Submit or other Engine methods.

func (*Engine) Generate added in v0.3.0 

func (e *Engine) Generate(ctx context.Context, messages [][]Message, tools []string, opts *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

func (*Engine) GetStatistics added in v0.3.0 

func (e *Engine) GetStatistics() *Statistics

GetStatistics returns generation performance metrics for the engine.

func (*Engine) Stop added in v0.3.0 

func (e *Engine) Stop()

func (*Engine) Submit added in v0.3.0 

func (e *Engine) Submit(ctx context.Context, messages []Message, tools []string, opts *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

Submit submits a generation request to the engine and returns channels for streaming output. Multiple goroutines may call Submit concurrently.

Callers should pass a context with a deadline or timeout. If more than 256 goroutines submit concurrently without deadlines, Stop may block until the excess callers' contexts are cancelled.

type GenerationOptions 

type GenerationOptions struct {
	MaxLength   int
	BatchSize   int
	Temperature *float64
	TopP        *float64
	Seed        *int
	Guidance    *Guidance
}

type Guidance added in v0.2.0 

type Guidance struct {
	Type GuidanceType
	Data string
	// EnableFFTokens speeds up generation by force-forwarding tokens that satisfy the grammar
	// without calling the model. Only valid when BatchSize=1 and beam_size=1.
	EnableFFTokens bool
}

Guidance configures constrained (guided) generation. Requires a recent OGA runtime.

type GuidanceType added in v0.2.0 

type GuidanceType string

GuidanceType specifies the constrained-generation strategy passed to OgaGeneratorParamsSetGuidance. 

const (
	GuidanceTypeJSONSchema  GuidanceType = "json_schema"
	GuidanceTypeRegex       GuidanceType = "regex"
	GuidanceTypeLarkGrammar GuidanceType = "lark_grammar"
)

type Images added in v0.0.2 

type Images struct {
	// contains filtered or unexported fields
}

Images represents a collection of loaded images for multimodal processing.

func LoadImage added in v0.0.2 

func LoadImage(imagePath string) (*Images, error)

LoadImage loads a single image from a file path or data URI.

func LoadImageFromBuffer added in v0.0.2 

func LoadImageFromBuffer(imageData []byte) (*Images, error)

LoadImageFromBuffer loads a single image from a byte buffer.

func LoadImages added in v0.0.2 

func LoadImages(imagePaths []string) (*Images, error)

LoadImages loads multiple images from file paths or data URIs.

func LoadImagesFromBuffers added in v0.0.2 

func LoadImagesFromBuffers(imageBuffers [][]byte) (*Images, error)

LoadImagesFromBuffers loads multiple images from byte buffers.

func (*Images) Destroy added in v0.0.2 

func (i *Images) Destroy()

Destroy releases the images resources.

type MaxLengthReachedError added in v0.1.0 

type MaxLengthReachedError struct{}

func (MaxLengthReachedError) Error added in v0.1.0 

func (e MaxLengthReachedError) Error() string

type Message 

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

type NamedTensors added in v0.0.2 

type NamedTensors struct {
	// contains filtered or unexported fields
}

NamedTensors represents a collection of named tensor inputs.

func (*NamedTensors) Destroy added in v0.0.2 

func (nt *NamedTensors) Destroy()

Destroy releases the named tensors resources.

type SequenceDelta 

type SequenceDelta struct {
	Sequence   int
	Token      string
	EOSReached bool
}

type Session 

type Session struct {
	// contains filtered or unexported fields
}

func CreateSession added in v0.1.0 

func CreateSession(modelPath string) (*Session, error)

func CreateSessionWithOptions added in v0.1.0 

func CreateSessionWithOptions(configDirectoryPath string, providers []string, providerOptions map[string]map[string]string) (*Session, error)

CreateSessionWithOptions builds a GenAI config from a config directory, applies execution providers and options, creates the model and tokenizer, and returns a Session. providers: list of EP names in priority order (e.g., ["cuda"], ["NvTensorRtRtx"], ["OpenVINO"]). providerOptions: map of EP name -> map of key/value options.

func (*Session) Destroy 

func (s *Session) Destroy()

func (*Session) Generate 

func (s *Session) Generate(ctx context.Context, messages [][]Message, tools []string, generationOptions *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

func (*Session) GenerateWithImages added in v0.0.2 

func (s *Session) GenerateWithImages(ctx context.Context, messages [][]Message, images *Images, tools []string, generationOptions *GenerationOptions) (<-chan SequenceDelta, <-chan error, error)

GenerateWithImages generates text using pre-processed named tensors (for multimodal inputs). Currently only supports a single prompt.

func (*Session) GetStatistics 

func (s *Session) GetStatistics() *Statistics

GetStatistics returns the last computed statistics for the session.

type Statistics 

type Statistics struct {
	AvgPrefillSeconds float64
	TokensPerSecond   float64
	// cumulative
	CumulativePrefillSum           float64
	CumulativePrefillCount         int
	CumulativeTokens               int
	CumulativeTokenDurationSeconds float64
}

Statistics captures generation performance metrics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.