onnxcraft

README

ONNXCraft

Production ONNX inference, crafted for Go.

ONNXCraft embeds production-grade machine-learning inference directly in Go applications. It manages and verifies the native ONNX Runtime, distributes reproducible model bundles, and provides one coherent API from typed tensors to ready-to-run text, vision, and multimodal pipelines—without Python, sidecars, or a manual runtime installation.

Why ONNXCraft

• Start high-level, drop down when needed. Use polished task pipelines or load any compatible ONNX tensor model through typed sessions and tensors.
• Reproducible by default. Runtime binaries and catalog artifacts are pinned, size-checked, SHA-256 verified, and installed transactionally.
• Designed for long-lived services. Context cancellation, concurrent use, reusable output buffers, offline operation, and explicit ownership are part of the API rather than afterthoughts.
• One maintained stack. The native binding, runtime lifecycle, artifact cache, tokenizers, preprocessing, and postprocessing ship together and are tested as a system.

Requirements

• Go 1.27.0 or later
• A C toolchain for cgo

ONNXCraft is tested with the race detector on Linux AMD64/ARM64, macOS ARM64, and Windows AMD64. Windows ARM64 is covered without the race detector. The native boundary also runs under Go's strict cgo pointer checks and AddressSanitizer.

Install

go get github.com/joeychilson/onnxcraft

Quick start

This example downloads a pinned BGE model, creates sentence embeddings, and compares them. Models and ONNX Runtime are cached after the first download.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/joeychilson/onnxcraft"
	"github.com/joeychilson/onnxcraft/embedding"
	"github.com/joeychilson/onnxcraft/modelhub"
	"github.com/joeychilson/onnxcraft/models"
	"github.com/joeychilson/onnxcraft/vector"
)

func main() {
	ctx := context.Background()

	hub, err := modelhub.New()
	if err != nil {
		log.Fatal(err)
	}
	runtime, err := onnxcraft.Open(ctx)
	if err != nil {
		log.Fatal(err)
	}
	defer runtime.Close()

	model, err := embedding.Open(ctx, runtime, hub, models.BGESmallENV15FP32)
	if err != nil {
		log.Fatal(err)
	}
	defer model.Close()

	vectors, err := model.EmbedBatch(ctx, []string{
		"A dog is playing in the park.",
		"A puppy plays outside.",
	}, embedding.Options{})
	if err != nil {
		log.Fatal(err)
	}

	similarity, err := vector.CosineSimilarity(vectors[0], vectors[1])
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("similarity: %.4f\n", similarity)
}

More runnable examples are available in examples.

Packages

Package	Purpose
onnxcraft	Runtime management, model inspection, typed tensors, and ONNX sessions
models	Curated, immutable model and pipeline specifications
modelhub	Verified model downloads, caching, bundles, and offline mode
tokenizer	Pure-Go WordPiece, RoBERTa byte-level BPE, and CLIP BPE; sentence pairs, exact byte offsets, overflow windows, and batching
fillmask	Single and batched masked-token prediction
embedding	Single and batched sentence, query, and document embeddings with pooling and normalization
textclassification	Batched single-text and sentence-pair classification
zeroshotclassification	Batched single-label and multi-label classification against arbitrary candidate labels
tokenclassification	Named-entity recognition with exact source-text spans
questionanswering	Windowed extractive question answering over short or long context
reranking	Cross-encoder scoring and document ranking
imageembedding	Batched general-purpose image features for similarity, retrieval, and clustering
visionlanguage	Shared image-text embeddings, cross-modal retrieval, and arbitrary-label image classification
imageclassification	Batched image classification
objectdetection	End-to-end object detection with pixel-coordinate boxes
depthestimation	Monocular relative-depth maps at original image resolution
imagematting	Soft portrait alpha mattes, transparent cutouts, and compositing
vision	Image resizing, normalization, and NCHW conversion
postprocess	Classification and object-detection postprocessing
vector	Validated dot products, norms, normalization, and cosine similarity
labels	ImageNet-1K and COCO labels

The package boundaries are deliberate. The root onnxcraft package owns ONNX Runtime resources and low-level inference. Its native binding is a maintained, private fork under internal/onnxruntime; applications use one supported ONNXCraft API rather than a second low-level runtime API. modelhub owns artifact transport and caching, while models owns complete immutable pipeline specifications. Task-oriented pipelines remain top-level packages so users can import only the capability they need without a catch-all interface hiding task-specific APIs.

Included models

The models catalog pins each model to an immutable revision, expected size, and SHA-256 digest. Custom HTTPS and Hugging Face artifacts use the modelhub API.

Model	Task	API
models.BGESmallENV15FP32	Retrieval and sentence embeddings	embedding.Open
models.BERTBaseUncasedINT8	Fill mask	fillmask.Open
models.ArcticEmbedXSINT8	Fast retrieval and sentence embeddings	embedding.Open
models.JinaEmbeddingsV2SmallENFP32	Long-context English embeddings	embedding.Open
models.DistilBERTSST2INT8	Sentiment classification	textclassification.Open
models.BERTBaseNERINT8	Named-entity recognition	tokenclassification.Open
models.DistilBERTSQuADINT8	Extractive question answering	questionanswering.Open
models.MSMarcoMiniLML6V2	Search reranking	reranking.Open
models.MiniLM2NLIFP16	Zero-shot classification	zeroshotclassification.Open
models.DINOv2SmallFP16	General-purpose image embeddings	imageembedding.Open
models.CLIPViTB32FP16	Image-text embeddings and arbitrary-label image classification	visionlanguage.Open
models.ConvNeXtTinyFP32	Modern image classification	imageclassification.Open
models.MobileNetV4ConvSmallFP32	Fast mobile image classification	imageclassification.Open
models.RTDETRV2R18FP32	Modern real-time object detection	objectdetection.Open
models.DepthAnythingV2SmallINT8	Monocular relative-depth estimation	depthestimation.Open
models.MODNetPortraitFP16	Portrait matting and background removal	imagematting.Open

Use models.Tasks, models.Get, models.All, and models.ForTask to discover supported tasks and complete model specifications. models.FetchID downloads every file in a preset and returns the verified local bundle. Model weights remain subject to their upstream licenses and are not included in this module. Each specification records its task, tensor roles, preprocessing, source revision, license, precision, minimum runtime, byte size, and SHA-256 digests.

modelhub.Client deduplicates concurrent fetches across goroutines and processes, installs private read-only bundles transactionally, cancels sibling work after a batch failure, retries transient failures, and resumes interrupted downloads with verified HTTP ranges. HTTPS is required by default; local test servers can be enabled explicitly with modelhub.WithInsecureHTTP(true). Production clients can configure download concurrency, retry count, maximum size, authentication headers, progress reporting, cache location, and offline operation:

hub, err := modelhub.New(
	modelhub.WithConcurrency(4),
	modelhub.WithRetries(3),
	modelhub.WithOriginHeader("https://huggingface.co", "Authorization", "Bearer "+token),
	modelhub.WithProgress(func(update modelhub.Progress) {
		fmt.Printf("%s: %d/%d\n", update.Artifact.Name, update.Downloaded, update.Total)
	}),
)

ONNX Runtime

onnxcraft.Open automatically downloads and verifies the CPU build of ONNX Runtime 1.29.0 on:

• macOS ARM64
• Linux AMD64 and ARM64
• Windows AMD64 and ARM64

Both the release archive and the extracted native library are checked against compiled-in SHA-256 digests. Runtime and model cache entries are private to the current user by default.

Set ONNXRUNTIME_SHARED_LIBRARY_PATH or use onnxcraft.WithLibraryPath to load a system or custom runtime. Core ML is available on supported Apple builds; CUDA, TensorRT, OpenVINO, DirectML, and other providers require a compatible custom runtime and the corresponding session option.

The native environment is shared safely across runtimes and sessions. Sessions remain usable until closed, even if their parent runtime has already closed. Runtime.Info reports the actual loaded version and library path, while Runtime.ExecutionProviderDevices reports hardware advertised by registered execution-provider plugins. Task pipelines enforce each model's declared minimum runtime before creating a session; low-level callers can use Runtime.RequireVersion for the same check. Use onnxcraft.WithOffline(true) to require a previously verified runtime and onnxcraft.WithDownloadRetries to tune transient download handling.

Low-level API

Use Runtime.Inspect to discover model inputs and outputs, Runtime.Load to create a session from that schema, and Session.Run or Session.RunNamed for inference. For steady-state workloads, Session.RunInto and RunIntoNamed write directly into reusable TensorBuffer storage and avoid runtime-owned output allocation and copying. InspectBytes and LoadBytes provide the same workflow for models already held in memory. Schema-aware sessions validate input type, rank, and fixed dimensions before crossing the native boundary and validate outputs on return.

session, err := runtime.Load("model.onnx")
if err != nil {
	log.Fatal(err)
}
defer session.Close()

input, err := onnxcraft.NewTensor([]int64{1, 4}, []float32{1, 2, 3, 4})
if err != nil {
	log.Fatal(err)
}
outputs, err := session.Run(ctx, input)
if err != nil {
	log.Fatal(err)
}
values, err := outputs[0].Data[float32]()
if err != nil {
	log.Fatal(err)
}

Sessions support concurrent calls, context cancellation, model metadata, sequential or parallel graph execution, thread counts, graph optimization, memory arenas and patterns, profiling, optimized-model output, custom operators, arbitrary session configuration, and these execution providers:

• Core ML
• CUDA
• TensorRT
• OpenVINO
• DirectML
• Generic provider names supported by the supplied ONNX Runtime build

DirectML sessions are automatically configured for its required sequential execution and memory-pattern behavior, and their Run calls are serialized.

Memory ownership

NewTensor copies both shape and data. TakeTensor copies the shape but adopts a caller-owned data slice, avoiding a large allocation when the buffer was created solely for inference. Do not mutate an adopted slice afterward.

Tensor.Data returns an independent copy. Performance-sensitive read-only code can use onnxcraft.BorrowData, which returns a zero-allocation view that must not be modified. Numeric and boolean input buffers are pinned for the entire native call, and returned tensors own Go memory independent of the session.

Types without native Go scalar representations—float16, bfloat16, float8, packed 2/4-bit values, and complex tensors—use NewRawTensor and TakeRawTensor. Their encoded storage is available through RawData and BorrowRawData; exact byte lengths are validated from the shape and ONNX data type.

NewTensorBuffer[T] allocates fixed-shape output storage once. RunInto writes into it directly; BufferData returns a safe copy, BorrowBufferData returns a zero-copy view until the next run, and Tensor creates an immutable snapshot. NewRawTensorBuffer, RawBufferData, and BorrowRawBufferData provide the equivalent direct-write path for encoded types. A buffer cannot be used by concurrent runs.

Cancellation and concurrency

All operations that can block or perform substantial work accept a context.Context. Cancellation propagates through downloads, file-lock waits, tokenization, image preprocessing, pooling, postprocessing, and ONNX Runtime execution. A session can be used by multiple goroutines. Session.Close waits for active calls, is safe to call concurrently, and returns the same teardown result to every caller.

Dense execution supports booleans, strings, standard integer types, float32, and float64 tensors.

Development

golangci-lint run ./...
go test -race -shuffle=on ./...
go build ./...

Run the native ONNX Runtime integration test with:

ONNXCRAFT_INTEGRATION=1 go test ./...

Set ONNXCRAFT_CACHE_DIR to reuse a downloaded runtime across runs.

Run the opt-in end-to-end suite against every pinned catalog model with:

ONNXCRAFT_MODEL_INTEGRATION=1 ONNXCRAFT_CACHE_DIR="$PWD/.cache/onnxcraft" \
  go test -race -count=1 -p=1 -run '^TestCatalogModel$' ./...

The catalog suite also runs weekly and on demand in GitHub Actions. Fuzz targets and allocation-reporting benchmarks are part of the test packages:

go test -fuzz=Fuzz -fuzztime=30s ./...
go test -run '^$' -bench=. -benchmem ./...

See CONTRIBUTING.md for development and commit conventions, CHANGELOG.md for notable changes, RELEASING.md for the maintainer release process, and SECURITY.md for private vulnerability reporting.

License

MIT

Documentation

Overview

Package onnxcraft runs ONNX models with a managed native runtime.

Open initializes ONNX Runtime, verifies and caches the native library when needed, and returns a Runtime that can create context-aware sessions.

Index

• Constants
• Variables
• func BorrowBufferData[T BufferElement](buffer *TensorBuffer) ([]T, error)
• func BorrowData[T TensorData](tensor Tensor) ([]T, error)
• func BorrowRawBufferData(buffer *TensorBuffer) ([]byte, error)
• func BorrowRawData(tensor Tensor) ([]byte, error)
• func BufferData[T BufferElement](buffer *TensorBuffer) ([]T, error)
• func Data[T TensorData](tensor Tensor) ([]T, error)
• func RawBufferData(buffer *TensorBuffer) ([]byte, error)
• func RawData(tensor Tensor) ([]byte, error)
• type BufferElement
• type DataType
• type ExecutionMode
• type ExecutionProviderDevice
• type LoggingLevel
• type ModelInfo
• type ModelMetadata
• type NativeError
• type NativeErrorCode
• type OptimizationLevel
• type Runtime
  • func Open(ctx context.Context, options ...RuntimeOption) (*Runtime, error)
  • func (r *Runtime) Close() error
  • func (r *Runtime) ExecutionProviderDevices() ([]ExecutionProviderDevice, error)
  • func (r *Runtime) Info() (RuntimeInfo, error)
  • func (r *Runtime) Inspect(modelPath string, options ...SessionOption) (result ModelInfo, resultErr error)
  • func (r *Runtime) InspectBytes(model []byte, options ...SessionOption) (result ModelInfo, resultErr error)
  • func (r *Runtime) Load(modelPath string, options ...SessionOption) (*Session, error)
  • func (r *Runtime) LoadBytes(model []byte, options ...SessionOption) (*Session, error)
  • func (r *Runtime) LoadedVersion() (string, error)
  • func (r *Runtime) NewSession(modelPath string, inputNames []string, outputNames []string, ...) (*Session, error)
  • func (r *Runtime) NewSessionFromBytes(model []byte, inputNames []string, outputNames []string, ...) (*Session, error)
  • func (r *Runtime) NewSessionFromInfo(modelPath string, info ModelInfo, options ...SessionOption) (*Session, error)
  • func (r *Runtime) RequireVersion(minimum string) error
• type RuntimeInfo
• type RuntimeOption
  • func WithCacheDir(path string) RuntimeOption
  • func WithDownloadRetries(count int) RuntimeOption
  • func WithHTTPClient(client *http.Client) RuntimeOption
  • func WithLibraryPath(path string) RuntimeOption
  • func WithOffline(enabled bool) RuntimeOption
• type Session
  • func (s *Session) Close() error
  • func (s *Session) InputNames() []string
  • func (s *Session) Inputs() []ValueInfo
  • func (s *Session) Metadata() (result ModelMetadata, resultErr error)
  • func (s *Session) OutputNames() []string
  • func (s *Session) Outputs() []ValueInfo
  • func (s *Session) Run(ctx context.Context, inputs ...Tensor) (result []Tensor, resultErr error)
  • func (s *Session) RunInto(ctx context.Context, outputs []*TensorBuffer, inputs ...Tensor) (resultErr error)
  • func (s *Session) RunIntoNamed(ctx context.Context, inputs map[string]Tensor, ...) error
  • func (s *Session) RunNamed(ctx context.Context, inputs map[string]Tensor) (map[string]Tensor, error)
• type SessionOption
  • func WithCPUMemoryArena(enabled bool) SessionOption
  • func WithCUDA(settings map[string]string) SessionOption
  • func WithCoreML(settings map[string]string) SessionOption
  • func WithCustomOperators(path string) SessionOption
  • func WithDirectML(deviceID int) SessionOption
  • func WithExecutionMode(mode ExecutionMode) SessionOption
  • func WithExecutionProvider(name string, settings map[string]string) SessionOption
  • func WithInterOpThreads(count int) SessionOption
  • func WithIntraOpThreads(count int) SessionOption
  • func WithLogging(level LoggingLevel) SessionOption
  • func WithMemoryPattern(enabled bool) SessionOption
  • func WithOpenVINO(settings map[string]string) SessionOption
  • func WithOptimization(level OptimizationLevel) SessionOption
  • func WithOptimizedModel(path string) SessionOption
  • func WithProfiling(prefix string) SessionOption
  • func WithSessionConfig(key, value string) SessionOption
  • func WithTensorRT(settings map[string]string) SessionOption
• type Tensor
  • func MustTensor[T TensorData](shape []int64, data []T) Tensor
  • func NewRawTensor(shape []int64, dataType DataType, data []byte) (Tensor, error)
  • func NewTensor[T TensorData](shape []int64, data []T) (Tensor, error)
  • func TakeRawTensor(shape []int64, dataType DataType, data []byte) (Tensor, error)
  • func TakeTensor[T TensorData](shape []int64, data []T) (Tensor, error)
  • func (t Tensor) Data[T TensorData]() ([]T, error)
  • func (t Tensor) Len() int
  • func (t Tensor) Shape() []int64
  • func (t Tensor) Type() DataType
• type TensorBuffer
  • func NewRawTensorBuffer(shape []int64, dataType DataType) (*TensorBuffer, error)
  • func NewTensorBuffer[T BufferElement](shape []int64) (*TensorBuffer, error)
  • func (b *TensorBuffer) Len() int
  • func (b *TensorBuffer) Shape() []int64
  • func (b *TensorBuffer) Tensor() Tensor
  • func (b *TensorBuffer) Type() DataType
• type TensorData
• type ValueInfo
• type ValueKind

Constants

const (
	NativeErrorFail                     = ort.ErrorCodeFail
	NativeErrorInvalidArgument          = ort.ErrorCodeInvalidArgument
	NativeErrorNoSuchFile               = ort.ErrorCodeNoSuchFile
	NativeErrorNoModel                  = ort.ErrorCodeNoModel
	NativeErrorEngineError              = ort.ErrorCodeEngineError
	NativeErrorRuntimeException         = ort.ErrorCodeRuntimeException
	NativeErrorInvalidProtobuf          = ort.ErrorCodeInvalidProtobuf
	NativeErrorModelLoaded              = ort.ErrorCodeModelLoaded
	NativeErrorNotImplemented           = ort.ErrorCodeNotImplemented
	NativeErrorInvalidGraph             = ort.ErrorCodeInvalidGraph
	NativeErrorEPFail                   = ort.ErrorCodeEPFail
	NativeErrorModelLoadCanceled        = ort.ErrorCodeModelLoadCanceled
	NativeErrorModelRequiresCompilation = ort.ErrorCodeModelRequiresCompilation
	NativeErrorNotFound                 = ort.ErrorCodeNotFound
	NativeErrorDeviceReset              = ort.ErrorCodeDeviceReset
)

Native ONNX Runtime failure categories.

const RuntimeVersion = "1.29.0"

RuntimeVersion is the native ONNX Runtime version used by onnxcraft.

Variables

var ErrRuntimeCorrupt = errors.New("onnxcraft: cached native runtime is corrupt")

ErrRuntimeCorrupt is returned when a cached native runtime fails integrity or file-type validation.

var ErrRuntimeNotCached = errors.New("onnxcraft: native runtime is not cached")

ErrRuntimeNotCached is returned when offline mode cannot find a verified bundled native runtime.

var ErrRuntimeTooOld = errors.New("onnxcraft: ONNX Runtime version is too old")

ErrRuntimeTooOld is returned when a model requires a newer ONNX Runtime than the loaded native library.

Functions

func BorrowBufferData

func BorrowBufferData[T BufferElement](buffer *TensorBuffer) ([]T, error)

BorrowBufferData returns a read-only view of the buffer elements without copying. The view remains valid until the buffer is passed to RunInto again. Callers must not access it concurrently with RunInto.

func BorrowData

func BorrowData[T TensorData](tensor Tensor) ([]T, error)

BorrowData returns a read-only view of tensor's elements as T without copying them. The caller must not modify the returned slice. Use Data when ownership or mutation is required.

func BorrowRawBufferData

func BorrowRawBufferData(buffer *TensorBuffer) ([]byte, error)

BorrowRawBufferData returns a read-only view of a raw output buffer's bytes. The view remains valid until the buffer is passed to RunInto again.

func BorrowRawData

func BorrowRawData(tensor Tensor) ([]byte, error)

BorrowRawData returns a read-only view of a raw tensor's encoded bytes. The caller must not modify the returned slice.

func BufferData

func BufferData[T BufferElement](buffer *TensorBuffer) ([]T, error)

BufferData returns an independent copy of the buffer elements.

func Data

func Data[T TensorData](tensor Tensor) ([]T, error)

Data returns a copy of tensor's data as T. It is equivalent to Tensor.Data.

func RawBufferData

func RawBufferData(buffer *TensorBuffer) ([]byte, error)

RawBufferData returns an independent copy of a raw output buffer's encoded bytes.

func RawData

func RawData(tensor Tensor) ([]byte, error)

RawData returns an independent copy of a low-precision, packed, or complex tensor's encoded bytes.

Types

type BufferElement

type BufferElement interface {
	bool |
		float32 | float64 |
		int8 | int16 | int32 | int64 |
		uint8 | uint16 | uint32 | uint64
}

BufferElement is an element type supported by reusable output buffers. Strings are excluded because ONNX Runtime cannot write string outputs into Go-backed storage.

type DataType

type DataType string

DataType identifies the element type stored in a Tensor.

const (
	DataTypeUndefined      DataType = "undefined"
	DataTypeBool           DataType = "bool"
	DataTypeString         DataType = "string"
	DataTypeFloat32        DataType = "float32"
	DataTypeFloat64        DataType = "float64"
	DataTypeFloat16        DataType = "float16"
	DataTypeBFloat16       DataType = "bfloat16"
	DataTypeFloat8E4M3FN   DataType = "float8e4m3fn"
	DataTypeFloat8E4M3FNUZ DataType = "float8e4m3fnuz"
	DataTypeFloat8E5M2     DataType = "float8e5m2"
	DataTypeFloat8E5M2FNUZ DataType = "float8e5m2fnuz"
	DataTypeFloat8E8M0     DataType = "float8e8m0"
	DataTypeFloat4E2M1     DataType = "float4e2m1"
	DataTypeComplex64      DataType = "complex64"
	DataTypeComplex128     DataType = "complex128"
	DataTypeInt8           DataType = "int8"
	DataTypeInt16          DataType = "int16"
	DataTypeInt32          DataType = "int32"
	DataTypeInt64          DataType = "int64"
	DataTypeInt4           DataType = "int4"
	DataTypeInt2           DataType = "int2"
	DataTypeUint8          DataType = "uint8"
	DataTypeUint16         DataType = "uint16"
	DataTypeUint32         DataType = "uint32"
	DataTypeUint64         DataType = "uint64"
	DataTypeUint4          DataType = "uint4"
	DataTypeUint2          DataType = "uint2"
)

Supported tensor element types.

type ExecutionMode

type ExecutionMode int

ExecutionMode controls whether independent graph nodes may execute in parallel. Sequential execution is the default.

const (
	ExecutionSequential ExecutionMode = iota
	ExecutionParallel
)

Supported graph execution modes.

type ExecutionProviderDevice

type ExecutionProviderDevice struct {
	Provider string
	Vendor   string
}

ExecutionProviderDevice describes one hardware target advertised by an execution-provider plugin registered with ONNX Runtime.

type LoggingLevel

type LoggingLevel int

LoggingLevel controls ONNX Runtime session log verbosity.

const (
	LoggingVerbose LoggingLevel = iota
	LoggingInfo
	LoggingWarning
	LoggingError
	LoggingFatal
)

Supported session logging levels.

type ModelInfo

type ModelInfo struct {
	Inputs  []ValueInfo
	Outputs []ValueInfo
}

ModelInfo describes a model's ordered inputs and outputs.

type ModelMetadata

type ModelMetadata struct {
	Producer    string
	Graph       string
	Domain      string
	Description string
	Version     int64
	Custom      map[string]string
}

ModelMetadata contains descriptive fields embedded in an ONNX model.

type NativeError

type NativeError = ort.Error

NativeError is an ONNX Runtime failure. Use errors.As to obtain its Code and Message through the context added by onnxcraft. Validation and context cancellation errors are not necessarily native errors.

type NativeErrorCode

type NativeErrorCode = ort.ErrorCode

NativeErrorCode identifies the category reported by ONNX Runtime.

type OptimizationLevel

type OptimizationLevel int

OptimizationLevel controls ONNX graph optimization for a Session.

const (
	OptimizationDisabled OptimizationLevel = iota
	OptimizationBasic
	OptimizationExtended
	OptimizationAll
)

Supported graph optimization levels.

type Runtime

type Runtime struct {
	// contains filtered or unexported fields
}

Runtime owns a reference to the process-wide ONNX Runtime environment. Close is safe to call more than once. Sessions retain their own reference, so closing Runtime does not invalidate sessions that are still open.

func Open

func Open(ctx context.Context, options ...RuntimeOption) (*Runtime, error)

Open initializes ONNX Runtime. When no library path is provided, Open uses ONNXRUNTIME_SHARED_LIBRARY_PATH or downloads a verified official artifact.

func (*Runtime) Close

func (r *Runtime) Close() error

Close releases this Runtime's reference to the native environment.

func (*Runtime) ExecutionProviderDevices

func (r *Runtime) ExecutionProviderDevices() ([]ExecutionProviderDevice, error)

ExecutionProviderDevices returns the hardware targets currently advertised by registered execution-provider plugins. The returned values are detached from ONNX Runtime and remain safe to use after Runtime is closed.

func (*Runtime) Info

func (r *Runtime) Info() (RuntimeInfo, error)

Info returns details about the selected native runtime.

func (*Runtime) Inspect

func (r *Runtime) Inspect(modelPath string, options ...SessionOption) (result ModelInfo, resultErr error)

Inspect reads the ordered inputs and outputs from modelPath. It creates a temporary ONNX session, so loading a large model may be expensive.

func (*Runtime) InspectBytes

func (r *Runtime) InspectBytes(model []byte, options ...SessionOption) (result ModelInfo, resultErr error)

InspectBytes reads ordered inputs and outputs from an in-memory ONNX model. It creates a temporary ONNX session, so loading a large model may be expensive.

func (*Runtime) Load

func (r *Runtime) Load(modelPath string, options ...SessionOption) (*Session, error)

Load inspects modelPath and creates a session using every graph input and output in model order. Use NewSession when only selected outputs are needed.

func (*Runtime) LoadBytes

func (r *Runtime) LoadBytes(model []byte, options ...SessionOption) (*Session, error)

LoadBytes inspects an in-memory ONNX model and creates a schema-aware session using every graph input and output.

func (*Runtime) LoadedVersion

func (r *Runtime) LoadedVersion() (string, error)

LoadedVersion returns the version reported by the loaded native library.

func (*Runtime) NewSession

func (r *Runtime) NewSession(
	modelPath string,
	inputNames []string,
	outputNames []string,
	options ...SessionOption,
) (*Session, error)

NewSession loads modelPath with positional input and output names.

func (*Runtime) NewSessionFromBytes

func (r *Runtime) NewSessionFromBytes(
	model []byte,
	inputNames []string,
	outputNames []string,
	options ...SessionOption,
) (*Session, error)

NewSessionFromBytes loads an in-memory ONNX model with positional input and output names. ONNX Runtime consumes model during construction; callers may reuse or release the byte slice after this function returns.

func (*Runtime) NewSessionFromInfo

func (r *Runtime) NewSessionFromInfo(
	modelPath string,
	info ModelInfo,
	options ...SessionOption,
) (*Session, error)

NewSessionFromInfo loads modelPath using a previously inspected graph schema. Inputs and outputs are validated on each run.

func (*Runtime) RequireVersion

func (r *Runtime) RequireVersion(minimum string) error

RequireVersion verifies that the loaded native library is at least minimum. Both versions must use Semantic Versioning 2.0 syntax.

type RuntimeInfo

type RuntimeInfo struct {
	Version     string
	LibraryPath string
	OS          string
	Arch        string
}

RuntimeInfo describes the native runtime selected for this process.

type RuntimeOption

type RuntimeOption func(*runtimeConfig) error

RuntimeOption configures Open.

func WithCacheDir

func WithCacheDir(path string) RuntimeOption

WithCacheDir stores downloaded native runtime files beneath path.

func WithDownloadRetries

func WithDownloadRetries(count int) RuntimeOption

WithDownloadRetries sets the number of retries after a transient native runtime download failure. The default is two.

func WithHTTPClient

func WithHTTPClient(client *http.Client) RuntimeOption

WithHTTPClient sets the client used to download ONNX Runtime.

func WithLibraryPath

func WithLibraryPath(path string) RuntimeOption

WithLibraryPath uses an existing ONNX Runtime shared library and disables automatic downloading.

func WithOffline

func WithOffline(enabled bool) RuntimeOption

WithOffline disables native runtime downloads. A custom library or a previously verified bundled runtime must be available.

type Session

type Session struct {
	// contains filtered or unexported fields
}

Session runs one ONNX model. Run may be called concurrently. Close waits for active runs and is safe to call more than once.

func (*Session) Close

func (s *Session) Close() error

Close releases the model session.

func (*Session) InputNames

func (s *Session) InputNames() []string

InputNames returns the model inputs in the positional order expected by Run.

func (*Session) Inputs

func (s *Session) Inputs() []ValueInfo

Inputs returns the schema used to validate inputs. Sessions constructed with NewSession return nil because only names were supplied.

func (*Session) Metadata

func (s *Session) Metadata() (result ModelMetadata, resultErr error)

Metadata returns descriptive fields embedded in the loaded ONNX model.

func (*Session) OutputNames

func (s *Session) OutputNames() []string

OutputNames returns the model outputs in the positional order returned by Run.

func (*Session) Outputs

func (s *Session) Outputs() []ValueInfo

Outputs returns the schema used to validate outputs. Sessions constructed with NewSession return nil because only names were supplied.

func (*Session) Run

func (s *Session) Run(ctx context.Context, inputs ...Tensor) (result []Tensor, resultErr error)

Run executes the model with positional input tensors. Returned tensors own independent Go memory and remain valid after the next run or Close.

func (*Session) RunInto

func (s *Session) RunInto(ctx context.Context, outputs []*TensorBuffer, inputs ...Tensor) (resultErr error)

RunInto executes the model with positional inputs and writes positional outputs directly into reusable caller-owned buffers. Every output must have the exact type and concrete shape produced by the model. Buffer contents are unspecified when the run returns an error.

func (*Session) RunIntoNamed

func (s *Session) RunIntoNamed(
	ctx context.Context,
	inputs map[string]Tensor,
	outputs map[string]*TensorBuffer,
) error

RunIntoNamed executes the model using named inputs and writes every declared output into its named reusable buffer. Missing and unknown names are rejected before inference.

func (*Session) RunNamed

func (s *Session) RunNamed(ctx context.Context, inputs map[string]Tensor) (map[string]Tensor, error)

RunNamed executes the model using input names and returns outputs keyed by name. Every declared input must be present and unknown inputs are rejected.

type SessionOption

type SessionOption func(*sessionConfig) error

SessionOption configures a Session.

func WithCPUMemoryArena

func WithCPUMemoryArena(enabled bool) SessionOption

WithCPUMemoryArena controls ONNX Runtime's CPU memory arena.

func WithCUDA

func WithCUDA(settings map[string]string) SessionOption

WithCUDA enables NVIDIA's CUDA execution provider. Provider settings use the keys documented by ONNX Runtime. A CUDA-enabled native runtime and its dependencies must be supplied with WithLibraryPath.

func WithCoreML

func WithCoreML(settings map[string]string) SessionOption

WithCoreML enables Apple's Core ML execution provider. Provider settings use the keys documented by ONNX Runtime.

func WithCustomOperators

func WithCustomOperators(path string) SessionOption

WithCustomOperators registers a custom-operator shared library.

func WithDirectML

func WithDirectML(deviceID int) SessionOption

WithDirectML enables Microsoft's DirectML execution provider on deviceID. A DirectML-enabled native runtime must be supplied with WithLibraryPath.

func WithExecutionMode

func WithExecutionMode(mode ExecutionMode) SessionOption

WithExecutionMode sets graph execution to sequential or parallel. Setting inter-op threads implicitly selects parallel execution unless this option is used explicitly.

func WithExecutionProvider

func WithExecutionProvider(name string, settings map[string]string) SessionOption

WithExecutionProvider enables a provider through ONNX Runtime's generic provider API. This supports providers such as QNN or XNNPACK when they are included in the supplied native runtime.

func WithInterOpThreads

func WithInterOpThreads(count int) SessionOption

WithInterOpThreads sets the number of threads used across operators.

func WithIntraOpThreads

func WithIntraOpThreads(count int) SessionOption

WithIntraOpThreads sets the number of threads used within an operator.

func WithLogging

func WithLogging(level LoggingLevel) SessionOption

WithLogging sets the ONNX Runtime session log severity threshold.

func WithMemoryPattern

func WithMemoryPattern(enabled bool) SessionOption

WithMemoryPattern controls ONNX Runtime memory-pattern optimization.

func WithOpenVINO

func WithOpenVINO(settings map[string]string) SessionOption

WithOpenVINO enables Intel's OpenVINO execution provider. An OpenVINO-enabled native runtime and its dependencies must be supplied with WithLibraryPath.

func WithOptimization

func WithOptimization(level OptimizationLevel) SessionOption

WithOptimization sets the graph optimization level. The default is OptimizationAll.

func WithOptimizedModel

func WithOptimizedModel(path string) SessionOption

WithOptimizedModel writes the optimized graph to path while loading.

func WithProfiling

func WithProfiling(prefix string) SessionOption

WithProfiling writes ONNX Runtime profiling output using prefix.

func WithSessionConfig

func WithSessionConfig(key, value string) SessionOption

WithSessionConfig sets an ONNX Runtime session configuration entry.

func WithTensorRT

func WithTensorRT(settings map[string]string) SessionOption

WithTensorRT enables NVIDIA's TensorRT execution provider. A TensorRT-enabled native runtime and its CUDA/TensorRT dependencies must be supplied with WithLibraryPath.

type Tensor

type Tensor struct {
	// contains filtered or unexported fields
}

Tensor is an immutable, row-major ONNX tensor.

func MustTensor

func MustTensor[T TensorData](shape []int64, data []T) Tensor

MustTensor is like NewTensor but panics if shape and data are incompatible. It is intended for package-level constants and tests.

func NewRawTensor

func NewRawTensor(shape []int64, dataType DataType, data []byte) (Tensor, error)

NewRawTensor constructs a low-precision, packed, or complex tensor and copies its encoded bytes. Use RawData to retrieve the encoded representation.

func NewTensor

func NewTensor[T TensorData](shape []int64, data []T) (Tensor, error)

NewTensor constructs a tensor and copies shape and data so callers can safely reuse their input slices.

func TakeRawTensor

func TakeRawTensor(shape []int64, dataType DataType, data []byte) (Tensor, error)

TakeRawTensor constructs a low-precision, packed, or complex tensor that adopts data without copying it. The caller must not modify data afterward.

func TakeTensor

func TakeTensor[T TensorData](shape []int64, data []T) (Tensor, error)

TakeTensor constructs a tensor that adopts data without copying it. The caller must not modify data after this function returns. Shape is copied. This is useful when the caller has created a buffer solely for the tensor.

func (Tensor) Data

func (t Tensor) Data[T TensorData]() ([]T, error)

Data returns a copy of the tensor elements as T.

func (Tensor) Len

func (t Tensor) Len() int

Len returns the flattened number of tensor elements.

func (Tensor) Shape

func (t Tensor) Shape() []int64

Shape returns a copy of the tensor dimensions.

func (Tensor) Type

func (t Tensor) Type() DataType

Type returns the tensor element type.

type TensorBuffer

type TensorBuffer struct {
	// contains filtered or unexported fields
}

TensorBuffer is caller-owned, reusable tensor storage for Session.RunInto. A buffer cannot participate in more than one run at a time. Shape and type are fixed at construction.

func NewRawTensorBuffer

func NewRawTensorBuffer(shape []int64, dataType DataType) (*TensorBuffer, error)

NewRawTensorBuffer allocates reusable output storage for a low-precision, packed, or complex tensor.

func NewTensorBuffer

func NewTensorBuffer[T BufferElement](shape []int64) (*TensorBuffer, error)

NewTensorBuffer allocates a zero-filled reusable output buffer.

func (*TensorBuffer) Len

func (b *TensorBuffer) Len() int

Len returns the flattened number of elements in the buffer.

func (*TensorBuffer) Shape

func (b *TensorBuffer) Shape() []int64

Shape returns a copy of the buffer dimensions.

func (*TensorBuffer) Tensor

func (b *TensorBuffer) Tensor() Tensor

Tensor returns an immutable snapshot of the buffer.

func (*TensorBuffer) Type

func (b *TensorBuffer) Type() DataType

Type returns the buffer element type.

type TensorData

type TensorData interface {
	bool | string |
		float32 | float64 |
		int8 | int16 | int32 | int64 |
		uint8 | uint16 | uint32 | uint64
}

TensorData is a Go type supported by an ONNX tensor.

type ValueInfo

type ValueInfo struct {
	Name  string
	Kind  ValueKind
	Shape []int64
	Type  DataType
}

ValueInfo describes one model input or output. Dynamic dimensions are represented by negative values, as reported by ONNX Runtime.

type ValueKind

type ValueKind string

ValueKind identifies the top-level ONNX type of a model input or output.

const (
	ValueKindUnknown      ValueKind = "unknown"
	ValueKindTensor       ValueKind = "tensor"
	ValueKindSequence     ValueKind = "sequence"
	ValueKindMap          ValueKind = "map"
	ValueKindOpaque       ValueKind = "opaque"
	ValueKindSparseTensor ValueKind = "sparse_tensor"
	ValueKindOptional     ValueKind = "optional"
)

ONNX value kinds reported by Inspect.