Redirected from 
README
¶
onnxcraft
An ONNX inference library for Go, powered by ONNX Runtime.
Requires Go 1.27 or later, cgo, a C compiler, and ONNX Runtime 1.29 or later. No Go module dependencies.
Installation
go get github.com/joeychilson/onnxcraft
Install an ONNX Runtime shared library
for your platform and execution provider. Pass its path to Open, keeping any
provider libraries alongside it. The C headers are included in this module.
Package
onnxcraft— model loading, typed tensors, inference, reusable outputs, and execution provider configuration.
Supports dense numeric and boolean tensors, including float16, bfloat16, scalars, empty tensors, and dynamic dimensions.
Example
Add two tensors using the included testdata/add.onnx model. Run from the
repository root with ONNXRUNTIME_SHARED_LIBRARY_PATH set to your native library.
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/joeychilson/onnxcraft"
)
func main() {
rt, err := onnxcraft.Open(os.Getenv("ONNXRUNTIME_SHARED_LIBRARY_PATH"))
if err != nil {
log.Fatal(err)
}
defer rt.Close()
session, err := rt.Load("testdata/add.onnx", nil)
if err != nil {
log.Fatal(err)
}
defer session.Close()
a, err := onnxcraft.NewTensor([]int64{1, 3}, []float32{1, 2, 3})
if err != nil {
log.Fatal(err)
}
b, err := onnxcraft.NewTensor([]int64{1, 3}, []float32{10, 20, 30})
if err != nil {
log.Fatal(err)
}
outputs, err := session.Run(context.Background(), a, b)
if err != nil {
log.Fatal(err)
}
values, err := outputs[0].Data[float32]()
if err != nil {
log.Fatal(err)
}
fmt.Println(values) // [11 22 33]
}
See examples/basic for the runnable example. Use
session.Inputs() and session.Outputs() to inspect your model's types,
dimensions, and argument order. LoadBytes accepts self-contained model data;
Load also supports models with external weights.
Usage
Load a session once and reuse it across requests. SessionOptions configures
thread counts, graph optimizations, and execution providers such as CUDA,
TensorRT, CoreML, and OpenVINO. Providers require a compatible native library.
| Operation | Behavior |
|---|---|
NewTensor(shape, data) |
Copies the shape and shares the Go data slice. |
tensor.Data[T]() |
Returns a mutable view without copying. |
session.Run(ctx, inputs...) |
Returns independent, Go-owned output tensors. |
session.RunInto(ctx, outputs, inputs...) |
Writes into reusable output tensors with exact result shapes and types. |
Tensors need no cleanup. Close runtimes and sessions when finished; sessions keep their runtime alive. CPU sessions allow concurrent runs. Sessions with explicit execution providers serialize runs. Use independent output buffers for concurrent callers, and do not access tensor data while inference writes it or mutate data while inference reads it.
Cancellation affects only the current run and waits for native execution to
finish. Its latency depends on the operator and provider. After a failed or
canceled RunInto, output contents are unspecified.
Benchmarks
Median of five runs on an Apple M2, macOS ARM64, Go 1.27.0, and ONNX Runtime
1.29.0 CPU. Uses testdata/sum.onnx, a float32 input of shape [1, 10], one
intra-op thread, and context.Background().
| Operation | Time/op | Go bytes/op | Go allocs/op |
|---|---|---|---|
| LoadBytes | 74.48 µs | 256 | 9 |
| Run | 1.454 µs | 64 | 3 |
| RunInto | 1.303 µs | 0 | 0 |
These measure binding overhead on a small model. Go allocation counts exclude
native allocations; the warmed RunInto path reuses internal storage.
Cancelable contexts require additional bookkeeping.
Development
Set ONNXRUNTIME_SHARED_LIBRARY_PATH to enable native integration tests.
go vet ./...
GOEXPERIMENT=cgocheck2 go test -race -shuffle=on ./...
go build ./...
go test -run '^$' -bench BenchmarkNativeSession -benchmem -count=5 .
CI runs native tests on Linux AMD64/ARM64, macOS ARM64, and Windows AMD64.
Generate test models with python3 testdata/generate.py; no Python packages
are needed.
Release
Push a semantic version tag in the form vMAJOR.MINOR.PATCH, optionally with a
prerelease suffix, to run CI and create a GitHub Release with generated notes.
The release publishes the Go module source and does not include binary artifacts.
License
MIT. The included ONNX Runtime headers are distributed under Microsoft's MIT license.
Documentation
¶
Overview ¶
Package onnxcraft runs ONNX models using the ONNX Runtime C API.
Open a Runtime, load a Session, and pass tensors to Session.Run. Reuse the session across requests. Use Session.RunInto to reuse output storage. Tensor data belongs to Go and needs no Close; runtimes and sessions must be closed explicitly. A session supports concurrent runs with independent output buffers. Applications must synchronize access to tensor data they mutate.
Building requires Go 1.27 and cgo. Running requires an ONNX Runtime 1.29 or newer shared library. Loading the library never downloads or installs code.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrClosed = errors.New("onnxcraft: closed")
ErrClosed is returned when using a closed runtime or session.
Functions ¶
This section is empty.
Types ¶
type BFloat16 ¶ added in v0.2.0
type BFloat16 uint16
BFloat16 holds the bfloat16 bits of a tensor element.
type Element ¶ added in v0.2.0
type Element interface {
~float32 | ~float64 | ~int8 | ~int16 | ~int32 | ~int64 |
~uint8 | ~uint16 | ~uint32 | ~uint64 | ~bool
}
Element is a fixed-size ONNX tensor element. Named Go types are supported. int and uint are excluded because their width depends on the architecture.
type Float16 ¶ added in v0.2.0
type Float16 uint16
Float16 holds the IEEE 754 binary16 bits of a tensor element.
type NativeError ¶ added in v0.1.1
NativeError preserves an ONNX Runtime error code and message. Use errors.As to inspect it. Codes are the OrtErrorCode values in the C API.
func (*NativeError) Error ¶ added in v0.2.0
func (e *NativeError) Error() string
type Optimization ¶ added in v0.2.0
type Optimization uint8
Optimization selects graph transformations during model loading.
const ( OptimizeAll Optimization = iota // Default: all ONNX Runtime optimizations. OptimizeNone OptimizeBasic OptimizeExtended )
type Provider ¶ added in v0.2.0
Provider configures an execution provider by its ONNX Runtime name, such as "CUDA", "TensorRT", "CoreML", "OpenVINO", or "XNNPACK". The library must include that provider; unavailable providers return an error during Load.
type Runtime ¶
type Runtime struct {
// contains filtered or unexported fields
}
Runtime owns a loaded ONNX Runtime library. It must not be copied. Close prevents new sessions; existing sessions keep the library alive.
func Open ¶
Open loads an explicit shared library path. Relative paths are resolved against the working directory. The library must support C API version 29.
func (*Runtime) Load ¶
func (r *Runtime) Load(path string, options *SessionOptions) (*Session, error)
Load opens an ONNX or ORT model file, including models with external weights. A nil options pointer selects the defaults. Options are consumed during Load.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session is a loaded model. Reuse it across inference calls; it must not be copied. CPU runs may execute concurrently. Sessions with explicit execution providers serialize runs because some providers require exclusive access.
func (*Session) Close ¶
Close waits for active inference calls, then releases the model. It is idempotent. Later calls to Run and RunInto return ErrClosed.
func (*Session) Inputs ¶
func (s *Session) Inputs() []TensorInfo
Inputs returns independent copies of the input metadata in Run argument order.
func (*Session) Outputs ¶
func (s *Session) Outputs() []TensorInfo
Outputs returns independent copies of the output metadata in result order.
func (*Session) Run ¶
Run executes all model outputs. Inputs follow Inputs order. Returned tensors own independent Go storage; they need no cleanup and can be reused as inputs. Cancellation asks ONNX Runtime to terminate and waits until native work stops. Cancellation latency depends on the executing operator and provider.
func (*Session) RunInto ¶
RunInto writes all outputs into caller-provided tensors, without tensor data copies. Outputs must have the exact resulting types and shapes, in Outputs order. They must not overlap inputs or each other. Do not read or mutate their data during the call or use them in concurrent runs. On error or cancellation their contents are unspecified; the buffers remain reusable.
type SessionOptions ¶ added in v0.2.0
type SessionOptions struct {
IntraOpThreads int // Threads within an operator; zero lets ORT choose.
InterOpThreads int // Threads between operators when Parallel is true.
Parallel bool
Optimization Optimization
Providers []Provider // In priority order; ORT's CPU fallback remains enabled.
Config map[string]string // ONNX Runtime session configuration entries.
}
SessionOptions configures model loading. Its zero value uses CPU execution, sequential graph scheduling, all graph optimizations, and ORT thread defaults.
type Tensor ¶
type Tensor struct {
// contains filtered or unexported fields
}
Tensor is a dense, contiguous, row-major tensor backed by Go memory. Copies share data; Shape returns a copy. Its zero value is invalid. A tensor and slices obtained from it remain valid after a session is closed.
func NewTensor ¶
NewTensor wraps data without copying it and copies shape. A nil or empty shape represents a scalar and requires one element. Zero dimensions are allowed; negative dimensions and overflowing sizes are rejected.
Do not mutate data while an inference call reads or writes it. To give a tensor independent storage, pass slices.Clone(data).
func (Tensor) Data ¶
Data returns a view of the tensor's data, with no copy. T must match its ONNX element type. Float16 and BFloat16 are distinct from uint16.
type TensorInfo ¶ added in v0.2.0
TensorInfo describes a model input or output. Negative dimensions are dynamic.
Source Files
Directories