coreml

README

coreml

Run a trained model on a Mac's Neural Engine, from pure Go, with no cgo and no Xcode.

The Neural Engine is the part of an Apple chip that costs almost nothing to use. Depth Anything V2 Small, on an M4 Max, one 518×392 frame:

	per frame	processor time per frame
processor only	46 ms	114 ms
processor and GPU	13 ms	5.1 ms
processor and Neural Engine	23 ms	0.4 ms

The Neural Engine is not the fastest of the three — the GPU is. It is the one that leaves the machine alone: a frame costs four tenths of a millisecond of CPU, nearly three hundred times less than doing it on the processor. On a laptop that is also drawing a browser and syncing files, that is the number a person feels.

Which to ask for is therefore a real choice, and this package makes you make it.

A model has to be compiled first

What a model is distributed as — an .mlpackage — is not what Core ML runs. Compile turns one into the other with the system compiler, at run time, with nothing installed:

// Seconds, once. Then it is on disk and this call does nothing.
err := coreml.Compile("DepthAnythingV2SmallF16.mlpackage", "Depth.mlmodelc")

Where it is kept matters. macOS compiles into a temporary directory it is free to empty, so a program that does not move the result out pays the seconds again on every start. Compile moves it where you asked.

Then it is four calls

m, err := coreml.Open("Depth.mlmodelc", coreml.CPUAndNeuralEngine)
defer m.Close()

in := m.Inputs()[0]   // {Name:"image" Kind:image Width:518 Height:392}

res, err := m.Predict(map[string]coreml.Value{
    in.Name: coreml.Image(bgra, in.Width, in.Height),
})
defer res.Close()

plane, err := res.Plane("depth")   // one channel, as float32
png.Encode(w, &image.Gray{Pix: plane.Normalised(), ...})

Inputs is not decoration. A model refuses a picture of any size but the one it was trained on, so this is what to scale to — and asking beats hard-coding a number that changes with the next model.

Half floats, and why they are the interesting part

The Neural Engine speaks IEEE binary16, and Go has no float16. A plane comes back expanded to float32, subnormals included — which is not a detail: a depth model puts its nearest surfaces at the top of its range and its far detail down at the bottom, and the naive expansion reads that bottom as zero. Rows are padded, too, by an amount CoreVideo chooses; a decoder that assumes otherwise shears the image a little more on every row.

Both of those are portable arithmetic, and both are held at 100% coverage.

Everywhere else

On any platform that is not macOS, every constructor returns ErrUnsupported and the rest are no-ops. A program that offers a Neural Engine path and a portable one cross-compiles without build tags of its own.

What this is not

Images in, images out. Multi-array inputs and outputs, batches, and stateful models are not here — they are worth adding when something needs them, and not before.

Testing against a real model

The live test needs a compiled model, which a build machine has no reason to carry, so it is asked for by name and skipped when absent:

COREML_TEST_MODEL=/path/to/Something.mlmodelc go test ./...

Apple publishes CoreML builds of Depth Anything V2 under Apache-2.0, which is what the numbers above were measured with.

Install

go get github.com/go-macos/coreml

CGO_ENABLED=0. The only dependencies are purego and go-macos/objc.

Documentation

Overview

Package coreml runs a trained model on a Mac's Neural Engine from pure Go, with no cgo and no Xcode.

The Neural Engine is the part of an Apple chip that costs almost nothing to use. Depth Anything V2 Small, on an M4 Max, one 518x392 frame:

processor only              46 ms per frame, 114 ms of processor time
processor and GPU           13 ms per frame,   5.1 ms of processor time
processor and Neural Engine 23 ms per frame,   0.4 ms of processor time

The Neural Engine is not the fastest of the three. It is the one that leaves the machine alone: a frame costs four tenths of a millisecond of CPU, which is nearly three hundred times less than doing it on the processor. On a laptop that is also drawing a browser and syncing files, that is the number a person feels.

// once, and it is kept: compiling is slow and the result is durable
err := coreml.Compile("Depth.mlpackage", "Depth.mlmodelc")

m, err := coreml.Open("Depth.mlmodelc", coreml.CPUAndNeuralEngine)
defer m.Close()

in := m.Inputs()[0]              // says what size the model wants
res, err := m.Predict(map[string]Value{in.Name: coreml.Image(bgra, in.Width, in.Height)})
defer res.Close()

plane, err := res.Plane("depth") // one channel, as float32

A model has to be compiled before it can be opened

What a model is distributed as — an .mlpackage — is not what CoreML runs. Compile turns one into the other using the system compiler, at run time, with nothing installed. It takes seconds, so the result belongs somewhere durable: macOS compiles into a temporary directory it is free to empty, and a program that does not move the result out pays that cost on every start.

Everywhere else

On a platform that is not macOS every constructor returns ErrUnsupported, so a program that offers a Neural Engine path and a portable one cross-compiles without build tags of its own.

Index

• Variables
• func Compile(src, dst string) error
• type Feature
• type Kind
  • func (k Kind) String() string
• type Model
  • func Open(path string, units Units) (*Model, error)
  • func (m *Model) Close()
  • func (m *Model) Inputs() []Feature
  • func (m *Model) Outputs() []Feature
  • func (m *Model) Predict(in map[string]Value) (*Result, error)
• type Plane
  • func (p Plane) Normalised() []byte
• type Result
  • func (r *Result) Close()
  • func (r *Result) Plane(name string) (Plane, error)
• type Units
• type Value
  • func Image(bgra []byte, width, height int) Value

Variables

var ErrUnsupported = errors.New("coreml: only macOS has Core ML")

ErrUnsupported is returned on any platform that is not macOS.

Functions

func Compile

func Compile(src, dst string) error

Compile turns a distributed .mlpackage into the .mlmodelc that Core ML actually runs, and puts it where you asked.

It does nothing if dst already exists, because compiling takes seconds and the result does not change. Moving it out of the temporary directory macOS compiles into is the whole point: that directory is emptied whenever the system feels like it, and a program that leaves the result there pays the seconds again on every start.

Types

type Feature

type Feature struct {
	Name          string
	Kind          Kind
	Width, Height int
}

Feature describes one input or output of a model. Width and Height are set only for an image, and are what the model demands: a picture of any other size is refused, so this is the size to scale to.

type Kind

type Kind int

Kind is what sort of thing a model takes or returns.

const (
	// KindOther is anything this package does not yet describe: a dictionary,
	// a sequence, a number. Naming it is more useful than leaving it out,
	// because a model whose input is KindOther cannot be fed from here and the
	// caller should be told so rather than left guessing.
	KindOther Kind = iota
	// KindImage is a picture, and the only input this package can supply.
	KindImage
	// KindMultiArray is a tensor.
	KindMultiArray
)

func (Kind) String

func (k Kind) String() string

String names a kind, so that an error about the wrong one reads.

type Model

type Model struct {
	// contains filtered or unexported fields
}

Model is a compiled model, open and ready to predict.

func Open

func Open(path string, units Units) (*Model, error)

Open loads a compiled model and says which processors it may use.

func (*Model) Close

func (m *Model) Close()

Close gives the model back.

func (*Model) Inputs

func (m *Model) Inputs() []Feature

Inputs is what the model takes.

func (*Model) Outputs

func (m *Model) Outputs() []Feature

Outputs is what the model returns.

func (*Model) Predict

func (m *Model) Predict(in map[string]Value) (*Result, error)

Predict runs the model once.

The picture is copied into memory CoreVideo owns rather than wrapped where it lies. It is one copy of a small image against tens of milliseconds of inference, and it removes the whole question of what Core ML may still be holding when the Go slice is collected.

type Plane

type Plane struct {
	Width, Height int
	Values        []float32
}

Plane is one channel of an image output, as float32 whatever the model stored it as.

func (Plane) Normalised

func (p Plane) Normalised() []byte

Normalised stretches the plane onto 0..255 for looking at.

A depth model's scale is relative — it says which of two things is nearer, not how far either is — so an unstretched map of a scene with no sky in it is a uniform grey that tells you nothing about whether the model worked.

Values that are not numbers are left at zero rather than poisoning the range.

type Result

type Result struct {
	// contains filtered or unexported fields
}

Result is what a prediction returned. Close it.

func (*Result) Close

func (r *Result) Close()

Close releases the result. The planes already read out of it stay valid: they are Go memory.

func (*Result) Plane

func (r *Result) Plane(name string) (Plane, error)

Plane reads one named output as numbers.

type Units

type Units int

Units says which processors a model may run on.

It is a request, not an instruction: Core ML decides for itself, per layer, and a model with a layer the Neural Engine cannot do will run that layer elsewhere. Asking for CPUAndNeuralEngine is how a model reaches the Neural Engine at all — the default leaves it out.

const (
	// CPUOnly is the slowest and the most predictable.
	CPUOnly Units = 0
	// CPUAndGPU is usually the fastest in wall-clock time.
	CPUAndGPU Units = 1
	// All lets Core ML choose, which in practice means mostly the GPU.
	All Units = 2
	// CPUAndNeuralEngine is the cheapest in processor time by a wide margin.
	CPUAndNeuralEngine Units = 3
)

type Value

type Value struct {
	// contains filtered or unexported fields
}

Value is one input. Build one with Image.

func Image

func Image(bgra []byte, width, height int) Value

Image is a picture to feed a model: four bytes per pixel, blue green red alpha, no padding between rows. That order is CoreVideo's, and it is what a screen capture and a decoded video frame already are on this platform.