depth3d

README

depth3d

One flat picture in, two eyes out — using the best path the machine has.

c, err := depth3d.New(depth3d.Options{Model: "Depth.mlpackage", MaxShift: 24})
defer c.Close()

err = c.Convert(left, right, stride, src, srcStride, w, h)

It is the same job whether the picture is a film frame or a captured desktop, and both callers in this organisation want it: player converts a flat film as it plays, and desk converts whatever the glasses are showing. The second of those is why this is a package rather than a file in one of them.

Two paths, and it always says which

With a Core ML depth model on a Mac, depth comes from a real network on the Neural Engine (go-macos/coreml) and the two views from compute kernels on the GPU (go-macos/metal). Measured on an M4 Max, a frame costs about 0.4 ms of processor time — which is what lets a compositor keep the rest of the machine.

With no model, or on any other platform, depth is guessed from cues in the picture itself (go-images/depth). It needs nothing at all, runs everywhere including a browser, and is visibly not as good.

Describe reports which one is running, and a fall back is always logged. A converter that quietly took the cheap path would look identical from the outside except for being worse.

The two eyes may be one frame

left and right may point into the same buffer — the two halves of a side-by-side frame, addressed by that frame's stride — or they may be two separate pictures. The player wants the first, the desk the second, and neither pays a copy for the other's preference.

What it cannot do

It cannot invent what the camera never saw. Where a near object moves aside, what is behind it is guessed from the same row, so an edge is a little smeared. That is the honest cost of the effect.

Curve is worth reading before turning on: an S-curve flattens both ends of the depth range and gives the middle their relief, so the subject stands out — it is not a comfort control, and a near object gets slightly more disparity. At a comfortable MaxShift it is swamped anyway: twelve pixels each way is thirteen distinct shifts for the whole range.

Testing

The portable layer — sizing, defaults, the fall back, and the synthesis on the processor — is held at 100%, gated in CI. The accelerated path needs a GPU and a depth model, which a build machine has neither of, so its tests skip:

XRKIT_TEST_MODEL=/path/to/Depth.mlpackage go test ./...

The kernels are a string compiled at run time, so nothing in an ordinary build looks at them; a test compiles every one of them, with a negative control. The curve test binds a table and checks it reaches the kernel, again with a control — without it, "the curve changed the picture" could be a rebound allocation.

Both implementations are checked against each other: on a real photograph with a real network's depth map, the GPU synthesis and the portable one agree on 0 bytes out of 86 999 040. Floating point would agree almost always, which is the worst kind of agreement, so the arithmetic is integer throughout and the curve travels as a table both sides index.

Install

go get github.com/go-xrkit/depth3d

CGO_ENABLED=0.

Documentation

Overview

Package depth3d turns a flat picture into two eyes, using the best path the machine has.

It is the same job whether the picture is a film frame or a captured desktop, and both callers in this organisation want it: go-xrkit/player converts a flat film as it plays, and go-xrkit/desk converts whatever the glasses are showing. The second of those is why this is a package rather than a file in one of them.

c, err := depth3d.New(depth3d.Options{Model: "Depth.mlpackage", MaxShift: 24})
defer c.Close()
err = c.Convert(left, right, stride, src, srcStride, w, h)

Two paths, and it always says which

With a Core ML depth model on a Mac, the depth comes from a real network on the Neural Engine and the two views from compute kernels on the GPU: a frame costs about four tenths of a millisecond of processor time, which is what lets a compositor keep the rest.

With no model, or on any other platform, the depth is guessed from cues in the picture itself (go-images/depth). It needs nothing at all, runs everywhere including a browser, and is visibly not as good.

Describe reports which one is running. A converter that quietly fell back would look identical from the outside except for being worse.

What it cannot do

It cannot invent what the camera never saw. Where a near object moves aside, what is behind it is guessed from the same row, so an edge is a little smeared. That is the honest cost of the effect.

Index

• Variables
• type Converter
  • func New(o Options) (Converter, error)
• type Options

Variables

var ErrNothingToConvert = errors.New("depth3d: nothing to convert in this frame")

ErrNothingToConvert is returned for a frame that cannot be turned into a pair — an empty picture, or a destination that does not match it.

Types

type Converter

type Converter interface {
	// Convert writes w by h pixels into left and right.
	//
	// left and right may point into the SAME buffer — the two halves of a
	// side-by-side frame — provided they do not overlap; stride is then that
	// frame's stride. They may equally be two separate pictures.
	Convert(left, right []uint32, stride int, src []uint32, srcStride, w, h int) error

	// Describe says which path is doing the work, for a log or a menu.
	Describe() string

	// Close releases whatever the path holds.
	Close()
}

Converter turns one flat frame into two eyes. It is not safe for concurrent use: it owns buffers sized to the last frame it saw.

func New

func New(o Options) (Converter, error)

New opens the best converter this machine has.

Everywhere but macOS that is the portable one: there is no Core ML to ask and no Metal to ask it with. A model named here is reported rather than ignored, because a caller who supplied one and got the cheap estimate should be told why.

type Options

type Options struct {
	// Model names a Core ML depth model — an .mlpackage or an already compiled
	// .mlmodelc. Empty falls back to the estimate from cues in the picture,
	// which needs nothing and is visibly not as good.
	Model string

	// MaxShift is how far apart the two eyes put the NEAREST thing, in pixels
	// of the source: the total between them, half of it each way, so that the
	// original stays in the middle. Zero means 24.
	//
	// Small on purpose. A large disparity makes an impressive still and an
	// unwatchable film, because the eyes must converge differently on every
	// cut.
	MaxShift int

	// Soften is how much the depth map is blurred before it moves any pixels.
	// Zero means 2, which is measured: it takes the worst movement of an edge
	// between one frame and the next from 4.94 pixels to 1.10, for 4.8% less
	// relief.
	Soften int

	// Curve reshapes depth before it becomes disparity, as an S-curve of this
	// strength. Zero means none.
	//
	// It flattens both ends of the range and gives the middle their relief, so
	// the subject stands out. It is NOT a comfort control: a near object gets
	// slightly MORE disparity. And at a comfortable MaxShift it is swamped by
	// quantisation — twelve pixels each way is thirteen distinct shifts for the
	// whole depth range — so raise MaxShift before expecting to see it.
	Curve float64

	// Log, when set, receives one line saying which path opened and why. It is
	// how a caller finds out that the accelerated path was unavailable rather
	// than discovering it as a slowdown.
	Log func(string)
}

Options configure a converter.