depth

README

depth

One picture in, two eyes out. Portable Go, no cgo, no model file — the same code on Linux, Windows, macOS and in a browser.

m := depth.Cues(img)                 // no model, no network
m = depth.Soften(m, 2)
left, right := depth.Views(img, m, depth.Options{MaxShift: 24})

Views takes any depth map, so a better one can be substituted without touching anything else. On a Mac that is a real depth network on the Neural Engine (go-macos/coreml) and the same synthesis as compute kernels (go-macos/metal). This package is the answer where neither exists, and the definition of what they must compute.

The synthesis gathers, it does not scatter

The obvious way to move pixels sideways is to paint them — far ones first, so near ones land on top, which is the whole of the occlusion handling. But that needs a global sort of every pixel by depth, and it forbids two threads from sharing a row.

Turned around, each output pixel asks which source pixels could have reached it and keeps the nearest. Last-writer-wins and highest-depth-wins are the same rule, so it is the same answer — with no sort and no shared write.

At 4K on sixteen cores that was 2.2× faster, and checked against a GPU implementation of the painting version it agreed on 0 bytes out of 66 355 200.

Soften is not cosmetic

A step in a depth map crossing the pixel grid makes an edge pixel jump about five pixels of disparity between one frame and the next. An edge that jumps like that reads as boiling, and it sits exactly where an eye looks.

It is real geometry, not noise — the pixel genuinely belonged to the near surface and now belongs to the far one. That is why nothing temporal fixes it: measured on a slow pan of a real photograph, a median over three frames removed none of it, and an exponential average bought a fifth of it for four frames of lag.

Softening the step spreads the jump over several frames instead:

radius	worst edge movement	relief kept
0	4.94 px	3.76
1	1.79 px	3.67
2	1.10 px	3.58
4	0.63 px	3.44

Radius 2 is the trade worth making: four fifths of the boiling gone for five per cent of the relief. Below half a pixel there is nothing left to see.

Cues is three honest guesses

Where there is no model, depth is estimated from the picture alone: what is lower in the frame is usually nearer, what is sharp is usually what was focused on, and what is neither very bright nor very dark tends to be the subject rather than sky or shadow.

It is wrong about a photograph of a wall and wrong about a picture taken looking down, and it costs a fraction of a millisecond. A real depth network is not comparable — but it needs a Mac, or a GPU, or a download, and this needs none of them.

Both eyes move

Each pixel moves by half its disparity, one eye each way, so the original stays in the middle. Generating only the second eye is cheaper and wrong: the whole film then feels as though it has slid sideways.

MaxShift is small on purpose. A large disparity makes an impressive still and an unwatchable film, because the eyes must converge differently on every cut.

Holes

Where a near object moves aside it reveals something the camera never saw. There is nothing correct to put there, so the nearest filled pixel on the same row is stretched into it — a slightly smeared edge, which is what every real-time converter does. Left black instead, it is a flickering outline the eye finds instantly. Both directions, because a hole can open before any filled pixel on its row.

Install

go get github.com/go-images/depth

CGO_ENABLED=0, no dependencies at all, 100% test coverage on every platform.

Documentation

Overview

Package depth turns one picture into two, for a headset or a 3D display.

It does two separable things. It estimates how far away each pixel is — either from cues in the picture itself, with no model and no network, or from a Map you supply from something better. And it synthesises the two eyes from a picture and a Map.

m := depth.Cues(img)
m = depth.Soften(m, 2)
left, right := depth.Views(img, m, depth.Options{MaxShift: 24})

Everything here is portable Go with no cgo: it runs the same on Linux, on Windows, in a browser and on a Mac. On a Mac, a far better Map comes from a depth network on the Neural Engine (go-macos/coreml) and the synthesis can be done by compute kernels (go-macos/metal) — this package is the answer where those are not available, and the definition of what they must compute.

Three things here were measured rather than assumed

The synthesis GATHERS rather than scatters. The obvious way to move pixels sideways is to paint them, far ones first, so near ones land on top — but that needs a global sort of every pixel by depth and it forbids two threads from sharing a row. Asking instead what could have reached each OUTPUT pixel and keeping the nearest gives the same answer, with no sort and no shared write. On sixteen cores that was 2.2 times faster at 4K, and the two agreed on 0 bytes out of 66 355 200.

Soften is not cosmetic. A step in a depth map crossing the pixel grid makes an edge pixel jump five pixels of disparity between one frame and the next — which reads as an edge boiling. It is REAL GEOMETRY, not noise: a temporal median removes none of it. Softening the step spreads the jump over several frames. Measured over a slow pan, radius 2 took the worst edge movement from 4.94 pixels to 1.10, for 4.8% less relief in the map.

MaxShift is small on purpose. A large disparity makes an impressive still and an unwatchable film, because the eyes must converge differently on every cut.

Index

• func Views(src *image.RGBA, m Map, opts Options) (left, right *image.RGBA)
• type Map
  • func Cues(src *image.RGBA) Map
  • func Soften(m Map, radius int) Map
  • func (m Map) Valid() bool
• type Options

Functions

func Views

func Views(src *image.RGBA, m Map, opts Options) (left, right *image.RGBA)

Views synthesises the two eyes from one picture and a depth map.

Each pixel moves sideways by half the disparity its depth calls for, one eye each way, so that the original stays in the middle.

It gathers: every OUTPUT pixel asks which source pixels could have reached it and keeps the nearest. That is the same rule as painting far pixels first and letting near ones land on top — last-writer-wins and highest-depth-wins are the same thing — but with no global sort and no two threads writing the same place, so it divides cleanly over the cores.

A nil source or an invalid map returns nil, nil: this is called per frame and an error to check on every one of them is an error nobody checks.

Types

type Map

type Map struct {
	Width, Height int
	At            []byte
}

Map is how far away each pixel is: one byte each, 0 furthest, 255 nearest.

The scale is relative and says nothing about metres. It only has to order the surfaces correctly, which is all the synthesis asks of it.

func Cues

func Cues(src *image.RGBA) Map

Cues estimates depth from the picture alone, with no model and no network.

It is three cheap agreements about how photographs are usually taken, and nothing more:

• what is lower in the frame is usually nearer, which is the ground under the camera and carries most of the weight;
• what is sharp is usually what was focused on, and therefore nearer;
• what is neither very bright nor very dark tends to be the subject rather than sky or shadow.

It is wrong about a photograph of a wall and wrong about a picture taken looking down, and it costs a fraction of a millisecond. Where a real depth network is available — go-macos/coreml on a Mac — its map is not comparable, and Views takes either.

The estimate is made at quarter resolution and smoothed, because the sharpness term is per-pixel noise until it is: an unsmoothed cue map moves individual pixels sideways and looks like grain crawling over the picture.

func Soften

func Soften(m Map, radius int) Map

Soften blurs a depth map, which is not cosmetic.

A step in the map crossing the pixel grid makes an edge pixel jump about five pixels of disparity between one frame and the next, and an edge that jumps like that reads as boiling. It is REAL GEOMETRY rather than noise — the pixel genuinely belonged to the near surface and now belongs to the far one — which is why nothing temporal touches it: a median over three frames removed NONE of it, and an exponential average bought a fifth of it for four frames of lag.

Softening the step spreads that jump over several frames instead. Measured over a slow pan of a real photograph, with a depth network's own map:

radius   worst edge movement   relief kept
     0             4.94 px          3.76
     1             1.79 px          3.67
     2             1.10 px          3.58
     4             0.63 px          3.44

Radius 2 is the trade worth making: four fifths of the boiling gone for five per cent of the relief. Below half a pixel there is nothing left to see, so radius 4 buys little more.

A box blur run twice separably, over the map rather than the picture, so it costs a fraction of a millisecond whatever size the frame is.

func (Map) Valid

func (m Map) Valid() bool

Valid reports whether the map's size and its bytes agree. A Map built by hand from a network's output is the likely caller, and one whose stride is wrong produces a picture that shears rather than an error.

type Options

type Options struct {
	// MaxShift is the disparity of the nearest thing, in pixels of the
	// source: the total between the two eyes, half of it each way, so that
	// the original stays in the middle. Zero means 24, which is comfortable
	// for a headset at arm's length.
	//
	// Generating only the second eye and leaving the first alone is cheaper
	// and wrong: the whole film then feels as though it has slid sideways.
	MaxShift int
}

Options control the synthesis.