README
¶
depth
One picture in, two eyes out. Portable Go, no cgo, no model file — the same code on Linux, Windows, macOS and in a browser.
m := depth.Cues(img) // no model, no network
m = depth.Soften(m, 2)
left, right := depth.Views(img, m, depth.Options{MaxShift: 24})
A player has the same two eyes every frame, and at 4K a fresh pair is
sixty-six megabytes. ViewsInto writes into pictures that already exist:
err := depth.ViewsInto(left, right, img, m, depth.Options{MaxShift: 24})
Views takes any depth map, so a better one can be substituted without
touching anything else. On a Mac that is a real depth network on the Neural
Engine (go-macos/coreml) and the same
synthesis as compute kernels (go-macos/metal).
This package is the answer where neither exists, and the definition of what
they must compute.
One pass over the source, per row
The textbook way to move pixels sideways is to paint them all — far ones first, so near ones land on top, which is the whole of the occlusion handling. That needs a global sort of every pixel by depth.
What the sort is avoiding is two threads writing the same place. Splitting the work by row already prevents that, and then each source pixel simply has one destination in each eye, with the nearer winning where two collide.
Asking instead what could have reached each output column needs no sort either, and that is what this package did first — but it costs a short search per pixel and samples the depth map thirteen times as often. Replacing it was worth five times the speed at 4K.
The answer did not change. Checked against a GPU implementation of the painting rule, the two agree on 0 bytes out of 86 999 040.
Speed
Per frame, whole chain: depth from cues, softened, and both eyes synthesised.
| 960×540 | 1920×1080 | 3840×2160 | |
|---|---|---|---|
| M4 Max, 16 cores | 0.6 ms | 1.9 ms | 6.3 ms |
| Snapdragon 8+ Gen 1, 6 cores | 5.0 ms | 17.2 ms | 46.9 ms |
A telephone converts 1080p video to 3D faster than it plays, in portable Go with no hardware acceleration of any kind — which matters, because there is none to be had there: reaching Vulkan or the neural unit from Go needs cgo and the Android NDK.
Those numbers are ViewsInto. Allocating a fresh pair per frame costs about
twice as much on the telephone at 540p, and nothing at all on the Mac.
Soften is not cosmetic
A step in a depth map crossing the pixel grid makes an edge pixel jump about five pixels of disparity between one frame and the next. An edge that jumps like that reads as boiling, and it sits exactly where an eye looks.
It is real geometry, not noise — the pixel genuinely belonged to the near surface and now belongs to the far one. That is why nothing temporal fixes it: measured on a slow pan of a real photograph, a median over three frames removed none of it, and an exponential average bought a fifth of it for four frames of lag.
Softening the step spreads the jump over several frames instead:
| radius | worst edge movement | relief kept |
|---|---|---|
| 0 | 4.94 px | 3.76 |
| 1 | 1.79 px | 3.67 |
| 2 | 1.10 px | 3.58 |
| 4 | 0.63 px | 3.44 |
Radius 2 is the trade worth making: four fifths of the boiling gone for five per cent of the relief. Below half a pixel there is nothing left to see.
The map is interpolated, not snapped
A depth map is routinely a different size from the picture — a network has its own input size and does not care what it was given. At 4K one map pixel covers four to seven of the image's.
Taking the nearest map pixel manufactures a depth step every time the picture crosses a map pixel boundary. Measured on one photograph at 1080p: 2099 depth steps where the map itself holds 28 — on a regular grid, answering to nothing in the picture, and reading as a staircase across every smooth surface. At 4K it was 3694 against 20.
The arithmetic is integer throughout, and that is not an optimisation: the same synthesis exists as GPU kernels, and the two are checked against each other byte for byte. Floating point would agree almost always, which is the worst kind of agreement.
Both axes sample at pixel centres. Off by half a map pixel is three and a half pixels of disparity in the wrong place at 4K.
Reshaping depth before it becomes disparity
VITURE's own player reshapes the depth at this point — sigmoidCoef in its
Metal library — and this offers the same, as a table of 256 entries rather
than a formula. A table because the GPU implementation is checked against this
one byte for byte, and both can index the same bytes; a formula would have to
be reimplemented there in floating point and would agree almost always, which
is the worst kind of agreement.
opts := depth.Options{MaxShift: 24, Curve: depth.Sigmoid(4)}
fmt.Println(depth.DisparityOf(opts.Curve, opts)) // what it costs, in pixels
Sigmoid flattens both ends of the range and expands the middle. Read that
precisely, because the obvious reading is wrong: what is compressed is the
range at each end, not the disparity there. A near object ends up with
slightly more disparity, while the differences among near objects shrink.
The background flattens into one plane, the foreground into another, and the
middle — where the subject usually is — gets the relief they gave up. It is not
a comfort control.
And at a comfortable disparity it barely matters. Twenty-four pixels
between the eyes is twelve each way, so the whole depth range has thirteen
distinct shifts to spend, and quantisation swamps any reshaping of it: measured
at the default, a sigmoid of strength 4 changes the disparity of fewer than half
the depth values, by one pixel. It earns its keep only when the disparity is
large. DisparityOf is there so that can be checked rather than assumed.
Cues is three honest guesses
Where there is no model, depth is estimated from the picture alone: what is lower in the frame is usually nearer, what is sharp is usually what was focused on, and what is neither very bright nor very dark tends to be the subject rather than sky or shadow.
It is wrong about a photograph of a wall and wrong about a picture taken looking down, and it costs a fraction of a millisecond. A real depth network is not comparable — but it needs a Mac, or a GPU, or a download, and this needs none of them.
Both eyes move
Each pixel moves by half its disparity, one eye each way, so the original stays in the middle. Generating only the second eye is cheaper and wrong: the whole film then feels as though it has slid sideways.
MaxShift is small on purpose. A large disparity makes an impressive still and
an unwatchable film, because the eyes must converge differently on every cut.
Holes
Where a near object moves aside it reveals something the camera never saw. There is nothing correct to put there, so the nearest filled pixel on the same row is stretched into it — a slightly smeared edge, which is what every real-time converter does. Left black instead, it is a flickering outline the eye finds instantly. Both directions, because a hole can open before any filled pixel on its row.
Install
go get github.com/go-images/depth
CGO_ENABLED=0, no dependencies at all, 100% test coverage on every platform.
Documentation
¶
Overview ¶
Package depth turns one picture into two, for a headset or a 3D display.
It does two separable things. It estimates how far away each pixel is — either from cues in the picture itself, with no model and no network, or from a Map you supply from something better. And it synthesises the two eyes from a picture and a Map.
m := depth.Cues(img)
m = depth.Soften(m, 2)
left, right := depth.Views(img, m, depth.Options{MaxShift: 24})
Everything here is portable Go with no cgo: it runs the same on Linux, on Windows, in a browser and on a Mac. On a Mac, a far better Map comes from a depth network on the Neural Engine (go-macos/coreml) and the synthesis can be done by compute kernels (go-macos/metal) — this package is the answer where those are not available, and the definition of what they must compute.
Three things here were measured rather than assumed ¶
The synthesis is ONE PASS OVER THE SOURCE columns, per row. The textbook way to move pixels sideways is to paint them all, far ones first, so near ones land on top -- but that needs a global sort of every pixel by depth. What the sort avoids is two threads writing the same place, and splitting the work BY ROW already prevents it: within one row each source pixel has one destination in each eye, and where two collide the nearer wins.
Asking instead what could have REACHED each output column needs no sort either, which is what this package did first -- but it costs a short search per pixel and samples the depth map thirteen times as often. Replacing it was worth five times the speed at 4K, and the answer did not change: checked against a GPU implementation of the painting rule, the two agree on 0 bytes out of 86 999 040.
Soften is not cosmetic. A step in a depth map crossing the pixel grid makes an edge pixel jump five pixels of disparity between one frame and the next — which reads as an edge boiling. It is REAL GEOMETRY, not noise: a temporal median removes none of it. Softening the step spreads the jump over several frames. Measured over a slow pan, radius 2 took the worst edge movement from 4.94 pixels to 1.10, for 4.8% less relief in the map.
MaxShift is small on purpose. A large disparity makes an impressive still and an unwatchable film, because the eyes must converge differently on every cut.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DisparityOf ¶ added in v0.4.0
DisparityOf is what a curve does, in the unit that matters: how many pixels apart the two eyes put a thing at each depth.
It exists so a curve can be judged by a number rather than by adjectives.
func Sigmoid ¶ added in v0.4.0
Sigmoid builds an S-curve: it flattens both ends of the depth range and expands the middle.
Read that precisely, because the obvious reading is wrong. What is compressed is the RANGE at each end, not the disparity there: a near object ends up with slightly MORE disparity than a straight proportional shift would give it, while the differences AMONG near objects shrink. The far background flattens into one plane, the near foreground into another, and the middle — where the subject usually is — gets the relief they gave up.
So this makes the subject stand out. It is not a comfort control: a curve that made a close object easier on the eyes would have to lower the top of the range, and this raises it.
strength is how hard: 0 or less returns nil, which means no curve at all. Around 4 is a gentle S; 10 is severe. There is no measured "right" value here and this package does not pretend otherwise — what it costs and buys is visible in DisparityOf, and what it LOOKS like needs a headset and a person.
func Views ¶
Views synthesises the two eyes from one picture and a depth map.
Each pixel moves sideways by half the disparity its depth calls for, one eye each way, so that the original stays in the middle.
It gathers: every OUTPUT pixel asks which source pixels could have reached it and keeps the nearest. That is the same rule as painting far pixels first and letting near ones land on top — last-writer-wins and highest-depth-wins are the same thing — but with no global sort and no two threads writing the same place, so it divides cleanly over the cores.
A nil source or an invalid map returns nil, nil: this is called per frame and an error to check on every one of them is an error nobody checks. Use ViewsInto when the reason matters, or when the pictures are being reused.
func ViewsInto ¶ added in v0.2.0
ViewsInto is Views writing into pictures the caller already has.
At 4K a pair of eyes is sixty-six megabytes. Allocating them per frame is most of a gigabyte a second of garbage, and on a telephone — where this package is the only path, there being no GPU binding and no neural engine — it was measured as three quarters of the frame time. Reuse the same two pictures and that cost disappears.
left and right must be the same size as src, and neither may be src.
Types ¶
type Map ¶
Map is how far away each pixel is: one byte each, 0 furthest, 255 nearest.
The scale is relative and says nothing about metres. It only has to order the surfaces correctly, which is all the synthesis asks of it.
func Cues ¶
Cues estimates depth from the picture alone, with no model and no network.
It is three cheap agreements about how photographs are usually taken, and nothing more:
- what is lower in the frame is usually nearer, which is the ground under the camera and carries most of the weight;
- what is sharp is usually what was focused on, and therefore nearer;
- what is neither very bright nor very dark tends to be the subject rather than sky or shadow.
It is wrong about a photograph of a wall and wrong about a picture taken looking down, and it costs a fraction of a millisecond. Where a real depth network is available — go-macos/coreml on a Mac — its map is not comparable, and Views takes either.
The estimate is made at quarter resolution and smoothed, because the sharpness term is per-pixel noise until it is: an unsmoothed cue map moves individual pixels sideways and looks like grain crawling over the picture.
func Soften ¶
Soften blurs a depth map, which is not cosmetic.
A step in the map crossing the pixel grid makes an edge pixel jump about five pixels of disparity between one frame and the next, and an edge that jumps like that reads as boiling. It is REAL GEOMETRY rather than noise — the pixel genuinely belonged to the near surface and now belongs to the far one — which is why nothing temporal touches it: a median over three frames removed NONE of it, and an exponential average bought a fifth of it for four frames of lag.
Softening the step spreads that jump over several frames instead. Measured over a slow pan of a real photograph, with a depth network's own map:
radius worst edge movement relief kept
0 4.94 px 3.76
1 1.79 px 3.67
2 1.10 px 3.58
4 0.63 px 3.44
Radius 2 is the trade worth making: four fifths of the boiling gone for five per cent of the relief. Below half a pixel there is nothing left to see, so radius 4 buys little more.
A box blur run twice separably, over the map rather than the picture, so it costs a fraction of a millisecond whatever size the frame is.
type Options ¶
type Options struct {
// MaxShift is the disparity of the nearest thing, in pixels of the
// source: the total between the two eyes, half of it each way, so that
// the original stays in the middle. Zero means 24, which is comfortable
// for a headset at arm's length.
//
// Generating only the second eye and leaving the first alone is cheaper
// and wrong: the whole film then feels as though it has slid sideways.
MaxShift int
// Curve reshapes depth before it becomes disparity: 256 entries, indexed by
// the depth byte. Nil means none, which is a straight proportional shift.
//
// Build one with Sigmoid, and see what it does with DisparityOf. A table
// rather than a formula because the GPU implementation of this synthesis is
// checked against it byte for byte, and both can index the same table.
//
// A slice of any other length is ignored rather than refused: this is on
// the path of every frame, and a per-frame error is an error nobody checks.
Curve []byte
}
Options control the synthesis.
Source Files