README
¶
depth
One picture in, two eyes out. Portable Go, no cgo, no model file — the same code on Linux, Windows, macOS and in a browser.
m := depth.Cues(img) // no model, no network
m = depth.Soften(m, 2)
left, right := depth.Views(img, m, depth.Options{MaxShift: 24})
A player has the same two eyes every frame, and at 4K a fresh pair is
sixty-six megabytes. ViewsInto writes into pictures that already exist:
err := depth.ViewsInto(left, right, img, m, depth.Options{MaxShift: 24})
Views takes any depth map, so a better one can be substituted without
touching anything else. On a Mac that is a real depth network on the Neural
Engine (go-macos/coreml) and the same
synthesis as compute kernels (go-macos/metal).
This package is the answer where neither exists, and the definition of what
they must compute.
One pass over the source, per row
The textbook way to move pixels sideways is to paint them all — far ones first, so near ones land on top, which is the whole of the occlusion handling. That needs a global sort of every pixel by depth.
What the sort is avoiding is two threads writing the same place. Splitting the work by row already prevents that, and then each source pixel simply has one destination in each eye, with the nearer winning where two collide.
Asking instead what could have reached each output column needs no sort either, and that is what this package did first — but it costs a short search per pixel and samples the depth map thirteen times as often. Replacing it was worth five times the speed at 4K.
The answer did not change. Checked against a GPU implementation of the painting rule, the two agree on 0 bytes out of 86 999 040.
Speed
Per frame, whole chain: depth from cues, softened, and both eyes synthesised.
| 960×540 | 1920×1080 | 3840×2160 | |
|---|---|---|---|
| M4 Max, 16 cores | 0.6 ms | 1.9 ms | 6.3 ms |
| Snapdragon 8+ Gen 1, 6 cores | 5.0 ms | 17.2 ms | 46.9 ms |
A telephone converts 1080p video to 3D faster than it plays, in portable Go with no hardware acceleration of any kind — which matters, because there is none to be had there: reaching Vulkan or the neural unit from Go needs cgo and the Android NDK.
Those numbers are ViewsInto. Allocating a fresh pair per frame costs about
twice as much on the telephone at 540p, and nothing at all on the Mac.
Soften is not cosmetic
A step in a depth map crossing the pixel grid makes an edge pixel jump about five pixels of disparity between one frame and the next. An edge that jumps like that reads as boiling, and it sits exactly where an eye looks.
It is real geometry, not noise — the pixel genuinely belonged to the near surface and now belongs to the far one. That is why nothing temporal fixes it: measured on a slow pan of a real photograph, a median over three frames removed none of it, and an exponential average bought a fifth of it for four frames of lag.
Softening the step spreads the jump over several frames instead:
| radius | worst edge movement | relief kept |
|---|---|---|
| 0 | 4.94 px | 3.76 |
| 1 | 1.79 px | 3.67 |
| 2 | 1.10 px | 3.58 |
| 4 | 0.63 px | 3.44 |
Radius 2 is the trade worth making: four fifths of the boiling gone for five per cent of the relief. Below half a pixel there is nothing left to see.
Cues is three honest guesses
Where there is no model, depth is estimated from the picture alone: what is lower in the frame is usually nearer, what is sharp is usually what was focused on, and what is neither very bright nor very dark tends to be the subject rather than sky or shadow.
It is wrong about a photograph of a wall and wrong about a picture taken looking down, and it costs a fraction of a millisecond. A real depth network is not comparable — but it needs a Mac, or a GPU, or a download, and this needs none of them.
Both eyes move
Each pixel moves by half its disparity, one eye each way, so the original stays in the middle. Generating only the second eye is cheaper and wrong: the whole film then feels as though it has slid sideways.
MaxShift is small on purpose. A large disparity makes an impressive still and
an unwatchable film, because the eyes must converge differently on every cut.
Holes
Where a near object moves aside it reveals something the camera never saw. There is nothing correct to put there, so the nearest filled pixel on the same row is stretched into it — a slightly smeared edge, which is what every real-time converter does. Left black instead, it is a flickering outline the eye finds instantly. Both directions, because a hole can open before any filled pixel on its row.
Install
go get github.com/go-images/depth
CGO_ENABLED=0, no dependencies at all, 100% test coverage on every platform.
Documentation
¶
Overview ¶
Package depth turns one picture into two, for a headset or a 3D display.
It does two separable things. It estimates how far away each pixel is — either from cues in the picture itself, with no model and no network, or from a Map you supply from something better. And it synthesises the two eyes from a picture and a Map.
m := depth.Cues(img)
m = depth.Soften(m, 2)
left, right := depth.Views(img, m, depth.Options{MaxShift: 24})
Everything here is portable Go with no cgo: it runs the same on Linux, on Windows, in a browser and on a Mac. On a Mac, a far better Map comes from a depth network on the Neural Engine (go-macos/coreml) and the synthesis can be done by compute kernels (go-macos/metal) — this package is the answer where those are not available, and the definition of what they must compute.
Three things here were measured rather than assumed ¶
The synthesis is ONE PASS OVER THE SOURCE columns, per row. The textbook way to move pixels sideways is to paint them all, far ones first, so near ones land on top -- but that needs a global sort of every pixel by depth. What the sort avoids is two threads writing the same place, and splitting the work BY ROW already prevents it: within one row each source pixel has one destination in each eye, and where two collide the nearer wins.
Asking instead what could have REACHED each output column needs no sort either, which is what this package did first -- but it costs a short search per pixel and samples the depth map thirteen times as often. Replacing it was worth five times the speed at 4K, and the answer did not change: checked against a GPU implementation of the painting rule, the two agree on 0 bytes out of 86 999 040.
Soften is not cosmetic. A step in a depth map crossing the pixel grid makes an edge pixel jump five pixels of disparity between one frame and the next — which reads as an edge boiling. It is REAL GEOMETRY, not noise: a temporal median removes none of it. Softening the step spreads the jump over several frames. Measured over a slow pan, radius 2 took the worst edge movement from 4.94 pixels to 1.10, for 4.8% less relief in the map.
MaxShift is small on purpose. A large disparity makes an impressive still and an unwatchable film, because the eyes must converge differently on every cut.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Views ¶
Views synthesises the two eyes from one picture and a depth map.
Each pixel moves sideways by half the disparity its depth calls for, one eye each way, so that the original stays in the middle.
It gathers: every OUTPUT pixel asks which source pixels could have reached it and keeps the nearest. That is the same rule as painting far pixels first and letting near ones land on top — last-writer-wins and highest-depth-wins are the same thing — but with no global sort and no two threads writing the same place, so it divides cleanly over the cores.
A nil source or an invalid map returns nil, nil: this is called per frame and an error to check on every one of them is an error nobody checks. Use ViewsInto when the reason matters, or when the pictures are being reused.
func ViewsInto ¶ added in v0.2.0
ViewsInto is Views writing into pictures the caller already has.
At 4K a pair of eyes is sixty-six megabytes. Allocating them per frame is most of a gigabyte a second of garbage, and on a telephone — where this package is the only path, there being no GPU binding and no neural engine — it was measured as three quarters of the frame time. Reuse the same two pictures and that cost disappears.
left and right must be the same size as src, and neither may be src.
Types ¶
type Map ¶
Map is how far away each pixel is: one byte each, 0 furthest, 255 nearest.
The scale is relative and says nothing about metres. It only has to order the surfaces correctly, which is all the synthesis asks of it.
func Cues ¶
Cues estimates depth from the picture alone, with no model and no network.
It is three cheap agreements about how photographs are usually taken, and nothing more:
- what is lower in the frame is usually nearer, which is the ground under the camera and carries most of the weight;
- what is sharp is usually what was focused on, and therefore nearer;
- what is neither very bright nor very dark tends to be the subject rather than sky or shadow.
It is wrong about a photograph of a wall and wrong about a picture taken looking down, and it costs a fraction of a millisecond. Where a real depth network is available — go-macos/coreml on a Mac — its map is not comparable, and Views takes either.
The estimate is made at quarter resolution and smoothed, because the sharpness term is per-pixel noise until it is: an unsmoothed cue map moves individual pixels sideways and looks like grain crawling over the picture.
func Soften ¶
Soften blurs a depth map, which is not cosmetic.
A step in the map crossing the pixel grid makes an edge pixel jump about five pixels of disparity between one frame and the next, and an edge that jumps like that reads as boiling. It is REAL GEOMETRY rather than noise — the pixel genuinely belonged to the near surface and now belongs to the far one — which is why nothing temporal touches it: a median over three frames removed NONE of it, and an exponential average bought a fifth of it for four frames of lag.
Softening the step spreads that jump over several frames instead. Measured over a slow pan of a real photograph, with a depth network's own map:
radius worst edge movement relief kept
0 4.94 px 3.76
1 1.79 px 3.67
2 1.10 px 3.58
4 0.63 px 3.44
Radius 2 is the trade worth making: four fifths of the boiling gone for five per cent of the relief. Below half a pixel there is nothing left to see, so radius 4 buys little more.
A box blur run twice separably, over the map rather than the picture, so it costs a fraction of a millisecond whatever size the frame is.
type Options ¶
type Options struct {
// MaxShift is the disparity of the nearest thing, in pixels of the
// source: the total between the two eyes, half of it each way, so that
// the original stays in the middle. Zero means 24, which is comfortable
// for a headset at arm's length.
//
// Generating only the second eye and leaving the first alone is cheaper
// and wrong: the whole film then feels as though it has slid sideways.
MaxShift int
}
Options control the synthesis.
Source Files