android

README

go-xrkit/android

Screen capture on Android from pure Go, CGO_ENABLED=0, for an XR compositor that redraws every frame — and the two things around it that a virtual desktop needs: displays this application makes (Wall), and the glasses, painted from Go (Screen).

if !android.Authorized() {
        ok, err := android.RequestAuthorization(ctx) // the system consent dialog
        ...
}
d, _ := android.DefaultDisplay(ctx)

s, err := android.CaptureDisplay(ctx, d, android.Options{FPS: 60}) // a CEILING
if err != nil {
        return err
}
defer s.Close()

for {
        f, fresh := s.Frame() // BORROWED pixels, no copy, no allocation
        if fresh {
                composite(f.Pix, f.Width, f.Height, f.Stride) // RGBA, stride carried
        }
        render()
}

It is the Android member of a family — go-macos/screencapture, go-freedesktop/screencast, go-mswin/screencapture — deliberately shaped so one consumer drives all of them through near-identical adapters: Displays, CaptureDisplay, Stream.Frame() (Frame, bool) lending borrowed pixels with a stride carried and never assumed, an idempotent Close.

---

Read this first: what Android will and will not allow

The interesting question for an XR virtual desktop is not "can I capture the screen" — it is "can I create extra desktops and put applications on them", the way go-xrkit/desk does on macOS with real virtual displays.

Half of it is no, and the other half is yes — and the earlier edition of this file got the boundary wrong by drawing it in the wrong place. It is not your own app versus other people's, and it is not virtual display versus real one. It is an activity launch versus a window, on a display you made versus a display you were given:

	a display you were GIVEN (external, overlay)	a display you MADE (createVirtualDisplay)
launch an activity on it (setLaunchDisplayId)	yes — ours and Settings	no — SecurityException, even for our own activity
show a Presentation on it	yes	yes, and the pixels arrive

So an ordinary, unprivileged APK cannot manufacture desktops for other people's applications, which is what a macOS virtual display gives go-xrkit/desk. But it can manufacture several hundred independent displays of its own content, each rendered by the real Android view system and each read back as pixels. That is a great deal more than one mirrored phone, and it is what the Android ribbon is actually built on.

Everything below was read off a live Android 15 (API 35, arm64) system, not off documentation. host/../docs carries no prose about this: the transcript is the documentation. The probe that produced it is XrDisplayProbeActivity, and it is in the APK: every number below can be re-read by running it.

Creating a virtual display and launching something on it

what was asked for	what Android 15 answered
VIRTUAL_DISPLAY_FLAG_PUBLIC | PRESENTATION	SecurityException: Requires CAPTURE_VIDEO_OUTPUT or CAPTURE_SECURE_VIDEO_OUTPUT permission, or an appropriate MediaProjection token in order to create a screen sharing virtual display.
… | OWN_CONTENT_ONLY	created — display id 2, flags=0x8 [PRESENTATION]. Note what is absent: FLAG_TRUSTED.
launch Settings on it, ActivityOptions.setLaunchDisplayId(2)	SecurityException: Permission Denial: starting Intent { act=android.settings.SETTINGS … } from ProcessRecord{… org.goxrkit.probe/u0a151} … with launchDisplayId=2
launch our own activity on it	the same denial — an app may not put even itself on its own virtual display
… | VIRTUAL_DISPLAY_FLAG_TRUSTED (1 << 10)	SecurityException: Requires ADD_TRUSTED_DISPLAY permission to create a trusted virtual display.

But a Presentation on that same display works, and the pixels arrive

This is the door the first edition of this file never tried, and it was sitting in plain sight: the display it did create came back carrying the very flag that says a Presentation may be shown there. A Presentation is a Dialog attached to a Display — not an activity start — and the check that refused the table above is an activity-start check.

Q1 created display 17 "xr-probe-own-0" real=640x480 @320dpi (density 2.0) appBounds=640x480 refresh=60.0Hz flags=0xc state=2 valid=true
Q1 ANSWER: Presentation.show() SUCCEEDED on the app's OWN virtual display 17
Q2 frame 640x480 rowStride=2560 pixelStride=4 distinctColours>=64 black=0/307200 meanRGB=85 topLeft=#FF00FF00 topRight=#FFFF0000 bottomLeft=#FFFF0000 bottomRight=#FF0000FF
Q2 ANSWER: the PIXELS ARRIVED and are the ones the Presentation drew — green top-left, blue bottom-right, red elsewhere, at the exact sampled coordinates

No exception, no permission, no consent dialog, and no MediaProjection.

The second line is the one that matters, because a show() that returns and a buffer that arrives are two different facts, and a buffer that arrives black is the third. So the probe does not look at the picture: it draws solid quadrants of exactly known colour and samples four coordinates, then reports the black count and the distinct-colour count alongside them. black=0/307200 and four sentinels that hold is a frame that carries what was drawn. A run where the sentinels did not hold would be reported as a failure, not as a success.

The picture is there anyway, because a person should be able to look: presentation-own-virtual-display.png, 640×480, the quadrants plus a label drawn by Android's own text stack. It is not committed — see Where captures go.

Display.getFlags() answers 0xc for that display — FLAG_PRESENTATION|FLAG_PRIVATE. The 0x8 [PRESENTATION] quoted in the table above is dumpsys's view of the underlying device, which does not carry the private bit. Same display, two objects, and neither is FLAG_TRUSTED. The trusted bit gates the activity launch and nothing else here.

How WIDE: 32768 pixels, and the pixels arrive

The width is the whole point. go-xrkit/desk puts a 6400-pixel spreadsheet in front of somebody on macOS; the Android ribbon can only do the same if an owned display can BE that wide. MaxDimension says 32768, but that is this package's own guard — not a measurement of any particular phone, and a width that is accepted and then hands back a black buffer is the silent failure worth fearing.

Measured by cmd/xrwide on a Pixel 11 Pro Fold, Android 17 (API 37, arm64), one display per width, each wall closed before the next:

width	pixels sampled	wrong	black
1920 x 1080	779 581	0	0 / 2 073 600
3840 x 1080	1 558 621	0	0 / 4 147 200
6400 x 1080	2 597 341	0	0 / 6 912 000
8640 x 1080	3 506 221	0	0 / 9 331 200
16384 x 1080	6 648 349	0	0 / 17 694 720
32768 x 1080	13 296 157	0	0 / 35 389 440

Every width opened and every width carried its content: RESULT widest display that carried its pixels: 32768. Thirty-five million pixels at the widest, all correct, and the whole sweep takes about 700 ms.

⚠ The sweep distinguishes three answers, and the distinction is the point. A width can be refused — a limit to respect. It can be accepted and blank — a limit that LIES, and the one a naive check reports as a success. Or it can arrive with its pixels, sampled at the coordinates SentinelColorAt vouches for, which is the same yardstick cmd/xrwall uses so the two commands cannot disagree about what a correct panel looks like.

⚠ One wall per width, closed immediately. A 32768 x 1080 surface is 135 MB of BGRA on its own; measuring where the edge is must not be what pushes the process over it.

How many at once: 304, and then the system dies

Not a graceful refusal — that is the finding, and it is the reason a product must impose its own limit rather than discovering the platform's.

Q3 1920x1080 #303 id=307 OK — frame 1920x1080 rowStride=7680 pixelStride=4 distinctColours>=64 black=0/2073600 meanRGB=85 topLeft=#FF00FF00 …
E AndroidRuntime: *** FATAL EXCEPTION IN SYSTEM PROCESS: android.ui
E AndroidRuntime: android.view.Surface$OutOfResourcesException: NO_MEMORY
E AndroidRuntime: 	at android.view.SurfaceControl.nativeCreate(Native Method)
E AndroidRuntime: 	at com.android.server.wm.WindowContainer.createSurfaceControl(WindowContainer.java:678)
E AndroidRuntime: 	at com.android.server.wm.DisplayAreaPolicyBuilder$PendingArea.instantiateChildren(DisplayAreaPolicyBuilder.java:963)
E AndroidRuntime: 	at com.android.server.wm.DisplayContent.<init>(DisplayContent.java:1224)
E AndroidRuntime: 	at com.android.server.wm.RootWindowContainer.onDisplayAdded(RootWindowContainer.java:2769)
E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause

system_server takes the exception, dies, and the device soft-reboots. An ordinary APK with no permissions at all did that.

Two runs, and the number is worth reading carefully:

displays created	display size	result
304	640×480	#303 fine; creating #304 killed system_server
304	1920×1080	#303 fine; creating #304 killed system_server — the same count

The same count at nine times the pixels, with the app's Java heap at 14 MiB of 192. So it is not graphics memory: it is SurfaceControl handles, and DisplayContent's constructor builds a whole DisplayAreaPolicy tree of them per display. The ceiling therefore does not move if you make the screens smaller, which is exactly the mitigation somebody would reach for.

64 displays at 1920×1080, each with a Presentation whose pixels arrived correct, cost 14 MiB of Java heap and nothing else observable. A ribbon wants a dozen. The useful reading is "as many as you want, with a hard cap you set yourself" — and 304 is a number measured on one emulator, not a constant to hard-code.

VIRTUAL_DISPLAY_FLAG_TRUSTED is not merely undocumented — it is not in the public SDK at all. javap on the API 35 android.jar lists exactly five flags, and TRUSTED is not among them:

$ javap -classpath $ANDROID_HOME/platforms/android-35/android.jar \
        android.hardware.display.DisplayManager | grep VIRTUAL_DISPLAY
  public static final int VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR;
  public static final int VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY;
  public static final int VIRTUAL_DISPLAY_FLAG_PRESENTATION;
  public static final int VIRTUAL_DISPLAY_FLAG_PUBLIC;
  public static final int VIRTUAL_DISPLAY_FLAG_SECURE;

And the permissions that would unlock it are, on the same device:

$ adb shell pm list permissions -f
  permission:android.permission.ADD_TRUSTED_DISPLAY   protectionLevel:signature|role
  permission:android.permission.CAPTURE_VIDEO_OUTPUT  protectionLevel:signature
  permission:android.permission.CREATE_VIRTUAL_DEVICE protectionLevel:internal|role

signature means "signed with the platform key". internal|role cannot even be granted by signature. None of the three is reachable by an APK you or I can install.

VirtualDeviceManager (Android 14+) does not change the answer

It is the obvious thing to hope for, and it is closed twice over. The public SDK class carries no createVirtualDevice at all — only read-only queries:

$ javap … android.companion.virtual.VirtualDeviceManager
  public java.util.List<VirtualDevice> getVirtualDevices();
  public VirtualDevice getVirtualDevice(int);
  public void registerVirtualDeviceListener(…);
  public void unregisterVirtualDeviceListener(…);

The method does exist on the platform class, so it was called reflectively to get the platform's own answer rather than the header's:

vdm: createVirtualDevice EXISTS on the platform class:
     public VirtualDeviceManager$VirtualDevice createVirtualDevice(int, VirtualDeviceParams)
vdm: createVirtualDevice REFUSED:
     java.lang.SecurityException: Access denied, requires: android.permission.CREATE_VIRTUAL_DEVICE

So what an XR application on Android actually gets

Mirroring, plus a ring of its own screens. One panel carrying the phone's real screen, and as many independent displays of the application's own content as it cares to create — each a real Android display with a real view hierarchy on it, each read back as pixels through an ImageReader.

What it does not get is other people's applications on those screens. That half of the refusal is real and is not going anywhere: ADD_TRUSTED_DISPLAY is signature. The ring is ours to fill.

This is the same shape as the answer on Linux, where xrandr --setmonitor carves virtual monitors out of a real one with no privileged API at all.

---

The Go API for owned displays

The probe proved the platform allows it; this is how a consumer uses it. A Wall is a set of displays this application created, and the limit on how many may exist at once.

w, err := android.NewWall(0)          // 0 = DefaultMaxDisplays (8)
defer w.Close()

d, err := w.Open(ctx, android.DisplaySpec{
        Width: 1280, Height: 720,
        Content: android.Web{URL: "https://example.org/"},
})
if errors.Is(err, android.ErrTooManyDisplays) { ... }
defer d.Close()

for {
        f, fresh := d.Frame()          // BORROWED pixels, stride carried
        if fresh {
                composite(f.Pix, f.Width, f.Height, f.Stride)
        }
}

OwnedDisplay embeds *Stream, so a ribbon consumes a display it made exactly as it consumes the mirrored phone — Frame, WaitFrame, Stats, Format, Size, an idempotent Close. That is the point: the ribbon does not have to know which kind of panel it is holding.

NewWall(max int) (*Wall, error)	0 means DefaultMaxDisplays; past MaxDisplays, or negative, is ErrTooManyDisplays and no wall
(*Wall).Open(ctx, DisplaySpec) (*OwnedDisplay, error)	one display, one Presentation, one feed
(*Wall).Len() int, (*Wall).Max() int	how full, and how full it may get
(*Wall).Close() error	releases every display it still holds; idempotent
(*OwnedDisplay).ID() int	the android.view.Display id, for logs and dumpsys
(*OwnedDisplay).Spec() DisplaySpec	the spec, with its zero fields filled in
(*OwnedDisplay).Close() error	releases the display and gives the slot back; idempotent
SentinelColorAt(w, h, x, y) (uint32, bool)	the colour a correct sentinel frame holds there, or false where anti-aliasing makes it unsafe to assert

The drawing seam: a DECLARATION, not pixels

DisplaySpec.Content names what Android should render. The obvious alternative — hand the host a framebuffer to blit — was rejected, and the reason is worth stating because it looks like the natural design:

it would be a round trip with no product. Pixels this process drew, sent to Android, read back unchanged. An application that is drawing should composite in Go and skip the display entirely.

The reason to route a ribbon panel through a real Android display is the opposite one: to get at what Android renders and a CGO-free Go process cannot — a WebView, a MediaCodec surface, a PdfRenderer, a maps view. None is reachable without JNI, all are ordinary Views, and a Presentation is how a View gets onto a display. So Content names it, the Java host builds it, and the pixels come back through the same borrowed-frame path as a capture. It also keeps the rule the rest of this package is built on: a URL is a decision the application made; a WebView is the host obeying it.

Two implementations, and the set is closed — a Content written outside this package would name a Java class the host does not have:

• Web{URL} — a WebView. http, https and data: URLs; JavaScript on, file access off. A network URL needs android.permission.INTERNET (protectionLevel normal); a data: URL needs nothing.
• Sentinel{Label} — flat quadrants of exactly known colour. It is the package's self-test content and it is exported because a consumer needs it too: a feed that "works" and delivers a black buffer is the silent failure of this whole mechanism, and the only way to catch it is to draw something whose pixels are known in advance and then sample them.

The limit is the safety, and it refuses in both directions

Wall refuses past its limit with ErrTooManyDisplays before asking the host, so a full wall costs no round trip and creates nothing. XrWallService enforces MAX_DISPLAYS = 32 independently, because a process that does not use this API must not be able to get past it either.

DefaultMaxDisplays	8 — a ribbon uses six to eight
MaxDisplays	32 — the most NewWall accepts however explicitly it is asked
measured on one emulator	304, at which system_server died and the device rebooted

Read this before raising the limit. The ceiling was the same count at 640×480 and at 1920×1080 — nine times the pixels, same number, app heap at 14 MiB of 192. It is SurfaceControl handles, not graphics memory, and making the screens smaller does not help, which is exactly the mitigation somebody reaches for. 304 is one emulator's measurement, is deliberately not a constant anywhere in the code, and a real device's number is unknown and could be lower.

Proved on the device, in both directions:

LIMIT 2 of 2 opened, as they must
LIMIT refused past 2, as it must: android: too many virtual displays: this wall holds 2 of at most 2; close one before opening another
LIMIT NewWall refuses past the ceiling of 32, as it must

One connection per display

The capture host memoises a single process-wide session, because there is one screen and one projection. A wall is the opposite: several displays live at once, each with its own shared buffer and its own frame stream. Multiplexing them down one socket would mean a feed id on every frame message and routing on the hot path, where a bug mixes two panels' pixels.

A connection each costs one socket per panel — nothing, for the six to eight a ribbon uses — and buys three things: Stream is reused unchanged, including its tested frame plumbing; a display's lifetime is exactly a socket's lifetime, so closing one cannot disturb another; and a client that dies has its displays released by the kernel closing its sockets.

XrWallService is therefore a second service on its own socket (<package>.xrwall). It needs no MediaProjection, no consent dialog and no foreground service — an owned display asks for no permission at all — so opening a ribbon panel cannot put a "recording your screen" chip in the status bar, and a capture session ending cannot take a panel with it.

A Presentation shown from a Service context works, which was not obvious and was measured rather than assumed: Presentation is a Dialog, and a Dialog from a non-Activity context normally fails with BadTokenException. It does not here, on a private display the app owns.

What the live proof reports

cmd/xrwall, packaged into the APK and run on the device:

WALL available=true
WALL package default 8, ceiling 32
OPEN owned display 7 640x480 @320dpi sentinel 0
FRAME panel 0 640x480 stride 2560 seq 1: sampled 115921, wrong 0, black 0/307200
OPEN owned display 8 640x480 @320dpi sentinel 1
FRAME panel 1 640x480 stride 2560 seq 1: sampled 115921, wrong 0, black 0/307200
OPEN owned display 9 640x480 @320dpi sentinel 2
FRAME panel 2 640x480 stride 2560 seq 1: sampled 115921, wrong 0, black 0/307200
WALL holds 3 of 8
OPEN owned display 10 640x480 @320dpi web data:text/html,<body style='margin:0;background:%23FFFF00'>...
FRAME web panel 640x480 seq 2: 98.5% of the pixels are the page's own colour
RESULT every panel carried the pixels its content drew, and the limit refused in both directions

115 921 pixels asserted per panel, none wrong, none black — sampled at the coordinates SentinelColorAt vouches for, never by looking at the picture. The web panel is asserted the same way, on the page's own background colour rather than on the absence of an error. The artefact is wall-sentinel.png; it is not committed — see Where captures go.

APP=./cmd/xrwall host/build.sh && adb install -r host/out/xrhost.apk
# `--ez capture false` skips the mediaProjection foreground service, which the
# platform KILLS THE PROCESS for starting without the project_media app-op.
adb shell am start -n org.goxrkit.androidhost/org.goxrkit.android.XrDemoActivity --ez capture false
adb logcat -s xrcapture xr-wall
host/pull-artifacts.sh wall-sentinel.png

---

What a capture delivers

Measured, on the emulator named at the bottom of this file, with the cmd/xrcapture binary packaged into the APK:

DISPLAY display 0 "Built-in Screen" 1080x2400 @420dpi 60Hz (default)
capture 1080x2400 stride 4320 slots 3 ceiling 60.0 fps
lent 3 slots of 10368000 bytes
FIRST frame 1080x2400 stride 4320 (width*4 = 4320, padding 0) seq 1
RESULT frames=492 over 11.99s = 41.0 fps; identical-to-previous=68/491
RESULT Frame(): 22.8 ns/op, 0.000 allocations/op over 200000 calls
RESULT stats {Frames:493 Superseded:1 Interval:16.755125ms}

size	the display's native pixels, 1080×2400 here — no downscale is imposed
format	RGBA_8888 (android.graphics.PixelFormat 1), packed, 4 bytes per pixel
stride	4320 bytes, which happens to be exactly width*4 on this device. The API carries the number anyway, because a graphics allocator is free to pad and a sheared image is the classic result of assuming it does not
rate	41.0 fps sustained at 1080×2400 on a software-rendered emulator, with the capturing app in the background. A second run of the same binary, against a slower-changing screen, measured 18.5 fps — the ceiling is a ceiling, and what the screen does sets the rest. A Java-only probe of the same ImageReader path, with no Go in it, measured 48.0 fps, which bounds this package's own overhead at a few frames a second
Frame()	22.8 ns and 0.000 allocations per call, over 200 000 calls on the device. That is the borrow: Frame.Pix aliases the shared mapping the host wrote into

The content really changes

A capture that hands back a frozen buffer looks exactly like a working one until somebody hashes it, so cmd/xrcapture hashes every frame: 68 of 491 frames were identical to their predecessor and 423 were not, while another application scrolled in front. A run where every frame matched would be reported as a failure, not as a capture.

One frame is saved as a PNG artefact: testdata/artifacts/android-capture.png — 1080×2400, the Settings app scrolling, taken while this app was in the background. The cast indicator in the status bar is Android's own "something is recording your screen" chip.

Frames only arrive when something changes

Like ScreenCaptureKit on macOS, a MediaProjection is change-driven and Options.FPS is a ceiling, not a rate. Left in front of a static screen, the same cmd/xrcapture binary took 35 frames in the first 0.6 seconds and then nothing at all for the remaining eleven — RESULT frames=35 over 0.63s. That is the platform saying nothing moved; the second return value of Frame is the truth about it, not a timer.

Going to the background does not stop it

This matters more here than in most apps, because our app is the thing being covered. The probe deliberately calls moveTaskToBack(true) the instant consent is granted, and then captures another application for twelve seconds: onCapturedContentVisibilityChanged(visible=true), 492 frames, no interruption.

What is required for that is a foreground service of type mediaProjection, and from API 34 it is required anyway: the platform refuses to issue a projection token to a process that is not already running one. FOREGROUND_SERVICE_MEDIA_PROJECTION is protectionLevel:normal, so an ordinary APK simply declares it.

Consent, every session

MediaProjectionManager.createScreenCaptureIntent() must be answered by the user, the token is single-use, and nothing remembers the answer — unlike a macOS TCC grant. Authorized() therefore answers "right now", not "ever", and RequestAuthorization exists as a separate call because the dialog takes over the screen and an application should choose when that happens.

---

Reaching the glasses

On a phone with XR glasses on USB-C DP Alt Mode, Android sees an ordinary second Display. Two routes were exercised against a trusted, public secondary display (see the honesty note below about which one):

• Presentation — a window of ours on the other display while the phone shows something else. Presentation.show() succeeded, reporting presentation display=8 context size=1920x1080 dpi=320 — the whole display — and dumpsys window confirms it from the outside: Display: mDisplayId=8 … mObscuringWindow=Window{961b015 u0 org.goxrkit.probe}.

• ActivityOptions.setLaunchDisplayId — this one is the surprise. Launching our own activity worked, and so did launching Settings:

  Display #8 (activities from top to bottom):
      topResumedActivity=ActivityRecord{… com.android.settings/.homepage.SettingsHomepageActivity}
  Display #0 (activities from top to bottom):
      topResumedActivity=ActivityRecord{… org.goxrkit.probe/.Probe}
  

  The framework's rule is "anyone may launch on a public display"; the restriction that bites in the table above applies to virtual displays that are not trusted. So on the one display the glasses provide, an ordinary app can place other applications. It still cannot make a second one.

The display an app-created virtual display gets is flags=0x8 [PRESENTATION] as a device and 0xc [PRESENTATION|PRIVATE] as a Display. The system-created secondary display is flags=0x88 [PRESENTATION|TRUSTED], and the built-in panel is 0x4083. The trusted bit is the whole difference.

"A screen larger than the phone's" is the glasses' own display

A competing application shows, in the glasses, a screen bigger than the phone's panel. There is nothing clever behind it and it is worth saying so rather than inferring a mechanism: it is the external display, used at its own resolution. Nothing virtual is involved.

Q5 display 0  "Built-in Screen" real=1080x1920 @420dpi (density 2.625) appBounds=1080x1920 refresh=60.0Hz flags=0x4083
Q5 display 16 "Overlay #1"      real=1920x1080 @320dpi (density 2.0)   appBounds=1920x1080 refresh=60.0Hz flags=0x88
Q5 GIVEN display 16 "Overlay #1": Presentation.show() SUCCEEDED, presentation context 1920x1080 @320dpi
Q5 GIVEN display 16 after layout, decor 1920x1080

An ordinary app gets the whole of it: 1920×1080 at the display's own 320 dpi, not the phone's 420, and the decor view really lays out at 1920×1080 — measured after layout, not read off the metrics, because a window that reports a size and lays out smaller is the thing worth catching. The phone stays 1080×1920 at 420 dpi and is free to show something else.

One trap, since it produced a wrong number here first: asking getMaximumWindowMetrics() of a plain createDisplayContext(d) answers with the default display's bounds. It has to be a window context — createDisplayContext(d).createWindowContext(TYPE_APPLICATION, null) — and the wrong answer looks entirely plausible.

The phone as a trackpad

The device's own touchscreen keeps working normally while the content people are looking at is somewhere else, and the Presentation does not steal the events:

Q4 READY: presentation up on display 146, phone touches so far 0
TOUCH phone view ACTION_DOWN (340.00195,930.0) displayId=0
TOUCH phone view ACTION_UP (340.00195,930.0) displayId=0
…
Q4 ANSWER: phone view received 16 MotionEvents while its content was on display 146;
           the presentation view received 0; last phone event ACTION_UP (620.001,1140.0)

Sixteen MotionEvents to the view on display 0, zero to the Presentation's view — which is the half that makes a trackpad possible rather than a conflict. An app-created virtual display has no input device attached to it at all (its DisplayViewport is touch VIRTUAL), so there is nothing to arbitrate: the phone's digitiser belongs to the phone's window, and the pointer drawn into the ribbon is ours to place. Nothing new is needed on the Go side — go-widgets/android already forwards touch as MsgPointer; what changes is only where the application chooses to draw the cursor.

What a headset's Display reports — its Name is what the catalogue in go-xrkit/xrkit/glasses matches models on. For the built-in panel Android reports a device string ("Built-in Screen" on this emulator); for an external sink it comes from the EDID. No Android artifact for any real headset exists here — see below, and please read that section before believing anything about a specific model.

Painting on the glasses from Go

Everything above is about pixels coming from Android. This is the other direction, and it is what a headset is actually for: the glasses are an output, and what belongs on them is the ribbon the application composited.

ds, err := android.Screens(ctx)          // the displays that will take a presentation
scr, err := android.ShowOn(ctx, ds[0], android.ScreenOptions{})
defer scr.Close()

for {
    c, err := scr.Next(ctx)              // a slot to draw into, waiting if all are in flight
    draw.Draw(c.RGBA(), c.RGBA().Bounds(), src, image.Point{}, draw.Src)
    err = scr.Present(ctx, c)            // hand it over; the slot comes back on the ack
}

Screens asks the wall host, not the capture host, and that is the difference between needing a MediaProjection and needing nothing at all. An application that only wants to paint on the glasses would otherwise have had to start the whole projection apparatus — a foreground service, a consent dialog, a "recording your screen" chip in the status bar — to find out that a pair of glasses is plugged in.

It keeps only displays the platform flagged and that are not the built-in panel. The flag alone is not the answer: some devices set FLAG_PRESENTATION on the default display, and an application that believed it would ask for a window over the launcher and be refused by a check it had already passed.

The back-pressure is the point

Next hands out a slot, Present gives it to the host, and the slot comes back only once the host says the frame reached the view. Nothing else paces the application: the pixels go the other way here, so there is no stream of host frames to wait on, and overwriting a slot mid-blit is a tear in a headset.

ScreenOptions.QueueDepth is how many slots there are, bounded by the same MinQueueDepth..MaxQueueDepth as a capture. Screen.Stats().Waited counts the times the application was faster than the glasses.

A canvas is valid from the Next that returned it until the Present that hands it back, and presenting one twice is refused — including in the window before the acknowledgement arrives, which is most of the time and is exactly when a second Present gets made by mistake.

The host's stride, not the frame's width

Canvas.Stride is the distance between two rows and is what the host announced. Canvas.RGBA() hands back an image.RGBA over the same memory, so the whole of image/draw paints a frame with no copy; Canvas.Row(y) gives one row's bytes to a compositor that writes spans, with its capacity stopping at the row so a span cannot run into the padding and corrupt the next one.

⛔ What an acknowledgement is, and what it is not

cmd/xrscreen reports how many frames the host took. That number separates "the mechanism works" from "the mechanism runs and shows black", which is this feature's silent failure — and it is worth having for exactly that. On a Pixel 11 Pro Fold with VITURE Beast glasses:

TARGET display 13 "VITURE Beast" 1920x1080 @110dpi 60Hz (presentation)
SCREEN screen on display 13, 1920x1080, 7680-byte rows, 0 presented, 0 acknowledged, 0 waits
PAINTED 3600 frames of 1920x1080 in 2m0s, 30.0 fps
STATS 3600 presented, 3600 acknowledged, 0 waits

Zero waits over two minutes. The host stayed ahead of the application throughout — 248 MB/s of copies through the shared buffer — so the queue depth was never the thing limiting the frame rate. The display id is 13 here and 12 in an earlier run: ids are per attachment and are good for logs and dumpsys display, nothing more.

It does not say a photon left the panel. There is nothing to read back: an ordinary application may not capture a display it does not own, so unlike cmd/xrwide and cmd/xrwall there is no pixel to sample. The glasses are the only instrument and the person wearing them is the only witness, which is why what gets drawn is deliberately unmistakable — flat quadrants of known colour with a bar sweeping across — so that one glance settles the rest.

What the witness reported

Colour, and movement. The person wearing the Beast saw the quadrants, and saw them change, on the run transcribed above.

Both halves were asked for separately and both matter, because they refute different failures. Colour alone refutes a black panel — the one an acknowledgement count cannot see. Movement refutes a screen frozen on its first frame, which is what a host that showed one bitmap and then stopped copying would look like: 3600 acknowledgements and one picture. That is the whole reason the bar is in the frame rather than a still pattern, and it is why it was asked about on its own after "I saw colours" came back.

This is a person's report, written down as a person's report. It is not a measurement, it does not belong in a table with the frame counts, and it is the best evidence that exists for this half of the package.

APP=./cmd/xrscreen host/build.sh && adb install -r host/out/xrhost.apk
adb shell am start -n org.goxrkit.androidhost/org.goxrkit.android.XrDemoActivity \
    --ez capture false --es args '-wait 20m -frames 3600 -fps 30 -hold 60s'
# unplug the cable, attach the glasses, look
adb shell run-as org.goxrkit.androidhost cat files/screen.txt

It waits for a display rather than asking once, because the phone has one USB-C port and the glasses want all of it: the command is started over the cable that then has to be moved. A command that read the display list at startup could only ever report the phone's own panel.

Following a head: the census, and the answer it gave

An XR ribbon has to know where somebody is looking, and a VITURE Beast will not say. It holds its own 3DOF tracking and anchors the picture it is given with it, but publishes no orientation at all — measured three ways on 2026-09-07 and none of them produced a number. See go-xrkit/xrkit/headflow, which recovers the yaw from the headset's camera instead, at 1.14 % residual over a there-and-back sweep.

⭐⭐ On Android it now does too, measured: a worn left-right sweep came back as 197.9° of travel — out to −52°, back through zero, out to +80° — through usbfs on an unrooted phone. See the stream section below for the whole chain and for what it costs.

On macOS that camera is an AVFoundation device and the matter ends there. On Android it was not obvious an application could reach it at all, and there are exactly two routes — so this package asks rather than guessing:

cams, _ := android.Cameras(ctx)      // camera2, including LENS_FACING_EXTERNAL
devs, _ := android.USBDevices(ctx)   // every interface, down to its endpoints
route := android.ChooseRoute(cams, devs)

Neither needs a permission. android.permission.CAMERA is required to open a camera and a USB permission to open a device; being told one is there is free. So the census costs nobody a dialog, which is what makes it usable before anything is decided. It is served by the wall host for exactly that reason.

⛔ The answer on a Pixel 11 Pro Fold: neither Android API can read it

Attaching the Beast adds three USB devices and no camera2 device at all:

BEFORE  2 camera(s): back 4000x3000, front 3840x2800 — 0 USB device(s)
AFTER   2 camera(s) — unchanged — and 3 USB device(s)

35ca:1102 "VITURE Microphone"        audio + 3 HID interfaces
0c45:6368 "USB 2.0 Camera" (Sonix)   UVC, 6 alternate settings
   interface 1 alt 1..6 class 0x0e/0x02  ep 0x81 in ISOCHRONOUS max 128..5120
35ca:1201 "VITURE Beast XR Glasses"  CDC-ACM + CDC data + audio + HID
   interface 1 class 0x0a            ep 0x83 in BULK 512, ep 0x03 out BULK 512
   interface 5 class 0x03            ep 0x85 in INTERRUPT 64

ROUTE USB, isochronous UVC

Two things decide it, and both are measurements rather than readings of the documentation:

camera2 offers nothing	the list is identical before and after. Supporting external USB cameras is left to the vendor's HAL, and this one does not
every streaming endpoint is isochronous	and Android's Java USB API submits control, bulk and interrupt transfers and nothing else — there is no isochronous request in UsbDeviceConnection or UsbRequest, which is why every UVC library on Android carries a native libusb

So reaching those frames means usbfs ioctls on the descriptor UsbDeviceConnection.getFileDescriptor() hands out — USBDEVFS_SUBMITURB with USBDEVFS_URB_TYPE_ISO, which is plain syscalls and therefore reachable from CGO-free Go, and a great deal more work than either of the other two routes. Nothing of it is written here.

What IS open, which is not nothing

USBDevice.Readable keeps the interfaces whose IN endpoints Android's own API can carry. On the Beast that is three: the CDC-ACM control interface, its bulk data interface, and a HID interface. "The camera is closed" is not "there is nothing to read", and a census that could not tell unreachable from reachable and silent would be worth little — the 2026-09-07 finding is that the headset says nothing about its orientation, not that nothing can be asked.

⭐ usbfs WORKS, on an unrooted phone

The census said the frames are behind usbfs, so cmd/xrusb goes and asks. It takes four steps and reports which one the kernel stopped at: open, read the descriptors, claim the interface, select the alternate, and then submit one isochronous request — the thing UsbDeviceConnection cannot do at all.

Measured on a Pixel 11 Pro Fold, Android 17, no root:

TARGET usb /dev/bus/usb/001/003 0c45:6368 "Sonix Technology Co., Ltd." "USB 2.0 Camera"
STEP 1 open                  OK — the host lent the kernel's descriptor
STEP 2 read descriptors      OK — 1424 bytes, 0c45:6368 class 0xef/0x02/0x01
       streaming on interface 1 alternate 6, ep 0x81 in isochronous max 5120 interval 1
STEP 3 claim interface 1     OK — taken from whatever driver held it
STEP 4 select alternate 6    OK — bandwidth reserved

ANSWER YES — the kernel ACCEPTED an isochronous request on endpoint 0x81

⚠ Accepted is not streaming, which is why this step is only a step: the probe submits one request and cancels it, and that says the kernel takes the call, not that pixels arrive.

⭐ What this paragraph used to say was missing is now written and measured — the probe/commit negotiation, frame assembly from the packet descriptors, and decoding both YUY2 and MJPEG. A worn head turn comes back as 197.9° of travel; see the stream below. The distinction is kept rather than deleted, because it held for four releases: accepted was true here long before arriving was, and reading one as the other is the mistake this whole file is a record of.

⛔⛔ Three walls were reported here as platform limits. All three were OUR BUGS

This section said, at length and with measurements beside it, that no route existed and that head tracking needed a privileged component. That was wrong, three times over, and each mistake has the same shape: an operation was never attempted, and the thing that stood in for it was believed.

what was written	what it actually was
"UsbManager will not hand over the descriptor"	openDevice was never called. The code gated on hasPermission, which answers false on this phone while dumpsys usb records the grant for the same uid and the same node. A predicate that disagrees with the operation is a predicate, not a verdict
"permission denied reading the descriptors"	the descriptors were read by reopening /proc/self/fd/N. Reopening /dev/bus/usb/… is exactly what an application may not do — and the whole point of being handed a descriptor is that the open already happened
"the kernel refuses isochronous transfers"	EMSGSIZE. wMaxPacketSize is not a size on a high-speed isochronous endpoint: bits 10:0 are the packet, bits 12:11 the additional transactions per microframe. 5120 is 0x1400 → 1024 × 3 = 3072. The raw field was passed, and the kernel said "message too long" — which was printed as a refusal

And EBUSY on the claim — the kernel's own uvcvideo holding the interface — is answered by USBDEVFS_DISCONNECT, which needs no privilege and is what libusb's detach_kernel_driver does. USBHandle.Close reconnects it, because a detached interface takes /dev/video0 away from the whole phone.

⛔ The probe itself told the lie, and that is the worst of it. It printed "the kernel refused" for EMSGSIZE. It now switches on the errno: EPERM and EACCES are verdicts, everything else is a malformed request, and it says which.

What that costs to believe again

Every claim in this file about something being impossible was made with a measurement beside it, and three of them were still wrong — because the measurement answered a question about our code and was read as an answer about the platform. Before writing that a platform forbids something:

1. attempt the operation. A capability predicate is a hint; the call is the authority. hasPermission said no and openDevice said yes;
2. read the errno, not the failure. EPERM is a refusal. EMSGSIZE, EINVAL, ENODEV and EBUSY are all statements about the request;
3. look for a working third party. Play Store apps read UVC cameras on unrooted phones. That fact was available the whole time and would have refuted the conclusion on its own.

⭐ And the reason the descriptor is never handed over: the KERNEL OWNS IT

The camera is not broken, unsupported or asleep. It is DRIVEN:

/dev/video0  crw-rw----  system camera
1-1.1:1.0  driver=uvcvideo      # the UVC control interface
1-1.1:1.1  driver=uvcvideo      # the streaming interface
video0 -> …/usb1/1-1/1-1.1/1-1.1:1.0/video4linux/video0

The kernel's own uvcvideo holds both interfaces and has published a V4L2 node for them. Android refuses an application a USB device a kernel driver owns, which is why requestPermission answers without asking — and it is circular: usbfs exists precisely to take an interface away from a driver, and it is behind that same refusal. The microphone (snd-usb-audio) and the glasses' own HID and CDC interfaces are in exactly the same position.

And the V4L2 node is not reachable either, for a plainer reason

V4L2 uid=10350 groups=[9997 20350 50350] camera(1006)=false
V4L2 /dev/video0  REFUSED: open /dev/video0: permission denied

With android.permission.CAMERA granted, the process is still not in the camera group: the old mapping from that permission to AID_CAMERA is gone. The node is system:camera, so the ordinary Unix check refuses — and it refuses FIRST. There is no avc: denied for a video_device in the log at all, so SELinux was never consulted. "SELinux would block it" was the prediction; the measurement is simpler and more final than that.

The four routes, and which one works

route	state
camera2	shut. No external-camera HAL publishes the device; the list is identical before and after
UsbDeviceConnection	shut. Every streaming endpoint is isochronous, which the Java API cannot submit
/dev/video0	shut. Owned system:camera, and CAMERA grants no supplementary group any more
usbfs	⭐ OPEN. openDevice hands the descriptor over, USBDEVFS_DISCONNECT takes the interface from uvcvideo, and the kernel accepts an isochronous request

The first three are measured refusals and stand. The fourth is the one that works, and it took three self-inflicted walls to find.

The camera agrees to a format: 640×480 YUY2 at 25 fps

formats, _ := desc.UVCFormats(iface.Number)   // the class descriptors, which no
                                              // general USB parser reports
want, size, _ := android.PickFormat(formats)
neg, _ := h.Negotiate(iface.Number, want, size)

Measured on the device:

STEP 5 read formats          OK — 2 format(s) on interface 1
STEP 6 pick                  YUY2 640x480 @25fps (interval 400000)
STEP 7 negotiate             OK — format 2 frame 6 interval 400000 (25fps) payload 2400

⛔ Negotiate returns what the camera AGREED TO, not what it was asked. A UVC device may answer a probe with another frame or another interval and the host is expected to accept it; UVCNegotiation.Changed says when that happened. Reporting the request as the outcome is how a stream comes out at a rate nobody chose. This camera agreed exactly, so that branch is exercised against a fake one in the suite — the real device could not reach it.

⛔⛔ And the frame was chosen the wrong way round first

PickFormat took the largest frame, reasoning that more pixels across means more to match a column profile on. The camera's own table says what that costs:

MJPEG	30 fps at every size, 1920×1080 down to 160×120
YUY2	1920×1080 @5 fps · 1280×720 @10 · 800×600 @20 · 640×480 @25

So "largest" chose 1920×1080 at five frames a second — 200 ms a frame, worse than the 46–107 ms the same algorithm already achieves on macOS, for a precision nothing here uses. Latency is what head tracking is bounded by: headflow's own documentation says 1:1 locking wants under 20 ms, and choosing which screen somebody faces is comfortable at 107 ms. It bins columns four wide, so 640 across is already 160 bins.

Fastest now maximises the frame rate at or above MinTrackingWidth, and the pick is 25 fps instead of 5.

⚠ MJPEG is faster on this camera and is still not the default — but the reason changed. It used to be that there was no JPEG decode in the frame path. There is now: image/jpeg is in the standard library and is pure Go, so it needed no cgo and no exception to this module's CGO_ENABLED=0 — the decoder was in the toolchain all along.

What MJPEG costs is a decode, and whether that fits inside a frame interval depends on the processor that has to fit it. Measured on this workstation:

size	MJPEG decode	share of a 30 fps interval
320×240	1.04–1.09 ms	3 %
640×480	5.0–6.0 ms	15–18 %
1920×1080	22.8–27.9 ms	69–84 %

(YUY2 is 0.25–0.43 ms at 640×480 for comparison, from the same benchmark on the same machine, under load average 15.)

⛔ And those figures are a workstation's: an optimistic bound for the phone, not a measurement of it. The phone, measured through UVCStreamStats.Decoding:

	workstation	the Pixel
YUY2 640×480	0.25–0.43 ms	3.86 ms
MJPEG 640×480	5.0–6.0 ms	19.84 ms
MJPEG 320×240	1.04–1.09 ms	1.0 ms

Three and a half to ten times slower at 640×480, so MJPEG there is 60 % of a 30 fps interval where the Mac said 15–18 %. Keeping PickFormat on YUY2 until that run existed was the right call, and the run confirms it rather than merely permitting it — MJPEG also delivered 1.97 fps at StreamURBs = 4 and 3.21 at 16, for reasons that are not settled and are set out below.

What is written, and what is not

cmd/xrusb and the layer under it go from "the kernel accepts the request" all the way to pixels on an unrooted phone — measured, see the stream below, where the frame rate that remains short of the camera's own is also recorded. The parts most likely to be silently wrong are pinned:

• the ioctl numbers are computed from this package's own mirrors of the kernel's structs, and the suite asserts the results against the values usbdevice_fs.h produces. A mirror that drifted changes the number, and a wrong ioctl number is not a compile error and not necessarily even an error;
• the URB type is not the endpoint attribute. An endpoint whose bmAttributes say 1 is isochronous; a URB whose type is 1 is INTERRUPT. The mapping is a table rather than a function that can fail, because a transfer type is two bits and an error return there would be a branch no input can reach — which the coverage gate found;
• wMaxPacketSize is decoded, not passed. USBEndpoint.ISOPacketSize multiplies the low eleven bits by the transactions per microframe. The raw field cost a day: the kernel answered EMSGSIZE and it was published as a refusal;
• the descriptors are read with pread(2) on the lent descriptor, never by reopening a path to it.

The stream: reaping, assembling, decoding

USBHandle.Stream keeps StreamURBs = 16 isochronous requests in flight, reaps them without blocking, assembles payloads into frames and decodes YUY2 or MJPEG into an *image.RGBA — which is what headflow takes.

⛔ Sixteen in flight is not a tuning choice, and neither was the 4 it replaced. An isochronous endpoint is scheduled every microframe whether anybody is listening: between reaping one request and submitting the next, the camera's packets go nowhere. At StreamPacketsPerURB = 8 packets a request each covers four milliseconds, so the ring covers 16 ms of schedule. Four covered 4 ms, and that was the frame rate's ceiling for five releases — see The ring was the ceiling below. Thirty-two, on either axis, makes this camera stop sending altogether, which is why the next move was structural rather than another doubling.

⛔⛔ The reaping is a goroutine, and the decode is NOT in it. Frame used to reap, assemble and decode in turn, so every millisecond of decoding was a millisecond with nothing submitted — the starvation above, wearing the consumer's hat instead of the ring's. The reader now does nothing but reap, assemble and resubmit, and hands whole frames over a channel of depth MaxPendingFrames; the decode happens on the caller's goroutine, where its cost is the caller's. The handover keeps the freshest frames: when it is full, room is made at the FRONT and the shed frame is counted in Late, because a tracker wants the newest picture and the oldest is the one worth losing.

⚠ MaxPendingFrames is deliberately not StreamURBs. The two numbers answer different questions — how many requests are in flight, and how stale a frame may be before a tracker stops wanting it. They were the same constant until raising the ring made the shedding branch unreachable, and the coverage gate said so.

The decisions that are easy to get silently wrong, and what each one is:

the frame boundary is the FID toggle, not the end-of-frame bit	cameras that never set EOF exist, and on one of those a boundary waiting for it would run every frame into the next forever and deliver nothing. EOF is still honoured when it comes, because it ends a frame one payload earlier
bHeaderLength is the offset and is NOT constant	cameras use 2, 12 and more. A fixed two bytes prepends ten bytes of timestamp to every frame, which does not fail — it shears the image by five YUY2 pixels and looks like a lens mounted crooked
a short frame is dropped, never padded	a lost packet leaves a band missing, and zeros there are a black stripe that a tracker reads as structure. Worse than no frame
Late is counted apart from Dropped	a dropped frame is the bus's fault and says the link is lossy; a late one is OURS and says the consumer is slower than the camera. One counter for both makes a slow tracker look like a bad cable
luminance goes into all three channels	not a shortcut: headflow.Profile sums R+G+B down each column, so grey carries exactly what it reads, and every other byte of YUY2 is the luminance. A full YUV conversion would compute chrominance for it to add up and discard
a reaped URB is matched by ADDRESS, compared as a number	converting the kernel's answer back to a *usbdevfs_urb is the unsafe misuse go vet refuses, and nothing keeps that object alive across the round trip. The slot already knows its own memory

⚠ The third row is a YUY2 rule and not a universal one. A compressed frame's length is whatever the compressor made it, so "shorter than expected" describes a scene rather than a loss; what tells a finished MJPEG frame from one the bus cut in half is its markers. NewFrameAssemblerFor takes the format for exactly that reason — the two ways of deciding "whole" are not interchangeable, and either one applied to the other format produces nothing.

⛔⛔ One request can complete TWO frames, and the first version of this code returned only the last. A frame small enough to fit in fewer than eight packets finishes twice inside one drain, so Stats said two frames and the caller got one — and the next call then hung until its deadline. It does not happen at 640×480, where a frame is some two hundred packets, which is exactly why a test found it and reading the code would not have. Completed frames are queued now, and the queue has a ceiling that sheds the oldest, because head tracking wants the freshest.

Two things the device chooses rather than this process, and so are bounded:

• the memory the kernel writes into is held by the DESCRIPTOR, not by the stream. A submitted request has the kernel writing into this process's heap on the camera's schedule until that request is discarded or the fd closes. A UVCStream that became unreachable without Close would let its slots be collected while the kernel still held their addresses: a use-after-free performed by the kernel, landing in whatever the allocator handed out next, and reported by nothing. The handle pins them, tying that memory to the one thing whose closing actually stops the writes;
• a frame's size comes from two 16-bit descriptor fields, so a broken or hostile camera can ask for 65535×65535 — 8.6 GB, allocated twice over before anything else happens. MaxFrameBytes refuses it, with the check written as a division because the product it guards against is the one that overflows.

⭐⭐ The pixels arrive — measured on the device, 2026-10-11

STEP 7b alternate 5          OK — 2400 bytes an interval for a 2400-byte payload
STEP 8 stream                10 frame(s) in 1.061s
    10 frames, 12 dropped, 0 late, 0 bad payloads; 8232 packets (0 empty), 6846160 bytes
ANSWER 10 WHOLE FRAMES at 9.42 fps, luminance 7..57

Pixel 11 Pro Fold, VITURE Beast, no root. The luminance spread is reported rather than the frame count because a black frame looks exactly like a working stream to a counter — four runs gave 7..58, 6..57, 7..56 and 7..57, so there is a picture and it is stable.

Three defects stood between the agreement and the picture, and each one took the measurement to find:

⛔⛔ A cancelled request still completes, and still waits in the queue. The first run reported 0 frame(s) in 0s — 0 packets and the verdict read that as a silent camera. It was not: step 4's own SubmitISO discards its test request without collecting it, the completion queue belongs to the descriptor rather than to whoever submitted, and the stream picked that orphan up on its first reap:

STOPPED BY: … completed a request at 0x79f466a000 that this stream does not own

A question has to clear away its own litter, which reapOwn now does. And the stream skips and counts a foreign completion instead of dying on it, because one stray request from anything else reading the same device must not end a working camera.

⛔⛔ The alternate setting goes AFTER the commit, not before. With it selected first the camera sent nothing for fifteen seconds while the bus cheerfully delivered 119 464 empty packets — one request a millisecond, exactly the isochronous schedule, and no data at all. UVC takes the move away from the zero-bandwidth alternate, following a commit, as the signal to start sending; arriving there beforehand means that signal never happens. It is the order the kernel's own uvcvideo uses. PickAlternate then takes the smallest alternate that fits the committed payload — alternate 5, 2400 bytes for a 2400-byte payload, where the old code reserved 3072 bytes of every microframe to carry the same thing.

⛔ And a break that dropped its error cost a cycle on its own. 0 packets in 0s is not a quiet camera: a quiet one still has its endpoint scheduled and still fills the wait. The reason was sitting in a discarded variable. The probe prints it now, and switches on the errno.

⭐⭐ "The frame rate is ~10 and not 25, and that is the CAMERA" — measured three ways, and WRONG

⛔ The heading is kept in quotation marks because it is what this section originally claimed, and a heading is what a skimming reader takes away. It was us, not the camera; the refutation is in this section and in the next one.

Three runs on 2026-10-18, same phone, same session, after the phantom frames below were fixed:

	YUY2 640×480	YUY2 320×240	MJPEG 640×480
delivered	9.41 fps	10 fps	1.97 fps (3.21 at StreamURBs = 16)
dropped	1	0	67, every one missing its tail (68 at 16)
payload the camera committed to	2400 B	800 B	800 B
decode, on the phone	3.86 ms	1.0 ms	19.84 ms

⛔⛔ AND THE CONCLUSION DRAWN FROM THAT TABLE WAS WRONG. It said: flat at ~10 fps across a fourfold change in pixel count, so a ceiling on the camera's own frame production. Every number above was taken with StreamURBs = 4, and the ring was starving — the consumer worked for longer than four milliseconds and the packets arriving in the gap were gone. Raising it to 16, on the same phone in the same session:

StreamURBs	fps	dropped
4	10.5	79
8	21.5	14
16	23.8	0
32	0	the camera stops sending

23.8 fps against the 25 the camera advertises, and nothing dropped. The camera was never the ceiling; we were. At 320×240 the same change takes 10 fps and 10 drops to 14.8 fps and none — measured over 10 frames, which includes the start-up wait, so its sustained rate is not measured and the "flat across pixel count" claim is withdrawn rather than replaced.

⚠ The experiment that misled me was right about its own conditions. "Sixteen requests changed nothing" was measured with a consumer costing 0.3 ms, where a four-millisecond ring is ample. Ring depth matters exactly when the consumer costs more than the ring covers — which is the mechanism I had already measured as too small to matter and noted was "real and simply below threshold". It went above threshold the moment a tracker joined the loop, and I did not re-run the experiment under the new condition. A negative result carries its conditions with it.

⚠ MJPEG is slower on this camera, and WHY is an open question. It advertises 30 fps at every size and negotiates a third of the bandwidth (800 bytes an interval against YUY2's 2400); it delivered 1.97 fps at StreamURBs = 4 and 3.21 fps at 16, with every loss showing a start-of-image and no end-of-image — ten whole JPEGs out of 77 boundaries in the first run, 68 losses of the same shape in the second.

⛔⛔ I published that as "the camera truncates its own frames", and that conclusion is WITHDRAWN. A missing tail is where the bytes stop arriving; it does not say who stopped sending them. The same signature is exactly what starvation produces — and starvation was demonstrably still present in the second run, because the MJPEG decode measured 19.09 ms against a ring covering 16 ms, so the ring was empty for part of every frame. The honest statement is that the losses are located at the end of the picture and the cause is not established; mistaking a location for a cause is the same error this section already records twice.

The decode has since moved off the reaping path entirely (see The stream above), which removes that explanation's mechanism. The run that would settle it has not been taken: a rate for MJPEG with a decoupled reader is not measured here, and until it is, neither "the camera truncates" nor "it was our starvation" is the fleet's position.

⭐ The decode figures are now the phone's, and they are 3.5–10× the workstation's: 19.84 ms for MJPEG is 60 % of a 30 fps interval where the Mac said 15–18 %. Keeping PickFormat on YUY2 until this run existed was the right call, and the run confirms it rather than merely permitting it.

⚠ What the rate is enough for is the question that matters, and headflow answers it: choosing which screen somebody faces is comfortable at 107 ms, and 23.8 fps is 42. Holding a picture locked to the head at 1:1 wants under 20 ms, so a frame interval of 42 ms plus 4.8 ms of work is still short of it — but it is twice as close as the shipped default was, and the gap is now the camera's 25 fps rather than anything in this code.

⭐⭐⭐ And a real head turn, measured end to end

2026-10-18, the Beast worn and plugged into the phone, a left-right sweep. 30 frames captured with xrusb -keep 30, pulled back with xrrun -pull, replayed through headflow by go-xrkit/xrkit/cmd/headflowreplay:

frames	yaw
2 → 9	0° → −52°
9 → 17	back through 0.000°
17 → 25	0° → +80°
25 → 30	the return begins, +65.7°

197.9° of travel in 2.9 s, 27 of 30 frames tracked, one refused at confidence 0.257 — which reacquires, and loses the motion during the gap, exactly as that behaviour is documented.

⭐⭐ What convinces here is the SHAPE, not the number. Out, back through zero, out the other way: that is the gesture. No reference instrument existed to compare an angle against, so the shape of the movement was the only oracle available — and a tracker that was merely noisy, or drifting, could not produce it.

⚠ Confidence runs 0.30–0.94 and dips at the ends of the sweep, where the head moves fastest: motion blur, at 100 ms a frame. And the frames are pictures of a room and of whoever is wearing the headset — they went to the durable directory outside every work tree that go-appdirs/outdir chooses, and were deleted from the phone and the workstation once the measurement existed. That is the whole of the hygiene, and xrrun says so when it brings them back.

⭐⭐⭐ And live, on the phone: 0.9 % of a frame interval

cmd/xrhead streams and tracks in one process on the device, which the replay could not speak to: there the tracker ran on a workstation, against files, with no clock.

TRACKED 91 frame(s) in 8.001s — 11.4 fps
    86 used, 3 reference(s), 2 refused for want of confidence
    the tracker itself: 72ms in all, 790µs a frame
    the decode: 4.04ms a frame
    consecutive frames: 90 differed, 0 identical
ANSWER 748.5° of travel, -26.3° to 113.2°, over 86 matched frame(s)

790 µs of an 88 ms interval is 0.9 %, and the decode is another 4.6 %. The arithmetic is free — which was a prediction until this ran, and predictions have been wrong twice in this file.

⛔⛔ AND THAT RUN'S 11.4 fps WAS OUR OWN CEILING, WHICH THIS PROBE IS WHAT EXPOSED. It drops 101 frames for every 91 delivered, and a consumer costing 4.8 ms against a ring covering 4 ms is exactly the starvation this file had declared too small to matter. Raising StreamURBs to 16 gives 23.8 fps and nothing dropped; see the ring. The numbers above are kept as they were taken, because the probe that found the ceiling is the one that was running under it.

⛔⛔ The row that says 90 differed, 0 identical is there because two earlier runs read "the head did not move" and I could not tell whether that was true. 0° of travel with 89 frames matched at good confidence is consistent with a still head and with wiring that is blind — and the stream hands back a reused image, which is exactly the shape of a bug that would produce a confident zero. Comparing one row of the middle band between frames tells them apart, costs a few hundred byte comparisons against an 88 ms interval, and turned two unreadable runs into a measured negative: the scene really was static. An instrument that cannot fail is not an instrument.

⛔⛔ And the dependency this needed cost ONE LINE, against an argument this file made four times. "go-xrkit/xrkit pulls in HCL and eight indirect modules, and a USB layer has no business acquiring a configuration language" — measured, importing headflow adds one line to go.mod and two to go.sum, because Go prunes a dependency's requirements for packages you do not import and headflow imports only the standard library. The file interface that argument justified is still worth having — offline replay, and pictures a person can look at — but it was justified by a reason that was not true, and asserting a cost four times without measuring it is the thing to notice.

The per-boundary census, and the diagnosis it first produced

Before the fix below, the same run read:

boundaries: 11 toggle(s), 11 EOF bit(s); the dropped frames were short by
            564118 bytes each on average of a 614400-byte frame
the losses ended on: 11 toggle(s), 1 EOF bit(s)

⛔⛔ AND THAT READING WAS WRONG — THE BUG WAS MINE. Every one of those "dropped frames" was invented here. A header-only payload — a length, the bits, and nothing after them — is how a camera keeps an isochronous stream alive between frames. It is not an empty packet, so the zero-length skip never saw it, and it parses perfectly: opening a frame on it meant the next identifier toggle completed a buffer with nothing in it, once per real frame. Measured at 320×240 on 2026-10-18, one condition changed:

	frames	dropped	toggles
before	10	10	10
after	10	0	0

At 640×480 the same change took the dropped count from 12 to 1, and the toggle count to zero at both sizes. Short by exactly a whole frame is a frame that carried nothing — the counters said the bus was losing half the frames while the bus was losing none. The diagnosis this section used to carry, that the camera sent same-identifier packets after its end-of-frame bit, was this defect reading itself back.

⚠ A header-only payload does still end a frame that is already open: that is the legitimate way to close one whose data has all arrived, and ignoring it would hold every frame until the next toggle — a whole frame-time of latency added to the one method latency bounds.

⭐⭐ And the byte budget says why: the camera under-fills every interval

Worked through from the numbers of that same run, and it closes:

packets a second	7759 of 8000 high-speed intervals — 97 % of the schedule
bytes in each	832 of the 2400 it committed to — 34.7 % full
so, delivered	6.45 MB/s, which is 10.5 frames a second of data
needed for 25 fps	1920 bytes an interval — under the allowance
10 whole frames + 12 fragments	6 144 000 + 702 160 = 6 846 160 bytes, the measured total

So no packets are being lost at all — 97 % of the schedule arrives — and the frame rate is bounded by how much the camera puts in each packet.

⛔⛔ THE BUDGET CLOSED AND THE CAUSAL READING WAS BACKWARDS. Every figure in that table is real, and "the camera under-fills each interval" is a true description of what the bytes did. What it is not is a statement about the camera: with StreamURBs = 16 the same camera fills 77 % of each interval and delivers 23.8 fps. It under-filled because nothing was asking — the ring was dry while the consumer worked, and an isochronous endpoint with no request outstanding is a schedule with nowhere to put the bytes.

⭐ An identity that balances tells you which quantities are related, not which of them the device is holding fixed. I read "832 bytes an interval" as the camera's choice; it was ours. The way to tell is to vary one of them on the device, which the sweep above finally did.

⛔ Two of my own explanations died here, and one of them came back. That the ring was too shallow: refuted by sixteen requests delivering the same ten frames — and that refutation was wrong, because it was measured with a consumer costing 0.3 ms where four milliseconds of ring is ample. That the consumer starved it: refuted by a benchmark —

BenchmarkDecodeAFrame/640x480     0.25–0.43 ms/frame   0 allocs
BenchmarkAssembleAFrame            0.08–0.15 ms/frame   1 alloc

⚠ taken under load average 121, which this fleet has a rule against, and the numbers spread accordingly. The conclusion survives it because it is an order-of-magnitude question rather than a close one: a hole of 32 ms cannot be explained by 0.3 ms of work.

⛔⛔ AND THAT WAS THE RIGHT ANSWER TO THE WRONG QUESTION. cmd/xrusb's consumer costs 0.3 ms, so for it the ring is never starved — and I generalised that into "ring depth is not what bounds this", which held until a tracker joined the loop and the consumer cost 4.8 ms. Then the ring at 4 lost two frames in three. The benchmark measured the consumer I had; the conclusion was about every consumer there would ever be.

The prediction this earned: a smaller frame should reach the full rate. 320×240 at 25 fps needs 480 bytes an interval, well inside the 832 measured, and MinTrackingWidth is 320 so headflow still accepts it — four-wide bins make eighty of them.

⛔⛔ IT WAS WRONG, AND THE DEVICE SAID SO — TWICE, IN OPPOSITE DIRECTIONS. 320×240 delivered 10 fps, the same as 640×480, and I read that as the camera choosing both its byte rate and its frame rate. It was the ring choosing both: with StreamURBs = 16 the same 320×240 gives 14.8 fps and no drops, and 640×480 gives 23.8. The prediction was wrong, the refutation of it was wrong, and what settled it was varying the one thing neither had varied.

⭐ And MJPEG is in the frame path now, which was the other half of the prediction, and it is slower on this camera — 1.97 fps at a ring of 4, 3.21 at 16, for reasons not yet established. A compressed frame is whole by its markers and an uncompressed one by its length, so the assembler had to learn which it is holding: NewFrameAssemblerFor takes the format, and UVCNegotiation.Kind is where it comes from, because a negotiation result should say what was agreed rather than leave a stream to guess.

⚠ What a closed byte budget does and does not buy. It refuted two explanations that were wrong, which was worth the arithmetic. It then produced a prediction that was also wrong — because an identity that balances tells you which quantities are related, not which of them the device is holding fixed. Only varying one of them on the device says that.

⛔⛔ The ring depth WAS the cause, and the control that said otherwise was narrow

Deepening it looked obviously right: a 640×480 YUY2 frame at 2400 bytes an interval takes 32 ms, while four 8-packet requests cover four. The first measurement, with cmd/xrusb and its 0.3 ms consumer:

requests × packets	packets/s	whole frames
4 × 8	7744	10
16 × 8	7968	10
4 × 32	480	0

⛔ Read as "ring depth is not what bounds this", which held only for that consumer. With cmd/xrhead spending 4.8 ms a frame decoding and tracking, the same axis gives 10.5 → 21.5 → 23.8 fps and 79 → 14 → 0 dropped. The shipped default of 4 was halving the frame rate for five releases.

⚠ Thirty-two requests fails the same way thirty-two packets a request does — and neither is explained by their product, since 4 × 32 and 16 × 8 are both 128 packets and only one of them works. Two axes, two ceilings, measured not reasoned.

Thirty-two packets a request made the camera stop sending altogether — 480 packets in fifteen seconds, carrying eleven bytes each. Putting 8 back with the device in the same state streamed again immediately, which is what makes the ring the cause of that collapse and not the phone's leftover state.

Both lanes are at 100 % statement coverage and race-clean, and a fake camera drives the whole path from packets to pixels off the device.

⛔ That lock screen cost three measurement cycles, every one of them at the end of a build. It is now read before anything is built — tools/xrrun asks dumpsys and names which of the two conditions is in the way, screen or keyguard — and the host catches the exception instead of dying in onResume.

Input

Nothing new is needed, and that is the finding rather than an omission. A Bluetooth or USB keyboard reaches an Android app as ordinary KeyEvents, which go-widgets/android already forwards to the Go process as MsgKey (Android key code plus the rune its key-character map produced), alongside touch and wheel events. A ribbon scrolled from the keyboard therefore works through the existing back-end with no capture-specific code at all. The emulator enumerates a KEYBOARD | ALPHAKEY | DPAD device, so key injection reaches an app there too.

---

How it is put together

Android hands no MediaProjection, and no drawable surface, to a process that is not the app: both are behind JNI, and JNI needs cgo. So the app is two processes, exactly as go-widgets/android established:

Java host (host/, ~600 lines over three files)	owns the projection token, the ImageReader and the mirroring VirtualDisplay. Forwards bytes. Decides nothing.
Go application	an ordinary CGO_ENABLED=0 GOOS=android executable. Owns which display, which size, which rate, and when to stop.

Why a separate repository, and a Service rather than an Activity

An APK may hold many components but only one of them may own the drawing surface, and in a go-widgets application that one is go-widgets/android's GwHostActivity. Screen capture also has no business living inside a widget toolkit. So this is its own module, and its host is:

• a Service — which is mandatory anyway from API 34, as above;
• a transparent Activity used for nothing but holding the consent dialog, because consent is an activity result and nothing else.

The two hosts coexist as components of one package, each owning what it must, and the single Go process talks to both over two sockets. Neither host knows the other exists.

That composition has one practical wrinkle, and it is solved here rather than by changing go-widgets: the process is spawned by go-widgets' Activity, which knows nothing about capture and sets no socket name for it. Android gives a spawned process no way to ask its own package name either. What it does give it is HOME — which that host sets to /data/user/0/<package>/files — so android.DeriveSocket reads the package out of it and derives the host's socket name. XR_ANDROID_SOCKET overrides it when something does name one.

  MsgStart ─────────────────────► XrHostService ──► MediaProjection
                                        │             │
  MsgConfig ◄─── stride, slots ─────────┤             ▼
  MsgBuffer ◄─── SCM_RIGHTS fd ─────────┤          ImageReader
                                        │             │
  Frame() ◄── mmap(PROT_READ) ◄── SharedMemory ◄── one memcpy
                                        │
  MsgFrame{seq, slot, w, h, stride} ◄───┘

The frame buffer belongs to the HOST, and that is forced

The natural design — the Go side creates a memfd and lends it to the host, the way go-widgets/android lends its framebuffer — does not work in this direction, because the host has to write. An Android app cannot map a descriptor it received read-write. All three routes were tried on the device:

route	Android 15's answer
SharedMemory.fromFileDescriptor(pfd) on a Go memfd	IllegalArgumentException: FileDescriptor is not a valid ashmem fd
new RandomAccessFile("/proc/self/fd/" + fd, "rw")	FileNotFoundException: /proc/self/fd/112: open failed: EACCES (Permission denied)
FileInputStream/FileOutputStream .getChannel().map(READ_WRITE, …)	NonWritableChannelException — each of those channels is open one way only

So the host creates the region itself with SharedMemory.create() — ashmem, which dirties no page cache at all, so tens of megabytes a second of pure scratch never reach flash — and lends the descriptor to the application, which maps it PROT_READ. A stray write on the Go side traps instead of silently corrupting a frame under the compositor.

SharedMemory keeps its descriptor to itself, but writeToParcel is public and writes exactly that descriptor, so a Parcel is the supported way back to one.

Three slots, and why not two

Frame() lends; the host keeps producing. With two slots a host that produced two frames while the consumer held one would overwrite the borrow under it. The default and the minimum are therefore three, which is the same reasoning — and the same number — behind go-macos/screencapture's MinQueueDepth. A slot is a whole framebuffer (10.4 MB for a 1080×2400 phone), so MaxQueueDepth is a real memory limit rather than a formality.

One copy per frame, and there is no zero-copy path

The ImageReader plane is gralloc memory. Mapping that needs the graphics HAL, which is behind cgo — the one thing this stack must not have. So there is exactly one copy: direct ByteBuffer to direct ByteBuffer, which is a native memcpy with no Java heap in the middle and no allocation per frame. Measured on a 1080×2400 frame (10.37 MB): 0.325 ms, about 32 GB/s.

Everything after that is free: the Go side maps the same pages and Frame() returns a subslice.

---

Where captures go

A screen capture is a picture of whoever ran the probe, at work, and this repository is public. So a capture is never written where it could be committed, and a .gitignore does not count: git add -f, a fresh clone, or any tool that does not consult it will publish the file anyway.

The probe writes its PNG to the app's external files directory on the device, which is inside no repository. Pulling it to a workstation is the puller's business, and the puller must send it somewhere durable and outside every git work tree — os.UserConfigDir()-based, with an environment override, and the chosen directory walked up to the filesystem root looking for a .git, failing if it finds one. That is captureDir(t) in go-macos/screencapture, and the refusal is the point:

$ XRKIT_ARTIFACT_DIR=./testdata/artifacts host/pull-artifacts.sh …
REFUSED: …/testdata/artifacts is inside the git work tree at …

Nor does it go to a temporary directory. The artefact exists so that a person can look at it, and t.TempDir() would be gone before anyone could.

⛔ When the workstation cannot reach the phone, suspect the WORKSTATION

Every command here runs over a cable, and the phone has one USB-C port that the glasses want all of. The obvious escape is adb over Wi-Fi. When that fails the symptom is "I cannot reach my phone", and an evening went into a diagnosis that was wrong three times before it was right. The ending is worth more than the measurements, so it comes first.

What it turned out to be

A leftover macOS network extension — an uninstalled VPN client whose system extension was still registered and still intercepting traffic:

$ systemextensionsctl list
enabled active teamID     bundleID (version)                             [state]
        *      47R2M6779T dev.firezone.firezone.network-extension (1.5.8) [activated waiting for user]

Removing it in System Settings → General → Login Items & Extensions → Network Extensions fixed everything at once: ping, adb connect, and adb mdns services, which had been returning an empty list while macOS's own dns-sd found the phone perfectly.

The recipe, once nothing is in the way

Wireless debugging, on the phone's Settings → System → Developer options. The pairing port and the connect port are DIFFERENT, and both are on that screen:

adb pair 192.168.2.5:35325      # the pairing port, then type the 6-digit code
adb connect 192.168.2.5:46187   # the listening port, from the same screen

From then on adb mdns services lists the device by name and adb -s adb-<serial>-<suffix>._adb-tls-connect._tcp shell … reaches it with no address at all. While the extension was in the way, adb pair answered protocol fault (couldn't read status message) and adb mdns services came back EMPTY — both of which read as adb being broken rather than as something eating its traffic.

⛔ It was named as a suspect and then EXONERATED on bad reasoning

Three pieces of circumstantial evidence said it was inert, and all three were beside the point:

the argument	why it was worthless
no application in /Applications	an extension is registered with the system, not with the app. Deleting the app does not unregister it
no process in ps	a network extension runs as a system-managed provider, not as something with the vendor's name on it
[activated waiting for user]	a state string. The fact is the * in the active column, one field to its left, and it said the extension was active

The listing had the answer in a column and prose beside it, and the prose was read. Before clearing a component, check the field that states its status, not the one that describes it.

⛔ And the control varied the wrong end

The same failure appeared on two different access points — a macOS Internet Sharing hotspot and a Freebox on 6 GHz — which was taken as a control: vary the access point, the symptom stays, therefore the phone is the constant. The access point was varied. The phone was varied, by having a third machine ping it successfully. The workstation was never varied, and the workstation was the answer.

A control names what you changed. It cannot name what you did not think to change, and "everything else" is where the cause was sitting the whole time.

⭐ The ten-second test that should have come first

Mac     → phone   0/3
fractal → phone   2/2, 142–294 ms       ← a third machine, straight away

One line from a third machine splits the problem in half: either the phone is refusing everyone, or it is one path that is broken. It costs nothing, needs no theory, and it is the first thing to do rather than the last.

⚠ A failure that is too fast never left the machine

Attempts failed in 5 ms with EHOSTUNREACH, read here as an active ICMP rejection. It is not: it is macOS's negative ARP cache short-circuiting the attempt without putting a packet on the wire. After a reboot cleared it, the same ping took its full 4.5 seconds — a silent drop, a different mechanism with different causes. Read the duration before the error: a reachable host with a closed port takes about a second, and an instant failure is local state.

Local Network Protection: real, enabled, and NOT the cause

It was blamed here, at length, and it was wrong. From Android 17 the protection is mandatory and traffic to and from a local network address needs ACCESS_LOCAL_NETWORK, including accepting incoming TCP connections — see the documentation. On the device:

$ adb shell device_config get android_core_networking \
      android.permission.flags.access_local_network_permission_enabled
true

That is true, and it matters for any application in this repository that wants to be reachable: it must declare and request that permission, and nothing here does yet. It did not stop adb, which is a system daemon, and it never had anything to do with ping. The flag cannot be turned off from a shell in any case — device_config put answers "allowlist permission granted, but must add flag to the allowlist" on a release build.

Android 17 also ships ADB Wi-Fi 2.0, a rewritten wireless-debugging stack needing adb 37 or newer on the host. Wireless debugging works better on Android 17, not worse, and that fact was in hand early enough to have cast doubt on the whole diagnosis.

What this repository does regardless

Each command writes its verdict with android.SaveTranscript into the app's private directory, which survives the cable being somewhere else, and adb shell run-as <pkg> cat files/<name>.txt collects it afterwards. That was built for the one-port problem, it is unaffected by any of this, and every measurement in this README was taken that way.

Building the APK

No Gradle and no Kotlin: the host is three Java files and the application is a Go binary, so the SDK's own tools are the whole tool chain.

export ANDROID_HOME=... JAVA_HOME=...
sdkmanager --install "platforms;android-35" "build-tools;35.0.0"
host/build.sh                                # → host/out/xrhost.apk
adb install host/out/xrhost.apk
adb shell pm grant org.goxrkit.androidhost android.permission.POST_NOTIFICATIONS
adb shell am start -n org.goxrkit.androidhost/org.goxrkit.android.XrDemoActivity
adb logcat -s xrcapture xr-host

APP_BIN=/path/to/binary PACKAGE=org.example.app host/build.sh packages an application from another module; the flags are the same ones go-widgets/android's script takes, because it is the same script.

arm64 is the only CGO-free Android architecture — android/arm, android/amd64 and android/386 all answer "requires external (cgo) linking, but cgo is not enabled" — and CI asserts both halves of that, so the day Go lifts the restriction is a red build rather than a silent one.

Running a probe on a device

go run ./tools/xrrun -app ./cmd/xrusb -args '-wait 30s' -perms android.permission.CAMERA

That is one command for what used to be seven, and it is in Go because the mistakes were in the incantation rather than in the code under test: a transcript read before it was written and reported as "the probe said nothing"; a stale APK measured because the install was skipped; an argument string the shell split so that am read 5m as its target package and started nothing. Each cost a cycle and one of them cost a wrong conclusion.

It waits for the transcript file rather than for a duration, notices a crash instead of timing out on one, refuses to choose when two devices are listed — wireless adb lists the same phone twice — and finds the SDK and JDK itself rather than demanding two exports.

⛔ And it asks whether the phone can run anything at all, before it builds. Behind the lock screen Android does not credit the application as foreground, however resumed its Activity is, so the host's own startService throws and the process dies in onResume — three times, each at the end of a forty-second build. It reads dumpsys power and dumpsys window policy, because the wakefulness is in one and the keyguard in the other, and it names which condition is in the way:

xrrun: … is not usable: it is asleep and it is locked.
⛔ WAKE AND UNLOCK THE PHONE, AND LEAVE IT ON.
  adb -s … shell svc power stayon usb   # keeps it awake while plugged in

⚠ The strings it looks for were read off the phone, not taken from the crash transcript. The first version grepped window policy for mWakefulness=Dozing — a string that section never prints — so it would have passed a sleeping device, behind ten green hand-written expectations. isKeyguardShowing appears in neither section; the keyguard says showing=true and inputRestricted=true in its own delegate block. And screenState is SCREEN_STATE_ON while the phone sleeps, because of the always-on display, so a check on the screen is the wrong check. One real transcript is pinned as the fixture.

⚠ A section it cannot read is not a refusal: dumpsys gets renamed between releases, and a tool that refused to run because it could not find a field would be worse than the crash it avoids — the crash at least names its own cause.

host/build.sh stays a script, because aapt2, javac, d8, zipalign and apksigner in a row have no decisions in them.

⚠ cmd/xrusb leaves a debt. Claiming the camera's streaming interface detaches uvcvideo, and USBDEVFS_CONNECT does not put it back on this phone — the ioctl succeeds and the driver still does not rebind:

before:  1-1.1:1.0 driver=uvcvideo   1-1.1:1.1 driver=uvcvideo
after:   1-1.1:1.0 driver=uvcvideo   1-1.1:1.1 driver=

/dev/video0 survives on the control interface, but the streaming one is left with no driver until the headset is unplugged and plugged back. USBHandle.Detached reports what was taken, Close returns a reconnect it could not make — it used to swallow it — and the probe says so in its transcript. Taking something from the whole phone and saying nothing is worse than failing.

Testing

100.0% statement coverage, gated in CI on both lanes — the Linux transport and the off-Android stub. A stub nobody tests is a stub that panics the day somebody links it.

⚠ And a portable function whose only test is on one lane is uncovered on the other, which is how ISOPacketSize — the arithmetic that cost a day — came to be tested on Linux alone. The gate caught it from the darwin side.

The transport is //go:build linux, and Android is Linux, so the suite runs against a fake host over a real socket, a real memfd and a real SCM_RIGHTS handover — not a mock:

CGO_ENABLED=0 GOOS=android GOARCH=arm64 go test -c -cover -coverpkg=. -o xr.test .
adb push xr.test /data/local/tmp/ && adb shell /data/local/tmp/xr.test

Two tests are worth naming because they assert things a reading of the code cannot:

• TestFrameIsBorrowedNotCopied writes through the host's mapping and reads it back out of a Frame the consumer already holds. The whole no-allocation promise rests on that frame not being a copy.
• TestStrayDescriptorsAreNotLeaked throws eighty misplaced file descriptors at the client and then counts the process's own /proc/self/fd. A leaked descriptor is invisible until the process runs out of them, so it is measured rather than reasoned about.

-race runs on the Linux lane only: the race detector needs cgo, which is the very thing an Android application binary must not have.

---

What was actually tested against

This section distinguishes three things and never rounds upward.

Reproducing all of it

# 1. an arm64 Android 15 emulator (Apple Silicon or an arm64 Linux host)
sdkmanager --install "platforms;android-35" "build-tools;35.0.0" \
           "system-images;android-35;default;arm64-v8a" "emulator"
avdmanager create avd -n xr35 -k "system-images;android-35;default;arm64-v8a"
emulator -avd xr35 -no-window -no-audio -gpu swiftshader_indirect &
adb wait-for-device

# 2. the unit suite, on the device
CGO_ENABLED=0 GOOS=android GOARCH=arm64 go test -c -cover -coverpkg=. -o xr.test .
adb push xr.test /data/local/tmp/
adb shell 'cd /data/local/tmp && chmod +x xr.test && TMPDIR=/data/local/tmp ./xr.test'

# 3. the end-to-end capture, through a real APK
host/build.sh && adb install -r host/out/xrhost.apk
adb shell pm grant org.goxrkit.androidhost android.permission.POST_NOTIFICATIONS
adb shell appops set org.goxrkit.androidhost PROJECT_MEDIA allow   # skips the
                       # consent dialog; an adb-only shortcut, never an app one
adb shell am start -n org.goxrkit.androidhost/org.goxrkit.android.XrDemoActivity
sleep 3 && adb shell am start -a android.settings.SETTINGS    # something moving
for i in $(seq 1 26); do adb shell input swipe 540 1600 540 900 300; \
                         adb shell input swipe 540 900 540 1600 300; done
adb logcat -s xrcapture xr-host
adb pull /storage/emulated/0/Android/data/org.goxrkit.androidhost/files/android-capture.png

# 4. the virtual-display refusals, and the secondary-display results
adb shell settings put global overlay_display_devices "1920x1080/320"
adb shell pm list permissions -f | grep -A4 ADD_TRUSTED_DISPLAY
javap -classpath "$ANDROID_HOME/platforms/android-35/android.jar" \
      android.hardware.display.DisplayManager | grep VIRTUAL_DISPLAY

# 5. every Q1..Q5 transcript quoted above, from the probe in the APK.
#    A realistic panel first: a bare avdmanager AVD is 320x640 at 160 dpi.
adb shell wm size 1080x2400 && adb shell wm density 420
adb shell am start -n org.goxrkit.androidhost/org.goxrkit.android.XrDisplayProbeActivity
#    … and while it prints "Q4 READY", inject taps on the PHONE's touchscreen:
adb shell input tap 540 1200
adb logcat -s xr-probe
host/pull-artifacts.sh presentation-own-virtual-display.png

# 6. the ceiling. WARNING: this one KILLS system_server and reboots the device.
adb shell am start -n org.goxrkit.androidhost/org.goxrkit.android.XrDisplayProbeActivity \
    --es smallCap 512 --es bigCap 400

The probe takes --es smallCap N and --es bigCap N so the ceiling can be hunted without rebuilding, and defaults both to 64 — comfortably below the number that kills the device, because a probe whose default reboots the machine is a probe nobody runs twice.

Hardware connected and exercised

A Pixel 11 Pro Fold, Android 17 (API 37, arm64), with VITURE Beast glasses on its USB-C port. What that device answered, and nothing more:

• the glasses are an ordinary Android display — id 12 "VITURE Beast" 1920x1080 @110dpi 60.0Hz flags 0x8088 presentation true, named from the sink's own EDID;
• owned displays go to 32768×1080, with 0 wrong and 0 black pixels over 35 389 440 sampled — see cmd/xrwide;
• startForeground with mediaProjection and no consent kills the process on API 37, which is why the host only goes foreground once consent exists;
• and the glasses take a Presentation carrying pixels Go painted — cmd/xrscreen, 3600 frames of 1920×1080 in 2m0s at 30.0 fps, 3600 acknowledged, 0 waits. The queue never emptied: 248 MB/s of copies through the shared buffer without once making the application wait for a slot. The person wearing them saw colour and saw it move, which is the only evidence this half of the package can have;
• and the headset's camera is reachable through usbfs, on an unrooted phone: the descriptor is lent, uvcvideo is detached from the streaming interface, and the kernel accepts an isochronous request. Neither Android camera API offers it — no external camera2 device, and the Java USB API cannot submit isochronous transfers — so usbfs is the one route. See the census and what came of it.

TARGET display 13 "VITURE Beast" 1920x1080 @110dpi 60Hz (presentation)
PAINTED 3600 frames of 1920x1080 in 2m0s, 30.0 fps
STATS 3600 presented, 3600 acknowledged, 0 waits

Everything else below was measured on the Android emulator, API level 35 (Android 15), system-images/android-35/default/arm64-v8a, running on an Apple Silicon Mac — fingerprint Android/sdk_phone64_arm64/emu64a:15/AE3A.240806.019/12368160:userdebug/test-keys, 1080×2400 at 420 dpi. A real arm64 Android system, a real MediaProjection, a real SharedMemory, a real socket, real frames. It is not a real phone — and where a figure below comes from the emulator rather than from the Fold, it says so.

Partially observed

• The ceiling of 304, and the crash. Reproduced twice, at two sizes, on one emulator with a software renderer. The shape of the failure — no refusal, system_server dies of SurfaceControl exhaustion, the count does not move with the display size — is the finding and should carry. The number is this emulator's and must not be hard-coded anywhere.
• A secondary display. The Presentation and setLaunchDisplayId results come from the emulator's simulated secondary display (settings put global overlay_display_devices "1920x1080/320"), which reports type=OVERLAY, uniqueId="overlay:1", FLAG_PRESENTATION|FLAG_TRUSTED. A display attached over DP Alt Mode is type=EXTERNAL and is likewise public and trusted, and the framework's launch check keys on virtual versus not and on private/trusted rather than on the type — so the result should carry. It was not verified on a real external display, and "should carry" is not "verified".
• Frame rates. 41–48 fps at 1080×2400 was measured on a software-rendered emulator. A device with a GPU-composited surface should do better, but this repository has not measured one.

Known only from documentation

• Every published claim about a specific headset — display name, USB identity, the modes it offers — including everything in go-xrkit/xrkit/glasses. There is no Android artifact for any real headset in this repository at all. A macOS display name is not an Android one, and neither is an EDID read on Linux.
• That VITURE's own SpaceWalker uses this route on a phone over DP Alt Mode.

Send us hardware

We will gladly add and verify a model we can hold. If you want a device supported and verified rather than quoted, send us the hardware and it will be tested against and listed here. Failing that, the next best thing is to plug one in yourself and tell us what adb shell dumpsys display said.

Known gaps

Deliberate, and stated rather than hidden:

• most of the capture figures are still the emulator's — the Fold answered the display, wide-display, API 37 and camera-census questions; the frame rates, the ceiling of 304 and the consent flow were not re-measured on it, and the section above says which is which;
• capture of a second display is impossible for an unprivileged app and is reported as ErrNotCapturable rather than attempted. MediaProjection mirrors the default display; anything else needs CAPTURE_VIDEO_OUTPUT, which is signature;
• no application here is reachable from a workstation, because none declares ACCESS_LOCAL_NETWORK — mandatory from Android 17 for anything that accepts an incoming connection. The commands sidestep it by writing transcripts, which is why they work at all; making one reachable is unwritten;
• head tracking on Android WORKS, ON THE PHONE, LIVE. cmd/xrhead streams and tracks in one process: 748.5° of travel over 86 matched frames in 8 seconds, and the tracker itself costs 790 µs of an 88 ms frame interval — 0.9 %. A worn left-right sweep replayed offline comes back as 197.9° of travel — out to −52°, back through zero, out to +80° — through usbfs on an unrooted phone. The rate is 23.8 fps with nothing dropped, against the 25 the camera advertises, once StreamURBs is 16; at the 4 this shipped with for five releases it was 10.5 with two frames in three lost, and an entry here called that the camera's own ceiling. It was ours. MJPEG remains slower on this device (1.97 fps at a ring of 4, 3.21 at 16) and why is an open question: the losses all lack an end-of-image, which locates them without naming a cause, and an earlier entry here read that location as "the camera truncates its own frames". That reading is withdrawn. Enough for choosing which screen somebody faces, which headflow puts at 107 ms against this 42; still short of locking a picture to the head at 1:1, which wants under 20 ms. Three earlier entries here said this was impossible, a fourth said it was unwritten, and a fifth blamed the camera for a rate we were imposing; the first three were our own bugs, the fourth was stale for four releases, and the fifth read our own starvation as the hardware's limit. All of them are kept, because being wrong in that direction — about what the platform allows, and about whose ceiling a number is — is the more useful half;
• nobody has read a pixel back off the glasses, and nobody can. An ordinary application may not capture a display it does not own, so Screen can report how many frames the host took and nothing more. The transport is measured end to end against a real socket and a real shared mapping — what the application draws is asserted to be the bytes the host finds — but between the host's bitmap and the panel there is only a person looking, and that will not change: the gap is in what Android permits, not in what is written here;
• only two Content kinds exist. Web and Sentinel. MediaCodec, PdfRenderer and a maps view are the obvious next ones and none is written;
• the limit is 32 by construction, not by measurement on hardware. It is enforced in both the Go API and the Java host, and it is far below the 304 that killed system_server on an emulator. What a real phone does at 32 owned displays has not been measured, and neither has what it does at 304;
• the trackpad is a finding, not a feature. That the phone's MotionEvents keep arriving while the content is elsewhere, and that the Presentation gets none of them, is measured. The pointer that a ribbon would draw from them does not exist yet;
• the row stride is measured, not guessed — but there is a fallback. The host waits up to two seconds for its first Image to learn the allocator's real stride. If none arrives (a screen that is off, say) it announces an aligned upper bound instead, and refuses — loudly, with MsgStopped — any later frame that would not fit. That path has not been provoked on a device.

License

BSD-3-Clause. See LICENSE.

Documentation

Overview

Package android captures the Android screen from a pure-Go, CGO-free application, for an XR compositor that redraws every frame.

What Android permits, and what it does not

Everything here is shaped by one measured fact: an ordinary, unprivileged APK may MIRROR the screen and may draw its own content on an attached display, and it may do nothing else. It cannot give itself extra desktops. On Android 15 (API 35), creating a virtual display and launching an activity on it fails with

SecurityException: Permission Denial: starting Intent { ... }
    ... with launchDisplayId=N

for ANY activity, including the caller's own; asking for VIRTUAL_DISPLAY_FLAG_TRUSTED — which is not in the public SDK at all — fails with "Requires ADD_TRUSTED_DISPLAY permission", a signature|role permission; and VirtualDeviceManager.createVirtualDevice fails with "Access denied, requires: android.permission.CREATE_VIRTUAL_DEVICE", which is internal|role. See the README for the whole transcript. So on Android an XR ribbon carries the phone's real screen, and the application's own content, and that is all.

The split, and why there is a Java host

Android hands no drawable surface, and no MediaProjection, to a process that is not the app: both are behind JNI, and JNI needs cgo. So the app is two processes, exactly as in go-widgets/android — a small Java host owning the Android objects, and an ordinary CGO_ENABLED=0 GOOS=android executable owning everything above the pixels. This package is the application half.

The hot path

Stream.Frame hands back a BORROWED view of the most recent frame: the bytes are the shared mapping the host wrote into, not a copy, and the borrow lasts until the next Stream.Frame, Stream.WaitFrame or Stream.Close. In steady state a Frame call performs no allocation at all. Copy out with Frame.CopyTight or Frame.NRGBA to keep it longer.

Stride

A captured frame's rows may be PADDED: Stride is the number of BYTES per row and it is not necessarily Width*4. Always index with Stride, or use Frame.Row. The Android 15 emulator measured here reported stride 4320 for a 1080-wide capture — exactly Width*4, no padding — which is precisely why the API carries the number instead of trusting it.

Frames only arrive when something changes

A MediaProjection capture is change-driven, like ScreenCaptureKit on macOS. Options.FPS is a CEILING, not a rate: with a still screen, this package's own probe took 39 frames in 4.0 s and then nothing at all; with the screen scrolling it took 574 frames in 11.9 s. Do not treat a missing frame as a failure; the second return value of Stream.Frame is the truth about whether anything changed.

Consent

Every capture session needs the user to agree through a system dialog, and the agreement dies with the session. Authorized reports whether the host already holds a projection token; RequestAuthorization asks the host to put the dialog in front of the user and blocks until they answer.

Index

• Constants
• Variables
• func ArtifactDir() (string, error)
• func Authorized() bool
• func Available() bool
• func ControlIn() uint8
• func ControlOut() uint8
• func DecodeBuffer(b []byte) (slots int, slotSize int64, err error)
• func DecodeConsent(b []byte) (bool, error)
• func DecodeOpenUSBDevice(b []byte) (string, error)
• func DeriveSocket(env string, envOK bool, home string) string
• func DeriveWallSocket(env string, envOK bool, home string) string
• func EncodeBuffer(slots int, slotSize int64) []byte
• func EncodeCameras(cs []Camera) []byte
• func EncodeConfig(c ConfigMessage) []byte
• func EncodeConsent(granted bool) []byte
• func EncodeDisplays(ds []Display) []byte
• func EncodeError(e ErrorMessage) []byte
• func EncodeFrame(f FrameMsg) []byte
• func EncodeOpenDisplay(m OpenDisplayMessage) []byte
• func EncodeOpenScreen(m OpenScreenMessage) []byte
• func EncodeOpenUSBDevice(name string) []byte
• func EncodePresent(m PresentMessage) []byte
• func EncodePresented(m PresentedMessage) []byte
• func EncodeStart(s StartMessage) []byte
• func EncodeStopped(s StoppedMessage) []byte
• func EncodeUSBDevices(ds []USBDevice) []byte
• func FrameMessage(typ uint8, body []byte) []byte
• func PickAlternate(ifaces []USBInterface, number int, payload uint32) (USBInterface, USBEndpoint, bool)
• func PickFormat(fs []UVCFormatDesc) (UVCFormatDesc, UVCFrameSize, error)
• func ReadMessage(r io.Reader) (typ uint8, body []byte, err error)
• func RequestAuthorization(ctx context.Context) (bool, error)
• func SaveFrame(name string, img *image.RGBA) (string, error)
• func SaveTranscript(name, text string) (string, error)
• func SentinelColorAt(w, h, x, y int) (uint32, bool)
• func WallAvailable() bool
• func WriteMessage(w io.Writer, typ uint8, body []byte) error
• type Camera
  • func Cameras(ctx context.Context) ([]Camera, error)
  • func DecodeCameras(b []byte) ([]Camera, error)
  • func (c Camera) External() bool
  • func (c Camera) String() string
• type CameraFacing
  • func (f CameraFacing) String() string
• type CameraRoute
  • func ChooseRoute(cams []Camera, devs []USBDevice) CameraRoute
  • func (r CameraRoute) String() string
  • func (r CameraRoute) Usable() bool
• type Canvas
  • func (c Canvas) RGBA() *image.RGBA
  • func (c Canvas) Row(y int) []byte
  • func (c Canvas) String() string
• type ConfigMessage
  • func DecodeConfig(b []byte) (ConfigMessage, error)
• type Content
• type Display
  • func DecodeDisplays(b []byte) ([]Display, error)
  • func DefaultDisplay(ctx context.Context) (Display, error)
  • func Displays(ctx context.Context) ([]Display, error)
  • func Screens(ctx context.Context) ([]Display, error)
  • func (d Display) Default() bool
  • func (d Display) Presentation() bool
  • func (d Display) String() string
• type DisplaySpec
  • func (s DisplaySpec) String() string
  • func (s DisplaySpec) Validate() error
• type ErrorMessage
  • func DecodeError(b []byte) (ErrorMessage, error)
  • func (e ErrorMessage) Error() string
• type Frame
  • func (f Frame) CopyTight(dst []byte) (int, error)
  • func (f Frame) NRGBA() (*image.NRGBA, error)
  • func (f Frame) Row(y int) []byte
  • func (f Frame) TightLen() int
  • func (f Frame) Valid() bool
• type FrameAssembler
  • func NewFrameAssembler(w, h int) (*FrameAssembler, error)
  • func NewFrameAssemblerFor(kind UVCFormat, w, h int) (*FrameAssembler, error)
  • func (a *FrameAssembler) BadJPEG() int
  • func (a *FrameAssembler) Boundaries() (toggles, eofs, shortBytes int)
  • func (a *FrameAssembler) Decode(dst *image.RGBA, frame []byte) (*image.RGBA, error)
  • func (a *FrameAssembler) Feed(p UVCPayload) ([]byte, bool)
  • func (a *FrameAssembler) Format() UVCFormat
  • func (a *FrameAssembler) Losses() (onToggle, onEOF int)
  • func (a *FrameAssembler) MissingMarkers() (noStart, noEnd int)
  • func (a *FrameAssembler) Size() (int, int)
  • func (a *FrameAssembler) Stats() (frames, dropped, bad int)
• type FrameMsg
  • func DecodeFrame(b []byte) (FrameMsg, error)
  • func (f FrameMsg) At() time.Time
• type OpenDisplayMessage
  • func DecodeOpenDisplay(b []byte) (OpenDisplayMessage, error)
• type OpenScreenMessage
  • func DecodeOpenScreen(b []byte) (OpenScreenMessage, error)
• type Options
  • func (o Options) Validate() error
• type OwnedDisplay
  • func (d *OwnedDisplay) Close() error
  • func (d *OwnedDisplay) ID() int
  • func (d *OwnedDisplay) Spec() DisplaySpec
  • func (d *OwnedDisplay) String() string
• type PixelFormat
  • func (f PixelFormat) BytesPerPixel() int
  • func (f PixelFormat) String() string
• type PresentMessage
  • func DecodePresent(b []byte) (PresentMessage, error)
• type PresentedMessage
  • func DecodePresented(b []byte) (PresentedMessage, error)
• type Screen
  • func ShowOn(ctx context.Context, d Display, o ScreenOptions) (*Screen, error)
  • func (s *Screen) Close() error
  • func (s *Screen) DisplayID() int
  • func (s *Screen) Err() error
  • func (s *Screen) Next(ctx context.Context) (Canvas, error)
  • func (s *Screen) Options() ScreenOptions
  • func (s *Screen) Present(ctx context.Context, c Canvas) error
  • func (s *Screen) Size() (int, int)
  • func (s *Screen) Stats() ScreenStats
  • func (s *Screen) Stride() int
  • func (s *Screen) String() string
• type ScreenOptions
  • func (o ScreenOptions) String() string
  • func (o ScreenOptions) Validate() error
• type ScreenStats
  • func (s ScreenStats) String() string
• type Sentinel
  • func (s Sentinel) Describe() string
  • func (s Sentinel) Validate() error
• type StartMessage
  • func DecodeStart(b []byte) (StartMessage, error)
• type Stats
  • func (s Stats) FPS() float64
• type StoppedMessage
  • func DecodeStopped(b []byte) (StoppedMessage, error)
  • func (s StoppedMessage) String() string
• type Stream
  • func CaptureDisplay(ctx context.Context, d Display, o Options) (*Stream, error)
  • func (st *Stream) Close() error
  • func (st *Stream) Err() error
  • func (st *Stream) Format() PixelFormat
  • func (st *Stream) Frame() (Frame, bool)
  • func (st *Stream) Options() Options
  • func (st *Stream) Size() (int, int)
  • func (st *Stream) Stats() Stats
  • func (st *Stream) String() string
  • func (st *Stream) WaitFrame(ctx context.Context) (Frame, error)
• type USBDescriptors
  • func (d USBDescriptors) Device() (vendor, product, class, subclass, protocol int, err error)
  • func (d USBDescriptors) Interfaces() ([]USBInterface, error)
  • func (d USBDescriptors) UVCFormats(iface int) ([]UVCFormatDesc, error)
  • func (d USBDescriptors) Walk(fn func(typ int, b []byte) bool) error
• type USBDevice
  • func DecodeUSBDevices(b []byte) ([]USBDevice, error)
  • func USBDevices(ctx context.Context) ([]USBDevice, error)
  • func (d USBDevice) BulkVideo() bool
  • func (d USBDevice) Readable() []USBInterface
  • func (d USBDevice) String() string
  • func (d USBDevice) VideoStreaming() []USBInterface
  • func (d USBDevice) Viture() bool
• type USBEndpoint
  • func (e USBEndpoint) ISOPacketSize() int
  • func (e USBEndpoint) In() bool
  • func (e USBEndpoint) String() string
  • func (e USBEndpoint) Type() USBTransferType
• type USBHandle
  • func OpenUSB(ctx context.Context, d USBDevice) (*USBHandle, error)
  • func (h *USBHandle) ClaimInterface(n int) error
  • func (h *USBHandle) Close() error
  • func (h *USBHandle) Control(requestType, request uint8, value, index uint16, data []byte) (int, error)
  • func (h *USBHandle) Descriptors() (USBDescriptors, error)
  • func (h *USBHandle) Detached() []int
  • func (h *USBHandle) Name() string
  • func (h *USBHandle) Negotiate(iface int, f UVCFormatDesc, z UVCFrameSize) (UVCNegotiation, error)
  • func (h *USBHandle) SetAltSetting(iface, alt int) error
  • func (h *USBHandle) Stream(neg UVCNegotiation, ep USBEndpoint, w, h2 int) (*UVCStream, error)
  • func (h *USBHandle) SubmitISO(endpoint USBEndpoint) error
• type USBInterface
  • func (i USBInterface) String() string
  • func (i USBInterface) Video() bool
  • func (i USBInterface) VideoStreaming() bool
• type USBTransferType
  • func (t USBTransferType) String() string
• type UVCFormat
  • func (f UVCFormat) String() string
• type UVCFormatDesc
  • func (d UVCFormatDesc) Fastest() (UVCFrameSize, bool)
  • func (d UVCFormatDesc) String() string
• type UVCFrameSize
  • func (s UVCFrameSize) FPS() float64
  • func (s UVCFrameSize) String() string
• type UVCNegotiation
  • func (n UVCNegotiation) Changed() bool
  • func (n UVCNegotiation) FPS() float64
  • func (n UVCNegotiation) String() string
• type UVCPayload
  • func ParsePayload(b []byte) (UVCPayload, error)
  • func (p UVCPayload) Bad() bool
  • func (p UVCPayload) EOF() bool
  • func (p UVCPayload) FID() bool
• type UVCStream
  • func (s *UVCStream) Assembler() *FrameAssembler
  • func (s *UVCStream) Close() error
  • func (s *UVCStream) Frame(ctx context.Context) (*image.RGBA, error)
  • func (s *UVCStream) Negotiation() UVCNegotiation
  • func (s *UVCStream) Stats() UVCStreamStats
• type UVCStreamStats
  • func (s UVCStreamStats) String() string
• type Wall
  • func NewWall(max int) (*Wall, error)
  • func (w *Wall) Close() error
  • func (w *Wall) Len() int
  • func (w *Wall) Max() int
  • func (w *Wall) Open(ctx context.Context, spec DisplaySpec) (*OwnedDisplay, error)
• type Web
  • func (w Web) Describe() string
  • func (w Web) Validate() error

Constants

const (
	// USBClassPerInterface is a device whose class is declared by each
	// interface rather than by the device. Every composite device is one, and a
	// headset that is a camera and something else is composite.
	USBClassPerInterface = 0x00
	// USBClassVideo is the Video class: UVC, which is what a webcam speaks and
	// what a headset's camera is.
	USBClassVideo = 0x0e
)

USB class codes this package names, from the USB-IF class definitions.

const (
	// USBSubclassVideoControl is the interface that describes the camera's
	// units and terminals. A UVC device has exactly one.
	USBSubclassVideoControl = 0x01
	// USBSubclassVideoStreaming is the interface that carries the frames, and
	// the one whose endpoints decide whether Android can read them.
	USBSubclassVideoStreaming = 0x02
)

USB video subclasses, from the UVC specification.

const (
	// MsgDisplays answers MsgListDisplays with everything the host can see.
	MsgDisplays uint8 = 0x01
	// MsgConfig announces a started capture: the frame geometry, the pixel
	// format and the layout of the shared buffer the frames will appear in.
	MsgConfig uint8 = 0x02
	// MsgFrame says a frame has been written into one slot of the shared
	// buffer. It carries no pixels: the pixels are already there.
	MsgFrame uint8 = 0x03
	// MsgStopped says the capture ended, and why. The user revoking the
	// projection from the system UI arrives here, not as an error on a call.
	MsgStopped uint8 = 0x04
	// MsgConsent answers MsgConsentRequest with the user's decision.
	MsgConsent uint8 = 0x05
	// MsgError reports a failure the host could not answer in-band.
	MsgError uint8 = 0x06
	// MsgBuffer hands the application the shared frame buffer, as an ancillary
	// descriptor, once a capture has been configured. It is the ONE message
	// carrying a descriptor, so it is always written in a single sendmsg.
	//
	// The buffer belongs to the HOST, not to the application, and that is
	// forced rather than chosen: an Android app cannot map a descriptor it
	// received read-write. SharedMemory.fromFileDescriptor rejects a Go memfd
	// outright ("FileDescriptor is not a valid ashmem fd"), and reopening it
	// through /proc/self/fd is refused with EACCES. Only a region the host
	// itself created with SharedMemory.create is writable by the host, so the
	// host creates it and lends it here.
	MsgBuffer uint8 = 0x07
	// MsgPresented says one frame the application wrote has reached the view on
	// an existing display, so its slot is free again.
	//
	// It is the ONLY back-pressure [Screen] has. The pixels travel the opposite
	// way there — this process draws and the host blits — so nothing else stops
	// the application overwriting a slot the host has not finished reading, and
	// a tear in a headset is not a cosmetic defect.
	MsgPresented uint8 = 0x08

	// MsgListDisplays asks the host what displays exist.
	MsgListDisplays uint8 = 0x81
	// MsgConsentRequest asks the host to put the screen-capture consent dialog
	// in front of the user. Only the host can: consent is an Activity result,
	// and the application owns no Activity.
	MsgConsentRequest uint8 = 0x82
	// MsgStart asks the host to begin capturing.
	MsgStart uint8 = 0x83
	// MsgStop asks the host to end the capture but stay connected.
	MsgStop uint8 = 0x84
	// MsgBye tells the host the application is going away.
	MsgBye uint8 = 0x85
	// MsgOpenDisplay asks the wall host to create ONE virtual display, show a
	// Presentation carrying the named content on it, and stream the pixels
	// Android renders there. The host answers with MsgConfig — whose DisplayID
	// carries the id the platform assigned — and then MsgBuffer, exactly as
	// MsgStart does, so one Stream implementation serves both.
	MsgOpenDisplay uint8 = 0x86
	// MsgOpenScreen asks the host to show a Presentation on a display that
	// ALREADY EXISTS — the glasses — carrying a view this process paints into.
	//
	// It is the mirror image of MsgOpenDisplay, and the difference is the
	// direction of the pixels, not the kind of window. MsgOpenDisplay exists to
	// get at what Android can render and a CGO-free Go process cannot; this
	// exists because the headset is an OUTPUT, and what belongs on it is the
	// ribbon this process composited. The host answers with MsgConfig and then
	// MsgBuffer, exactly as the other two do.
	MsgOpenScreen uint8 = 0x87
	// MsgPresent says the application has finished drawing one slot and the
	// host should put it on the screen. It carries no pixels: the pixels are
	// already in the shared buffer.
	MsgPresent uint8 = 0x88
)

Message types. Host→app messages are below 0x80 and app→host at or above it, so a misrouted message is a decode error rather than a plausible other message.

const (
	// StopUser is the user revoking the projection — from the cast chip in the
	// status bar, or from the notification the foreground service must show.
	// It is the normal way a capture ends and is NOT a malfunction.
	StopUser uint8 = 0
	// StopSystem is the platform stopping us: a new projection started
	// elsewhere, the device locked, the service was killed.
	StopSystem uint8 = 1
	// StopApp is this application having asked, with MsgStop or MsgBye.
	StopApp uint8 = 2
)

Reasons a capture stopped, carried by StoppedMessage.

const (
	// MsgCameras answers MsgListCameras with every camera the camera2 API can
	// see, including external ones.
	MsgCameras uint8 = 0x09
	// MsgUSBDevices answers MsgListUSBDevices with every device on the USB host
	// port, down to each interface's endpoints.
	MsgUSBDevices uint8 = 0x0a

	// MsgListCameras asks the host what cameras exist.
	MsgListCameras uint8 = 0x89
	// MsgListUSBDevices asks the host what is attached to the USB host port.
	//
	// It reads the device list, which needs no permission: a permission is
	// needed to OPEN a device, not to see that it is there. So a census costs
	// the user no dialog, which is what makes it usable before deciding
	// anything.
	MsgListUSBDevices uint8 = 0x8a
)

CensusMessage types. The census asks what hardware is there, which needs no permission of any kind; see census.go.

const (
	// MsgUSBHandle answers MsgOpenUSBDevice with the device's file descriptor,
	// as an ancillary descriptor. It carries no body: the descriptor is the
	// message.
	//
	// It is the SECOND message in this protocol to ride on SCM_RIGHTS, and for
	// the opposite reason from MsgBuffer: there the host owns memory the
	// application reads, here the host owns a KERNEL HANDLE the application
	// drives with ioctl. Android will not let an ordinary process open
	// /dev/bus/usb at all, so the descriptor can only come from a component
	// that asked the user.
	MsgUSBHandle uint8 = 0x0b

	// MsgOpenUSBDevice asks the host to open one USB device by name, after
	// obtaining the user's permission for it, and to lend the descriptor.
	MsgOpenUSBDevice uint8 = 0x8b
)

USB handover messages. Opening a USB device is the one thing in this package that ASKS THE USER — a system dialog naming the device and the application — so it is the one census question that costs a click.

const (
	// MinScreenEdge is the smallest frame a screen will carry. Smaller than
	// this is a mistake rather than a frugal choice.
	MinScreenEdge = 16

	// MaxScreenPixels bounds one frame, and with it the shared buffer.
	//
	// The WHOLE buffer has to fit an int32, which is what Android's
	// SharedMemory.create takes, and it is MaxQueueDepth slots of four bytes a
	// pixel: 32 megapixels is the largest frame that leaves the product below
	// two gigabytes at every accepted queue depth. It is eight 4K panels' worth,
	// so it constrains nothing anybody would ask for and still refuses a width
	// times a height that overflowed somewhere upstream.
	MaxScreenPixels = 32 << 20
)

Limits on a screen's frames.

const (
	FlagSupportsProtectedBuffers uint32 = 1 << 0
	FlagSecure                   uint32 = 1 << 1
	FlagPrivate                  uint32 = 1 << 2
	FlagPresentation             uint32 = 1 << 3
	FlagRound                    uint32 = 1 << 4
)

Display flags, mirroring android.view.Display's public FLAG_* constants. The two that matter to an XR application are FlagPresentation, which says the system considers a display suitable for an app's own full-screen content, and FlagPrivate, which says it belongs to one app.

const (
	// DefaultFPS is the frame-rate ceiling used when Options.FPS is zero.
	DefaultFPS = 60.0
	// DefaultQueueDepth is the slot count used when Options.QueueDepth is zero.
	DefaultQueueDepth = 3
	// MinQueueDepth is the smallest slot count this package accepts.
	MinQueueDepth = 3
	// MaxQueueDepth bounds the slot count. A slot is a whole framebuffer —
	// 10.4 MB for a 1080x2400 phone — so this is a real memory limit, not a
	// formality.
	MaxQueueDepth = 8
	// MaxDimension is the largest frame edge accepted, a sanity bound well
	// above any real display.
	MaxDimension = 32768
)

Defaults applied to the zero value of the corresponding Options field.

const (
	// MinDescriptorBytes is the smallest a blob can be and still hold a device
	// descriptor. Anything shorter is a short read rather than a device.
	MinDescriptorBytes = 18

	// MaxDescriptorBytes bounds how much is read from a descriptor.
	//
	// ⛔ A READ LOOP WITH NO CEILING IS A READ LOOP. usbfs serves a few hundred
	// bytes, and 64 KiB is far past any real device -- but the thing on the
	// other end is a descriptor this process was handed, and a kernel that kept
	// answering would otherwise be read until memory ran out.
	MaxDescriptorBytes = 64 << 10
)

Bounds on a descriptor blob.

const (
	// DefaultMaxDisplays is the limit [NewWall] applies when it is given 0. A
	// ribbon uses six to eight panels, so this is already generous.
	DefaultMaxDisplays = 8

	// MaxDisplays is the largest limit [NewWall] accepts, however explicitly a
	// caller asks. It is nowhere near the count that killed system_server, and
	// the Java host refuses past it too.
	MaxDisplays = 32

	// DefaultDensityDPI is the density given to an owned display whose spec
	// leaves DensityDPI at zero. It is Android's DENSITY_XHIGH, the density a
	// 1920x1080 panel of the size XR glasses present usually reports.
	DefaultDensityDPI = 320

	// MinDisplayEdge is the smallest edge an owned display may have. A display
	// smaller than this cannot lay a view hierarchy out at all.
	MinDisplayEdge = 16
)

Limits on how many displays one application may create.

const (
	SentinelBackground  uint32 = 0xFF0000 // red
	SentinelTopLeft     uint32 = 0x00FF00 // green
	SentinelBottomRight uint32 = 0x0000FF // blue
)

The colours Sentinel paints, as packed 0xRRGGBB.

const ArtifactEnv = "XRKIT_ARTIFACT_DIR"

ArtifactEnv names the directory artefacts are written to, overriding the default.

const BytesPerPixelYUY2 = 2

BytesPerPixelYUY2 is what YUY2 costs: two bytes a pixel, the first of each pair being the luminance.

const ControlTimeout = 1000

ControlTimeout bounds one control transfer.

⚠ A CONTROL TRANSFER WITH NO TIMEOUT CAN HANG THE CALLER FOREVER. A camera that stops answering mid-negotiation would otherwise take the goroutine with it, and a probe that hangs is worse than one that fails: it reports nothing at all. One second is far longer than a camera takes to answer a nine-byte request.

const DefaultDisplayID = 0

DefaultDisplayID is android.view.Display.DEFAULT_DISPLAY.

const EnvSocket = "XR_ANDROID_SOCKET"

EnvSocket names the environment variable carrying the abstract socket the XR host listens on. A host that spawns this process sets it.

It is not the only route, and in the composition this package is really for it is not the route taken: the process is spawned by go-widgets/android's Activity, which owns the drawing surface, knows nothing about capture and sets no such variable. DeriveSocket covers that case from HOME.

const EnvWallSocket = "XR_ANDROID_WALL_SOCKET"

EnvWallSocket names the environment variable carrying the wall host's socket, the counterpart of EnvSocket.

const ISOPacketsPerURB = 8

ISOPacketsPerURB is how many isochronous packets one request carries.

A UVC camera is scheduled one packet per microframe, so 8 packets is a millisecond of video. It is small deliberately: this package submits ONE request to find out whether the kernel accepts it at all, and a large request that failed for being large would answer a different question.

const MaxFrameBytes = 64 << 20

MaxFrameBytes is the largest frame this package will assemble.

⛔⛔ IT EXISTS BECAUSE THE SIZE COMES FROM THE DEVICE. A frame's dimensions are read out of USB descriptors — two 16-bit fields — so a broken or hostile camera can ask for 65535x65535, and 65535*65535*2 is 8.6 GB that this process would try to allocate twice over (the assembly buffer and the image) before anything else happened. The device is on the other side of a cable and is not trusted with an allocation.

The ceiling is 64 MB — 33,554,432 YUY2 pixels, or 4096x8192 — which is some sixteen times the largest frame any camera here advertises; the Beast's own largest is 1920x1080, or 4 MB.

const MaxPayload = 1 << 16

MaxPayload bounds one decoded message body. The largest message a host legitimately sends is a display list, so a frame beyond this is a desynchronised stream — refused rather than allocated.

const MaxPendingFrames = 4

MaxPendingFrames is how many assembled frames may wait for the caller.

⛔ IT USED TO BE StreamURBs, WHICH WAS A COINCIDENCE RATHER THAN A REASON. This ceiling answers "how stale may a frame be before it is worthless", which is a question about the CONSUMER — head tracking wants the freshest picture — and has nothing to do with how many requests are in flight. Borrowing the ring's depth meant that raising the ring silently made the shedding branch unreachable, which the coverage gate caught the moment the ring went to sixteen.

Four, because a consumer four frames behind at 24 fps is already 170 ms late and a fifth frame would not help it.

const MinPayloadHeader = 2

MinPayloadHeader is the shortest UVC payload header: a length and the bits.

const MinProbeBytes = 26

MinProbeBytes is the shortest probe block any UVC version defines: 26 bytes for 1.0. Anything shorter is not a negotiation block.

const MinTrackingWidth = 320

MinTrackingWidth is the narrowest frame worth asking for.

headflow bins columns [headflow.BinWidth] wide — four — so 320 pixels across is already eighty bins to match a profile on, and its measured reach is bounded in bins rather than in pixels. Below this there is little left to match; above it, more width buys precision a head does not move with.

const ReapTries = 64

SubmitISO queues ONE isochronous request against an endpoint, and reports what the kernel made of it.

⛔ THIS IS THE QUESTION THE WHOLE FILE EXISTS FOR. Android's Java USB API cannot submit an isochronous transfer at all; whether SELinux lets an untrusted application do it through usbfs is undocumented, and every line of a UVC implementation depends on the answer.

It does not reap the result: a submission that the kernel ACCEPTS has already answered the question, and waiting for pixels would confuse "the transfer was allowed" with "the camera was streaming", which are different findings. ReapTries bounds how long USBHandle.SubmitISO waits to collect the request it cancelled.

⚠ IT IS A BOUND, NOT A TIMEOUT TO TUNE. A cancelled isochronous request completes within a frame — a millisecond at high speed — so a handful of tries is already generous. What the bound is really for is the case where it never comes back at all: waiting forever there would hang a probe on a question it has already answered.

const SocketSuffix = ".xr"

SocketSuffix is appended to the application's package name to form the abstract socket the host listens on when nothing names one explicitly. It matches XrHostService.SOCKET_SUFFIX.

const StreamPacketsPerURB = 8

StreamPacketsPerURB is how many isochronous packets one STREAMING request carries, which is a different question and needs a different answer.

⛔⛔ IT IS 8 BECAUSE 32 WAS MEASURED AND WAS WORSE, which is the opposite of what the arithmetic predicted. A 640x480 YUY2 frame at the committed 2400 bytes an interval takes 256 intervals — 32 milliseconds — so a deeper ring looked obviously right. On a Pixel 11 Pro Fold, 2026-10-11, with StreamURBs held at 4:

8 packets   7744 packets/s, 6.24 MB, 10 frames   ← this
32 packets    480 packets/s, 5.4 KB,  0 frames   collapsed

480 packets in fifteen seconds is one request a SECOND where the schedule gives one a millisecond, and the packets that did arrive carried 11 bytes each: the camera stopped sending. Raising StreamURBs to 16 at 8 packets changed nothing either (7968 packets, same 10 frames), so the ring depth is not what bounds this.

⚠ A CONTROL SAYS SO RATHER THAN A STORY: the previous run had left the interface detached, so two things differed at once. Putting 8 back with that same device state streamed again immediately — which is what makes the ring the cause and not the phone.

The frame-rate deficit that remains has a different cause, localised by the per-boundary census in FrameAssembler.Losses: every dropped frame ends on a FID toggle and almost none on an EOF, so the camera is sending same-identifier packets AFTER its end-of-frame bit. That is still open.

const StreamURBs = 16

StreamURBs is how many isochronous requests are kept in flight.

⛔ ONE IS NOT ENOUGH, AND THAT IS NOT A TUNING CHOICE. An isochronous endpoint is scheduled every microframe whether anybody is listening: between reaping one request and submitting the next, the camera's packets go nowhere.

⛔⛔ AND SIXTEEN IS MEASURED, AGAINST FOUR WHICH HALVED THE FRAME RATE. This was 4 for five releases on the strength of an experiment with a CHEAP consumer where deepening the ring changed nothing — an experiment that was right about its own conditions and wrong as a general conclusion. With a consumer costing more than the ring covers, the ring is dry while it works and the packets that arrive then are gone. Measured on a Pixel 11 Pro Fold with cmd/xrhead, which spends 4.8 ms a frame decoding and tracking:

requests   fps     dropped
       4   10.5    79
       8   21.5    14
      16   23.8     0      <- this
      32    0       the camera stops sending

23.8 fps against the 25 the camera advertises, and nothing dropped. Thirty-two fails the same way thirty-two PACKETS a request does; both axes have a ceiling of their own and neither is their product, since 4x32 and 16x8 are both 128 packets and only one of them works.

⛔ WHAT THIS CORRECTS IS A PUBLISHED CONCLUSION. This repository said "~10 fps is the camera's ceiling, flat across a fourfold change in pixel count", and the byte budget behind it said the camera filled 34.7 % of each interval. The budget closed; the causal reading was backwards. The camera under-filled because NOTHING WAS ASKING — at sixteen requests the same camera fills 77 %.

const VitureVendorID = 0x35ca

VitureVendorID is VITURE's USB vendor id, recorded in go-xrkit/xrkit/glasses: every VITURE headset reports it.

const WallSocketSuffix = ".xrwall"

WallSocketSuffix is appended to the package name to get the abstract socket the WALL host listens on, matching XrWallService.SOCKET_SUFFIX.

It is a second socket, and a second service, because the two have nothing in common but a wire format. Capture needs a MediaProjection, a consent dialog and — from API 34 — a mediaProjection foreground service, none of which an owned display needs: it asks for no permission at all. Keeping them apart means creating a ribbon panel cannot put a "recording your screen" chip in the status bar, and that a capture session ending cannot take a panel with it. The capture host also serves exactly one connection, whereas the wall serves one PER DISPLAY.

const ZeroBandwidthAlternate = 0

ZeroBandwidthAlternate is alternate setting 0 of a UVC streaming interface.

⛔ IT IS WHERE A STREAMING INTERFACE MUST SIT WHILE IT IS NEGOTIATED. Alternate 0 of a video streaming interface carries no isochronous endpoint at all — that is what "zero bandwidth" means — and the host parks there when not streaming so the bus's isochronous budget is not held. UVC then treats the move AWAY from it, after a commit, as the signal to start sending.

Variables

var (
	// ErrUnsupported is reported on every platform that is not Android, and on
	// Android when the process was not started by a host that speaks this
	// protocol. The module still builds and vets everywhere, so an application
	// keeps cross-building on a developer's workstation.
	ErrUnsupported = errors.New("android: screen capture is unavailable " +
		"(not running under a go-xrkit Android host)")

	// ErrPermissionDenied is reported when the user refused the screen-capture
	// consent dialog, or revoked a projection already running. Its message
	// names the remedy.
	ErrPermissionDenied = errors.New("android: screen-capture consent denied — " +
		"Android asks the user for permission once per capture session and the " +
		"answer cannot be remembered; call RequestAuthorization again and accept " +
		"the system dialog")

	// ErrNoDisplay is reported when a capture was asked for and the host
	// listed no display at all.
	ErrNoDisplay = errors.New("android: no capturable display")

	// ErrNotFound is reported when a display ID does not name anything the
	// host can see.
	ErrNotFound = errors.New("android: no such display")

	// ErrNotCapturable is reported for a display that exists but that an
	// unprivileged app may not capture. MediaProjection mirrors the DEFAULT
	// display and nothing else: capturing a second display needs
	// CAPTURE_VIDEO_OUTPUT, which is a signature permission.
	ErrNotCapturable = errors.New("android: only the default display can be captured " +
		"by an unprivileged app (capturing another display needs the signature " +
		"permission CAPTURE_VIDEO_OUTPUT)")

	// ErrClosed is reported by every [Stream] method after [Stream.Close].
	ErrClosed = errors.New("android: stream is closed")

	// ErrNoFrame is reported by [Stream.WaitFrame] when no frame arrived
	// before its context expired. It is NOT a malfunction: a motionless screen
	// legitimately produces no frames.
	ErrNoFrame = errors.New("android: no frame available")

	// ErrInvalidOption is reported by [Options.Validate] and wraps a
	// description of the offending field.
	ErrInvalidOption = errors.New("android: invalid option")

	// ErrShortBuffer is reported by [Frame.CopyTight] when the destination is
	// too small to hold the frame.
	ErrShortBuffer = errors.New("android: destination buffer too short")
)

Sentinel errors. All are stable and may be matched with errors.Is.

var ErrBadPayload = errors.New("android: malformed message payload")

ErrBadPayload reports a message whose body is structurally impossible — a string longer than the body that holds it, a negative count.

var ErrInRepository = errors.New("a capture must never be written where it can be committed")

ErrInRepository is reported when the chosen artefact directory is inside a git work tree.

var ErrNotPresentable = errors.New("android: display will not take a presentation")

ErrNotPresentable reports a display that cannot carry a Presentation.

The built-in panel is the usual one: it is where the launcher and every other application live, and the platform will not let an ordinary app put a Presentation over them. Display.Presentation says in advance, which is why this is answered here without asking the host.

var ErrScreenClosed = errors.New("android: screen is closed")

ErrScreenClosed is reported by Screen.Next and Screen.Present after Screen.Close.

var ErrShortPayload = errors.New("android: truncated message payload")

ErrShortPayload reports a message whose body is too short for its type.

var ErrTooManyDisplays = errors.New("android: too many virtual displays")

ErrTooManyDisplays is reported by NewWall when the requested limit exceeds MaxDisplays, and by Wall.Open when the wall is already at its limit. Nothing is created in either case.

var ErrUSBPermissionDenied = errors.New("android: permission to open the USB device was refused")

ErrUSBPermissionDenied reports the user declining the system's USB permission dialog, or an application asking for a device it was never granted.

It is a decision rather than a failure: opening a USB device shows a dialog naming the device and the application, and a person may say no.

A device that is no longer attached reports ErrNotFound instead, which this package already had: a headset on the end of a cable is unplugged more often than anything else here, and a second sentinel for it would be two names for one condition.

var ErrWallClosed = errors.New("android: wall is closed")

ErrWallClosed is reported by Wall.Open after Wall.Close.

Functions

func ArtifactDir

func ArtifactDir() (string, error)

ArtifactDir is where a command may write what it captured, created if it is not there. The rule above is what it enforces.

func Authorized

func Authorized() bool

Authorized reports whether the host already holds a projection token, so a capture can start without a dialog. It never prompts.

Unlike a macOS TCC grant this is NOT a lasting permission: Android issues a token per session and forgets it, so this answers "right now", not "ever".

func Available

func Available() bool

Available reports whether this process is running under a host that speaks this protocol. It consults the environment only: it opens nothing and blocks on nothing, so it is safe to call before deciding whether Android capture is this program's platform at all.

func ControlIn

func ControlIn() uint8

ControlIn builds the bmRequestType of a class request that brings data back from one interface, which is every UVC GET.

func ControlOut

func ControlOut() uint8

ControlOut builds the bmRequestType of a class request that carries data to one interface, which is every UVC SET.

func DecodeBuffer

func DecodeBuffer(b []byte) (slots int, slotSize int64, err error)

DecodeBuffer decodes the layout of the shared buffer.

func DecodeConsent

func DecodeConsent(b []byte) (bool, error)

DecodeConsent decodes the user's decision.

func DecodeOpenUSBDevice

func DecodeOpenUSBDevice(b []byte) (string, error)

DecodeOpenUSBDevice decodes it.

func DeriveSocket

func DeriveSocket(env string, envOK bool, home string) string

DeriveSocket works out which abstract socket to dial.

An explicit name wins: a host that spawned this process sets EnvSocket and that is the end of it. But in the composition this package is really for, the process is spawned by a DIFFERENT host — go-widgets/android's Activity, which owns the drawing surface and knows nothing about capture — and never sets it. Android gives such a process no way to ask its own package name either.

What it does give it is HOME, which that host sets to the app's private files directory: "/data/user/0/org.example.app/files". The package name is in there, and the host's socket name follows from it. That is why the two hosts can live in one APK without either one having to know about the other.

It returns "" when neither route yields a name.

func DeriveWallSocket

func DeriveWallSocket(env string, envOK bool, home string) string

DeriveWallSocket works out which abstract socket the wall host listens on, exactly as DeriveSocket does for the capture host.

func EncodeBuffer

func EncodeBuffer(slots int, slotSize int64) []byte

EncodeBuffer encodes the layout of the shared buffer accompanying MsgBuffer.

func EncodeCameras

func EncodeCameras(cs []Camera) []byte

EncodeCameras encodes a camera list.

func EncodeConfig

func EncodeConfig(c ConfigMessage) []byte

EncodeConfig encodes a ConfigMessage.

func EncodeConsent

func EncodeConsent(granted bool) []byte

EncodeConsent encodes the user's decision.

func EncodeDisplays

func EncodeDisplays(ds []Display) []byte

EncodeDisplays encodes a display list.

func EncodeError

func EncodeError(e ErrorMessage) []byte

EncodeError encodes an ErrorMessage.

func EncodeFrame

func EncodeFrame(f FrameMsg) []byte

EncodeFrame encodes a FrameMsg.

func EncodeOpenDisplay

func EncodeOpenDisplay(m OpenDisplayMessage) []byte

EncodeOpenDisplay encodes an OpenDisplayMessage.

func EncodeOpenScreen

func EncodeOpenScreen(m OpenScreenMessage) []byte

EncodeOpenScreen encodes an OpenScreenMessage.

func EncodeOpenUSBDevice

func EncodeOpenUSBDevice(name string) []byte

EncodeOpenUSBDevice encodes the name of the device to open — the node name from USBDevice.Name, which is what a USB permission is granted against.

func EncodePresent

func EncodePresent(m PresentMessage) []byte

EncodePresent encodes a PresentMessage.

func EncodePresented

func EncodePresented(m PresentedMessage) []byte

EncodePresented encodes a PresentedMessage.

func EncodeStart

func EncodeStart(s StartMessage) []byte

EncodeStart encodes a StartMessage.

func EncodeStopped

func EncodeStopped(s StoppedMessage) []byte

EncodeStopped encodes a StoppedMessage.

func EncodeUSBDevices

func EncodeUSBDevices(ds []USBDevice) []byte

EncodeUSBDevices encodes a USB device list, down to each endpoint.

func FrameMessage

func FrameMessage(typ uint8, body []byte) []byte

FrameMessage returns one framed message: a 4-byte big-endian length covering the type byte and the body, then the type byte, then the body.

It exists as bytes rather than as writes because a message that carries an ancillary descriptor has to reach the host in ONE sendmsg: split across two writes, the host could attribute the descriptor to the wrong message.

func PickAlternate

func PickAlternate(ifaces []USBInterface, number int, payload uint32) (USBInterface, USBEndpoint, bool)

PickAlternate chooses the alternate setting of one interface that can carry a committed payload, and that alternate's isochronous IN endpoint.

⛔⛔ THE SMALLEST THAT FITS, AND AFTER THE COMMIT. Not the largest: an alternate's endpoint reserves bus bandwidth for as long as it is selected, so taking 3072 bytes an interval to carry 2400 holds part of every high-speed microframe away from everything else on the bus for nothing. And not before the commit, because the payload is not known until the camera has answered.

func PickFormat

func PickFormat(fs []UVCFormatDesc) (UVCFormatDesc, UVCFrameSize, error)

PickFormat chooses what to ask a camera for: the fastest frame in a format this package can actually read.

⭐ YUY2 FIRST, AND THE REASON IS WHAT IS IMPLEMENTED. Head tracking reads luminance only, and in YUY2 every other byte IS the luminance — no decoder, no colour conversion, nothing allocated per frame. MJPEG on the camera measured here is FASTER, 30 frames a second at every size against YUY2's 25, and it is the better choice the day there is a JPEG decode in the frame path. There is not, so preferring YUY2 reflects the code rather than the hardware, and saying so is the point.

⚠ A format this package cannot name is skipped rather than attempted. A camera offering only those is a different finding from one offering none, and the error says how many there were.

func ReadMessage

func ReadMessage(r io.Reader) (typ uint8, body []byte, err error)

ReadMessage reads one framed message. It returns io.EOF when the stream ends cleanly between messages, so a caller can tell a closed host from a truncated one.

func RequestAuthorization

func RequestAuthorization(ctx context.Context) (bool, error)

RequestAuthorization asks the host to put Android's screen-capture consent dialog in front of the user, and blocks until they answer or ctx expires.

It is a separate call rather than something CaptureDisplay does implicitly because the dialog takes over the screen: an application wants to choose when that happens, and to say why first.

func SaveFrame

func SaveFrame(name string, img *image.RGBA) (string, error)

SaveFrame writes one captured frame beside the transcript, as a PNG, and returns the path it used.

⛔⛔ A FRAME IS A PICTURE OF WHATEVER THE CAMERA WAS POINTED AT, which on a headset is a room and the people in it. It goes exactly where a transcript goes and under exactly the rule above — never inside a work tree, including one somebody points ArtifactEnv at. That is also why there is no option to put it anywhere else.

⭐ PNG, AND NOT A FORMAT OF OUR OWN. These frames exist to be fed to go-xrkit/xrkit/headflow, which lives in another module with a dependency tree this one has no business acquiring — HCL and eight indirect modules, for a diagnostic binary. A FILE is the interface between them, `image/png` is in the standard library on both sides, so the two halves share no code at all; and a person can also just look at the picture.

⚠ GREY, BECAUSE GREY IS WHAT IS READ. headflow.Profile sums R+G+B down each column, and every decoder here already writes the luminance into all three channels, so a grey PNG loses nothing a tracker uses and is a quarter of the size.

func SaveTranscript

func SaveTranscript(name, text string) (string, error)

SaveTranscript writes a command's transcript where it can still be read after the cable has been somewhere else, and returns the path it used.

⛔ THE APP'S OWN files DIRECTORY, NOT ITS EXTERNAL ONE. A file in /storage/emulated/0/Android/data/<pkg>/ is hidden from the shell user on a modern Android, and `adb shell run-as` cannot traverse the FUSE layer that mediates it: a whole run was measured, written there, and unreadable. The private directory -- which the host hands over as HOME -- is what run-as can read, and this is where that lands.

Off a device there is no HOME of that kind, so it falls back to ArtifactDir. A transcript is text this process composed rather than a picture of anybody's screen, so the rule about work trees does not govern it -- but the fallback obeys it anyway, because the cheapest way to keep a rule is to have one path.

func SentinelColorAt

func SentinelColorAt(w, h, x, y int) (uint32, bool)

SentinelColorAt is the colour a correct Sentinel frame holds at (x, y) on a display of the given size, as 0xRRGGBB. It reports false for a point outside the display, or one close enough to a quadrant edge or to the label that anti-aliasing makes the exact value unsafe to assert on.

Sample with it rather than comparing whole images: the label is rendered by Android's own text stack and will not be byte-identical between releases, while the quadrants are flat colour and are.

func WallAvailable

func WallAvailable() bool

WallAvailable reports whether this process is running under a host that serves owned displays. It dials nothing.

func WriteMessage

func WriteMessage(w io.Writer, typ uint8, body []byte) error

WriteMessage writes one framed message.

Types

type Camera

type Camera struct {
	// ID is the camera2 id, which is what opens it. It is a small number as a
	// string for built-in cameras and is not necessarily one for an external.
	ID string
	// Facing is which way it points; see [FacingExternal].
	Facing CameraFacing
	// Width and Height are the largest still size it advertises, in pixels.
	// Zero when the host could not read its characteristics.
	Width, Height int
}

Camera is one camera the camera2 API can see.

func Cameras

func Cameras(ctx context.Context) ([]Camera, error)

Cameras returns every camera the camera2 API can see, including external USB ones.

⛔ IT ASKS THE WALL HOST, like Screens, and for the same reason: listing cameras needs NO permission at all. android.permission.CAMERA is required to OPEN one, not to be told it is there, so a census costs the user no dialog — which is exactly what makes it usable before deciding anything.

An empty list is an answer: it says this phone's HAL offers no external camera, which is the thing that decides whether a headset's camera can be reached through the platform at all. See ChooseRoute.

func DecodeCameras

func DecodeCameras(b []byte) ([]Camera, error)

DecodeCameras decodes a camera list.

func (Camera) External

func (c Camera) External() bool

External reports whether this camera is one that can be unplugged — the only kind a headset's can be.

func (Camera) String

func (c Camera) String() string

String renders the camera for logs.

type CameraFacing

type CameraFacing int

CameraFacing says which way a camera points, as android.hardware.camera2's LENS_FACING.

const (
	// FacingFront is the selfie camera.
	FacingFront CameraFacing = 0
	// FacingBack is the main camera.
	FacingBack CameraFacing = 1
	// FacingExternal is a camera that is not fixed to the device and may come
	// and go — a USB camera, which is what a headset's is.
	//
	// ⭐ THIS IS THE ONE THAT MATTERS HERE, and the one a phone may simply not
	// offer: supporting external cameras is left to the vendor's HAL.
	FacingExternal CameraFacing = 2
)

The facings camera2 defines.

func (CameraFacing) String

func (f CameraFacing) String() string

String renders the facing.

type CameraRoute

type CameraRoute int

CameraRoute says how an application could reach a headset's camera on this phone, which is a measurement rather than a choice; see the file comment.

const (
	// RouteNone is no route at all: the phone offers no external camera and no
	// USB device that could be read.
	RouteNone CameraRoute = iota
	// RouteCamera2 is the camera2 API with an external camera, which is the
	// route to want: the platform does the UVC negotiation, the format
	// conversion and the buffering.
	RouteCamera2
	// RouteUSBBulk is claiming the UVC interface over the USB host API. It
	// needs the streaming endpoints to be bulk, and it means writing the UVC
	// negotiation by hand.
	RouteUSBBulk
	// RouteUSBIsochronous is a UVC device whose frames are isochronous. It is
	// listed because it is what a census will usually find, and NOT because it
	// is usable through Android's Java USB API — reaching it means usbfs ioctls
	// on the descriptor that API hands out.
	RouteUSBIsochronous
)

The routes, in the order they are preferred.

func ChooseRoute

func ChooseRoute(cams []Camera, devs []USBDevice) CameraRoute

ChooseRoute says how a headset's camera could be reached, given what the census found.

It prefers camera2 whenever an external camera is there, because the platform then does the UVC negotiation, the format conversion and the buffering that the USB route would mean writing by hand.

⚠ IT ANSWERS FROM THE CENSUS, NOT FROM THE HEADSET. A phone with some other USB camera attached would be reported as reachable, and it would be — just not the headset's. USBDevice.Viture is how a caller narrows it, and cmd/xrcam prints the device it is talking about alongside the verdict.

func (CameraRoute) String

func (r CameraRoute) String() string

String renders the route.

func (CameraRoute) Usable

func (r CameraRoute) Usable() bool

Usable reports whether this route can be taken with Android's own APIs and nothing else.

type Canvas

type Canvas struct {
	// Pix is the frame's pixels, Stride bytes a row, Height rows.
	Pix []byte
	// Stride is the distance in bytes between two rows. It is at least
	// Width*4 and is usually more: the host announces the alignment its
	// surface wants, and writing Width*4 instead would shear the image.
	Stride int
	// Width and Height are the frame's size in pixels.
	Width, Height int
	// contains filtered or unexported fields
}

A Canvas is one frame slot of a screen's shared buffer, lent to the application to draw into.

The bytes are BORROWED and belong to the host: they are valid from the Screen.Next that returned them until the Screen.Present that hands them back, and touching them afterwards races the blit. A Canvas is not safe for concurrent use.

Pix is RGBA, eight bits a channel — the same memory layout as image.RGBA, which is what Canvas.RGBA hands back so the whole of image/draw can paint a frame.

func (Canvas) RGBA

func (c Canvas) RGBA() *image.RGBA

RGBA returns the canvas as an image.RGBA sharing the same memory, so a frame can be painted with image/draw, a font renderer, or anything else that takes a draw.Image.

It allocates only the header: the pixels are the host's, and drawing through this writes straight into the shared buffer.

func (Canvas) Row

func (c Canvas) Row(y int) []byte

Row returns the bytes of one row, for a compositor that writes spans rather than going through image/draw. It reports nil for a row outside the frame.

The slice stops at Width*4 and cannot be appended past the row: the padding a stride leaves is the host's, and a span that ran into it would corrupt the next row rather than fail.

func (Canvas) String

func (c Canvas) String() string

String renders the canvas for logs.

type ConfigMessage

type ConfigMessage struct {
	// Width and Height are the frame size in pixels.
	Width, Height int
	// Stride is the number of BYTES per row in the shared buffer. It is the
	// host's ImageReader row stride and is NOT necessarily Width*4.
	Stride int
	// Format is the pixel layout, always [FormatRGBA] today.
	Format PixelFormat
	// Slots is how many frame slots the shared buffer holds.
	Slots int
	// SlotSize is the byte size of one slot, Stride*Height rounded up to the
	// host's alignment.
	SlotSize int64
	// DisplayID names the display being captured.
	DisplayID int
}

ConfigMessage announces a started capture.

func DecodeConfig

func DecodeConfig(b []byte) (ConfigMessage, error)

DecodeConfig decodes a ConfigMessage.

type Content

type Content interface {
	// Describe renders the content for logs.
	Describe() string
	// Validate reports whether the content is usable, wrapping
	// [ErrInvalidOption].
	Validate() error
	// contains filtered or unexported methods
}

Content is what Android renders on an owned display.

The seam, and why it is a DECLARATION rather than pixels

The obvious seam would hand the host a framebuffer for it to blit. That is the wrong way round here, and would be a round trip with no product: pixels this process drew, sent to Android, read back unchanged. If the application is drawing, it should composite in Go and skip the display entirely.

The reason to route a ribbon panel through a real Android display is the opposite one — to get at what Android renders and this process CANNOT: a WebView, a MediaCodec surface, a PdfRenderer, a maps view. None of those can be reached from a CGO-free Go process, all of them are ordinary Views, and a Presentation is how a View gets onto a display. So Content NAMES what should render, the Java host builds it, and the pixels come back through the same borrowed-frame path as a screen capture.

That also keeps the rule the rest of this package is built on: the host owns the Android objects and decides nothing. A URL is a decision the application made; a WebView is the host obeying it.

The set is deliberately closed — a Content implemented outside this package would name a Java class the host does not have. Implementations are Web and Sentinel.

type Display

type Display struct {
	// ID is the android.view.Display display id. 0 is the built-in screen.
	ID int
	// Name is what the system calls the display. For the built-in panel this
	// is a device string ("Built-in Screen" on the Android 15 emulator); for a
	// display attached over USB-C DP Alt Mode it comes from the sink's EDID,
	// which is how XR glasses announce their model.
	Name string
	// Width and Height are the display's real size in pixels.
	Width, Height int
	// DensityDPI is the display's density in dots per inch.
	DensityDPI int
	// RefreshRate is the active mode's rate in hertz.
	RefreshRate float64
	// Flags are android.view.Display's FLAG_* bits; see [FlagPresentation].
	Flags uint32
}

Display is a display the host can see.

Width and Height are in PIXELS — Android reports no separate point space, so unlike the macOS sibling there is only one pair of numbers, and DensityDPI carries what a scale factor would carry elsewhere.

func DecodeDisplays

func DecodeDisplays(b []byte) ([]Display, error)

DecodeDisplays decodes a display list.

func DefaultDisplay

func DefaultDisplay(ctx context.Context) (Display, error)

DefaultDisplay returns the built-in screen, the only display an unprivileged app may capture.

func Displays

func Displays(ctx context.Context) ([]Display, error)

Displays returns every display the host can see.

Only the default one can be captured (see ErrNotCapturable), but the others matter all the same: a headset attached over USB-C DP Alt Mode appears here, and its Display.Name is how a catalogue identifies the model.

func Screens

func Screens(ctx context.Context) ([]Display, error)

Screens returns the displays an application may put a Screen on: the ones the platform will take a Presentation on, minus the built-in panel.

⛔ IT ASKS THE WALL HOST, NOT THE CAPTURE HOST, and that is the difference between needing a MediaProjection and needing nothing. Displays goes to the capture host because that is where a capture's display list belongs; an application that only wants to paint on the glasses would have had to start the whole mediaProjection apparatus — a foreground service, a consent dialog, a "recording your screen" chip — to find out that a pair of glasses is plugged in. This asks the service that already owns Presentations.

The connection is dialled and dropped: there is no session to keep, and a display list is a question with an answer rather than a stream.

func (Display) Default

func (d Display) Default() bool

Default reports whether this is the built-in screen, the only display an unprivileged app may capture.

func (Display) Presentation

func (d Display) Presentation() bool

Presentation reports whether the system considers this display suitable for an application's own full-screen content — which is what an XR headset attached over DP Alt Mode is for.

func (Display) String

func (d Display) String() string

String renders the display for logs.

type DisplaySpec

type DisplaySpec struct {
	// Width and Height are the display's size in PIXELS. Both are required.
	//
	// Making them small does NOT buy more displays — the count that kills
	// system_server was the same at 640x480 and at 1920x1080 — so choose the
	// size the content wants to be read at.
	Width, Height int

	// DensityDPI is the display's density. Zero means [DefaultDensityDPI].
	// Android lays the view hierarchy out against this, so it decides how
	// large text and controls come out in the ribbon.
	DensityDPI int

	// Content is what Android renders on the display. It is required; see
	// [Content].
	Content Content

	// QueueDepth is how many frame slots the shared buffer holds, exactly as
	// in [Options.QueueDepth]. Zero means [DefaultQueueDepth].
	QueueDepth int
}

DisplaySpec says what an owned display should be.

The zero DisplaySpec is not usable: a display has no natural size to fall back on the way a capture does, so Width, Height and Content must be given.

func (DisplaySpec) String

func (s DisplaySpec) String() string

String renders the spec for logs.

func (DisplaySpec) Validate

func (s DisplaySpec) Validate() error

Validate reports whether the spec describes a display Android could make, wrapping ErrInvalidOption. It does not consult the system, and it does not apply defaults: a zero DensityDPI or QueueDepth is refused here and filled in by Wall.Open before this is reached.

type ErrorMessage

type ErrorMessage struct {
	// Code is the host's own numeric code, 0 when it has none.
	Code int
	// Op names what failed, e.g. "getMediaProjection".
	Op string
	// Detail is the platform's message, e.g. a SecurityException's text.
	Detail string
}

ErrorMessage is a failure the host could not answer in-band.

func DecodeError

func DecodeError(b []byte) (ErrorMessage, error)

DecodeError decodes an ErrorMessage.

func (ErrorMessage) Error

func (e ErrorMessage) Error() string

Error renders the operation, the code and the platform's message.

type Frame

type Frame struct {
	// Pix is the frame's bytes in [FormatRGBA], Stride bytes per row, Height
	// rows. len(Pix) == Stride*Height.
	Pix []byte
	// Width and Height are the frame's size in pixels.
	Width, Height int
	// Stride is the number of BYTES per row. It may be padded and is NOT
	// necessarily Width*4.
	Stride int
	// Seq counts frames since the stream started; it is 0 before the first
	// frame and strictly increases afterwards.
	Seq uint64
	// At is when the host's ImageReader delivered the frame.
	At time.Time
}

Frame is a BORROWED view of one captured frame.

Pix aliases the shared mapping the host wrote into. It stays valid only until the next Stream.Frame, Stream.WaitFrame or Stream.Close on the stream that produced it. Do not retain it; copy with Frame.CopyTight or Frame.NRGBA if you need it to outlive the borrow.

func (Frame) CopyTight

func (f Frame) CopyTight(dst []byte) (int, error)

CopyTight copies the frame into dst with the row padding removed, so dst holds Width*4*Height bytes of contiguous RGBA. It reports how many bytes it wrote, or ErrShortBuffer if dst is too small. It allocates nothing.

func (Frame) NRGBA

func (f Frame) NRGBA() (*image.NRGBA, error)

NRGBA copies the frame into a freshly allocated image.NRGBA. It is the convenience path for saving a frame to disk; it allocates, so it does not belong in a per-frame loop.

Android's RGBA_8888 is already image.NRGBA's byte order, so unlike the macOS sibling this is a copy with no channel swap — but it is still a copy, because the source is padded and borrowed.

func (Frame) Row

func (f Frame) Row(y int) []byte

Row returns row y of the frame, Width*4 bytes with the padding trimmed off. It does not allocate. It returns nil for an out-of-range y or an invalid frame.

func (Frame) TightLen

func (f Frame) TightLen() int

TightLen is the number of bytes the frame occupies with no row padding, Width*4*Height.

func (Frame) Valid

func (f Frame) Valid() bool

Valid reports whether the frame holds pixels.

type FrameAssembler

type FrameAssembler struct {
	// contains filtered or unexported fields
}

A FrameAssembler turns a stream of payloads into whole frames.

⭐ IT ENDS A FRAME ON THE FID TOGGLE, not on the end-of-frame bit. EOF is optional in practice — cameras that never set it exist — and a boundary that depended on it would run every frame into the next. EOF is still honoured when it comes, because it ends the frame one payload earlier and saves carrying a frame until the next one starts.

⛔ AND A SHORT FRAME IS DROPPED RATHER THAN PADDED. A stream that lost a packet leaves a frame missing a band; padding it with zeros produces a black stripe that a tracker reads as structure, which is worse than no frame at all. [FrameAssembler.Dropped] counts them.

func NewFrameAssembler

func NewFrameAssembler(w, h int) (*FrameAssembler, error)

NewFrameAssembler returns an assembler for uncompressed YUY2 frames of the given size. See NewFrameAssemblerFor for the compressed ones.

func NewFrameAssemblerFor

func NewFrameAssemblerFor(kind UVCFormat, w, h int) (*FrameAssembler, error)

NewFrameAssemblerFor returns an assembler for frames of the given format and size.

⛔⛔ THE FORMAT CHANGES WHAT "A WHOLE FRAME" MEANS, which is the whole reason this is not one code path with a flag. A YUY2 frame is whole when it is w*h*2 bytes: the LENGTH is the test, and a short one is a frame that lost packets. An MJPEG frame is a JPEG, so its length is whatever the compressor made it and the length says nothing at all — a frame that lost packets is the same length as a frame of a plainer scene. What tells them apart is the MARKERS, and nothing else does.

Reaching for MJPEG is measured rather than preferred: the Beast's camera advertises 30 fps at every size for MJPEG and 25 fps only at 640x480 and below for YUY2, and the byte budget of a measured run says why — it fills 832 of the 2400 bytes an interval it committed to, which is 10.5 uncompressed frames a second whatever is asked of it.

func (*FrameAssembler) BadJPEG

func (a *FrameAssembler) BadJPEG() int

BadJPEG is how many MJPEG frames arrived whole — both markers present — and still would not decode, or decoded to a size other than the one committed.

⛔ IT IS NOT PART OF Stats BECAUSE IT IS A DIFFERENT FAULT. A dropped frame never arrived; one of these did, and is still not a picture, which points at the camera's compressor or at a corrupted middle rather than at a lost tail.

func (*FrameAssembler) Boundaries

func (a *FrameAssembler) Boundaries() (toggles, eofs, shortBytes int)

Boundaries reports how each frame ended and how much the dropped ones were missing: identifier toggles, end-of-frame bits, and the total shortfall in bytes.

⭐ IT IS A CENSUS PER CHANNEL. "frames" and "dropped" cannot say whether a camera split its frames or a bus lost packets — the first shows up as more boundaries than frames, the second as a shortfall spread thinly. Averaging the shortfall over the dropped count says which.

func (*FrameAssembler) Decode

func (a *FrameAssembler) Decode(dst *image.RGBA, frame []byte) (*image.RGBA, error)

Decode turns one assembled YUY2 frame into an RGBA image.

⭐ IT WRITES LUMINANCE INTO ALL THREE CHANNELS, which is not a shortcut. The consumer is go-xrkit/xrkit/headflow, whose Profile sums R+G+B down each column: a grey image carries exactly what it reads, and a full YUV-to-RGB conversion would compute chrominance for it to add up and discard. Every other byte of YUY2 IS the luminance, so this is one pass and no arithmetic.

dst is reused when it is already the right size, so a steady stream allocates nothing.

func (*FrameAssembler) Feed

func (a *FrameAssembler) Feed(p UVCPayload) ([]byte, bool)

Feed takes one payload and returns a whole frame's bytes when one completed.

⛔⛔ THE BYTES ARE THE CALLER'S, AND THIS COMMENT USED TO SAY THE OPPOSITE. It claimed they were borrowed until the next Feed; [FrameAssembler.complete] hands the buffer over and allocates a fresh one, so a completed frame stays valid for as long as the caller keeps it. The code was the safer of the two and the promise was the wrong one — which matters, because that ownership is exactly what makes UVCStream's queue of pending frames legal. A reader who trusted the old sentence would have read that queue as a use-after-free.

⚠ It costs one allocation a frame, measured at 611 KB and under 0.15 ms at 640x480 — two orders of magnitude below the frame interval, which is why it is spent rather than pooled. A free list would buy nothing here and would put an ownership bug where there is now none.

func (*FrameAssembler) Format

func (a *FrameAssembler) Format() UVCFormat

Format is the format this assembler was built for.

func (*FrameAssembler) Losses

func (a *FrameAssembler) Losses() (onToggle, onEOF int)

Losses reports which boundary ended the frames that were dropped.

⛔ THE SPLIT IS THE DIAGNOSIS. Equal numbers of toggles and EOFs with half the frames nearly empty fits two opposite stories — a camera setting EOF early, or one toggling early — and they call for opposite fixes.

func (*FrameAssembler) MissingMarkers

func (a *FrameAssembler) MissingMarkers() (noStart, noEnd int)

MissingMarkers splits the MJPEG frames that were dropped by WHICH marker they lacked: one that began mid-picture, and one that lost its tail.

⛔ THEY ARE OPPOSITE FAULTS. No start-of-image means a boundary fired where the camera had not finished a picture; a start with no end means the tail went missing. "Dropped" cannot tell them apart, and they call for different answers — the same reason FrameAssembler.Losses splits by boundary.

func (*FrameAssembler) Size

func (a *FrameAssembler) Size() (int, int)

Size is the frame size this assembler was built for.

func (*FrameAssembler) Stats

func (a *FrameAssembler) Stats() (frames, dropped, bad int)

Stats reports how many whole frames came out, how many were dropped for being short, and how many payloads the camera itself marked bad.

type FrameMsg

type FrameMsg struct {
	// Seq counts frames since the capture started, from 1.
	Seq uint64
	// Slot is which slot of the shared buffer holds it.
	Slot int
	// Width, Height and Stride describe THIS frame. They normally repeat the
	// config, but the host may resize a capture in flight — a phone rotating
	// under a capture does exactly that — so every frame carries its own
	// geometry rather than inheriting one that may be stale.
	Width, Height, Stride int
	// AtUnixNano is when the host's ImageReader delivered it.
	AtUnixNano int64
}

FrameMsg says a frame has been written into a slot of the shared buffer.

func DecodeFrame

func DecodeFrame(b []byte) (FrameMsg, error)

DecodeFrame decodes a FrameMsg.

func (FrameMsg) At

func (f FrameMsg) At() time.Time

At renders the delivery instant. A zero AtUnixNano yields the zero Time rather than the epoch, so "the host did not say" stays distinguishable.

type OpenDisplayMessage

type OpenDisplayMessage struct {
	// Width and Height are the display's size in pixels.
	Width, Height int
	// DensityDPI is the density Android lays the view hierarchy out against.
	DensityDPI int
	// Slots is how many frame slots the shared buffer holds.
	Slots int
	// ContentKind names the View the host should build; see [Content].
	ContentKind uint32
	// ContentPayload is that content's one string argument — a URL for a
	// WebView, a label for the sentinel.
	ContentPayload string
}

OpenDisplayMessage asks the wall host for one owned display.

func DecodeOpenDisplay

func DecodeOpenDisplay(b []byte) (OpenDisplayMessage, error)

DecodeOpenDisplay decodes an OpenDisplayMessage.

type OpenScreenMessage

type OpenScreenMessage struct {
	// DisplayID is the android.view.Display id to present on. It must be a
	// display the host can see and whose FLAG_PRESENTATION is set; the host
	// refuses anything else rather than guessing.
	DisplayID int
	// Width and Height are the size of the frames this process will write, in
	// pixels. They need not match the display: the host scales the bitmap to
	// fill it, which is how a ribbon rendered at one resolution reaches panels
	// of another.
	Width, Height int
	// Slots is how many frame slots the shared buffer holds.
	Slots int
}

OpenScreenMessage asks the host for a Presentation on a display that already exists, and for the shared buffer this process will paint into.

func DecodeOpenScreen

func DecodeOpenScreen(b []byte) (OpenScreenMessage, error)

DecodeOpenScreen decodes an OpenScreenMessage.

type Options

type Options struct {
	// Width and Height are the requested frame size in PIXELS. Zero means the
	// display's native size.
	Width, Height int

	// FPS is the CEILING on the frame rate, not a guarantee: a MediaProjection
	// only produces a frame when the content changed. Zero means [DefaultFPS].
	FPS float64

	// QueueDepth is how many frame slots the shared buffer holds. Zero means
	// [DefaultQueueDepth]. It must leave room for the frame lent out and the
	// one being written, so values below [MinQueueDepth] are rejected — with
	// two slots a host that produced two frames while the consumer held one
	// would overwrite the borrow.
	QueueDepth int
}

Options configures a capture stream.

The zero Options is usable: it captures the default display at its native pixel size, at up to 60 frames per second.

func (Options) Validate

func (o Options) Validate() error

Validate reports whether the options are self-consistent, wrapping ErrInvalidOption. It does not consult the system.

type OwnedDisplay

type OwnedDisplay struct {
	*Stream
	// contains filtered or unexported fields
}

OwnedDisplay is one display this application created, and the feed of pixels Android rendered on it.

It embeds Stream, so a ribbon consumes it exactly as it consumes a screen capture — Stream.Frame lends borrowed pixels with the stride carried and says whether they are fresh, Stream.WaitFrame blocks for the next one. That is the point: the ribbon does not care whether a panel is the mirrored phone or a display we made.

func (*OwnedDisplay) Close

func (d *OwnedDisplay) Close() error

Close releases the display and its feed, and gives the slot back to the wall. It is idempotent.

func (*OwnedDisplay) ID

func (d *OwnedDisplay) ID() int

ID is the android.view.Display id the platform gave this display. It is only meaningful for logs and for matching against `adb shell dumpsys display`; no other application may do anything with it.

func (*OwnedDisplay) Spec

func (d *OwnedDisplay) Spec() DisplaySpec

Spec is the specification this display was opened with, with its zero fields filled in.

func (*OwnedDisplay) String

func (d *OwnedDisplay) String() string

String renders the display for logs.

type PixelFormat

type PixelFormat uint32

PixelFormat names the layout of a captured frame, using Android's own android.graphics.PixelFormat constants so the value on the wire is the one the host read off the ImageReader.

const FormatRGBA PixelFormat = 1 // android.graphics.PixelFormat.RGBA_8888

FormatRGBA is 32-bit RGBA, android.graphics.PixelFormat.RGBA_8888. It is the only format this package streams, and it is what ImageReader produces for a MediaProjection mirror. It is also byte-for-byte what a go-widgets painter writes, so a captured frame composites with no conversion.

This differs from the sibling packages on other platforms: macOS ScreenCaptureKit hands back BGRA. Frame.NRGBA hides the difference, and anything compositing raw bytes must ask Stream.Format.

func (PixelFormat) BytesPerPixel

func (f PixelFormat) BytesPerPixel() int

BytesPerPixel is the size of one pixel in this format, 0 for a format this package does not stream.

func (PixelFormat) String

func (f PixelFormat) String() string

String renders the format for logs.

type PresentMessage

type PresentMessage struct {
	// Seq numbers the frame, from 1. It comes back in [PresentedMessage].
	Seq int64
	// Slot is which slot of the shared buffer holds the pixels.
	Slot int
	// Width, Height and Stride describe the pixels in that slot.
	Width, Height, Stride int
}

PresentMessage hands one slot of the shared buffer to the host to put on the screen. The geometry travels with it rather than being implied by the config, for the same reason FrameMsg carries it: the one place a wrong number would become an out-of-range slice over shared memory is the place to state it explicitly and check it.

func DecodePresent

func DecodePresent(b []byte) (PresentMessage, error)

DecodePresent decodes a PresentMessage.

type PresentedMessage

type PresentedMessage struct {
	// Seq is the [PresentMessage.Seq] of the frame that was presented.
	Seq int64
	// Slot is the slot it occupied, which the application may now draw into.
	Slot int
}

PresentedMessage says one frame reached the screen and its slot is free.

func DecodePresented

func DecodePresented(b []byte) (PresentedMessage, error)

DecodePresented decodes a PresentedMessage.

type Screen

type Screen struct {
	// contains filtered or unexported fields
}

A Screen is a Presentation on a display that ALREADY EXISTS — the glasses — carrying pixels this process painted.

Why this is a different thing from a Wall

A Wall creates displays and reads what Android renders on them. A Screen does neither: the display is the headset, the platform made it when the cable was plugged in, and the pixels travel the other way. The two share a host, a socket and a wire, and nothing else — a Wall is an INPUT, a Screen is the OUTPUT, and an XR application wants both at once.

Why an application may have one at all

An ordinary APK may not start an activity on a display it does not own: that is refused with a SecurityException, and the flag that would lift it needs a signature permission. A Presentation is not an activity start, so it is not what that check refuses, and it is the only route there is. Whether a given display will take one is the platform's own answer, carried in Display.Presentation — measured on a Pixel 11 Pro Fold with VITURE Beast glasses on the USB-C port:

APPEARED id 12 "VITURE Beast" 1920x1080 @110dpi 60.0Hz flags 0x8088 presentation true

No permission, no consent dialog and no MediaProjection are involved, exactly as for a Wall.

The back-pressure is the point

The host lends a shared buffer of ScreenOptions.QueueDepth slots. Screen.Next hands out a free one, the application draws into it, Screen.Present gives it to the host, and the slot comes back only once the host says the frame reached the view. Nothing else stops the application overwriting a slot mid-blit: the pixels go the other way here, so there is no stream of host frames to pace against, and a tear in a headset is not a cosmetic defect.

A Screen is safe for concurrent use. A Canvas is not: it is a window onto shared memory, and two goroutines drawing into one race.

func ShowOn

func ShowOn(ctx context.Context, d Display, o ScreenOptions) (*Screen, error)

ShowOn puts a Presentation on an existing display and returns the screen the application draws into.

It refuses a display the platform will not take a Presentation on with ErrNotPresentable, BEFORE dialling anything: the answer is already in the Display the host handed over, and a round trip would come back saying less. It wraps ErrInvalidOption for options that cannot describe a frame, and reports ErrUnsupported where there is no host.

Close the screen when done; it holds a socket, a mapping and a window.

func (*Screen) Close

func (s *Screen) Close() error

Close takes the Presentation down, unmaps the buffer and releases the socket. It is idempotent.

Any Canvas still out is invalid afterwards: the memory under it is gone.

func (*Screen) DisplayID

func (s *Screen) DisplayID() int

DisplayID is the android.view.Display id this screen presents on.

func (*Screen) Err

func (s *Screen) Err() error

Err reports the failure that ended the screen, or nil while it is live.

func (*Screen) Next

func (s *Screen) Next(ctx context.Context) (Canvas, error)

Next returns a canvas to draw the next frame into, waiting for a slot if every one is still in flight.

The pixels it hands back are BORROWED from the shared buffer and are whatever the frame that last used that slot left there, not blank: a compositor that paints the whole frame wants that, and one that does not must clear it.

It reports the context's error if the wait is abandoned, ErrScreenClosed after Screen.Close, and the host's failure if the host went away.

func (*Screen) Options

func (s *Screen) Options() ScreenOptions

Options are the options this screen was opened with, with the zero fields filled in.

func (*Screen) Present

func (s *Screen) Present(ctx context.Context, c Canvas) error

Present hands a drawn canvas to the host to put on the screen. The canvas is invalid afterwards whether this succeeded or not.

It refuses a canvas this screen did not lend, and one presented twice, with ErrInvalidOption: the slot number reaches the host as an offset into shared memory, and freeing a slot that is in use would let the next frame be drawn over one being blitted.

func (*Screen) Size

func (s *Screen) Size() (int, int)

Size is the frame size in pixels, which is what Screen.Next hands back. The display may be a different size; the host scales to fill it.

func (*Screen) Stats

func (s *Screen) Stats() ScreenStats

Stats reports what the screen has done.

func (*Screen) Stride

func (s *Screen) Stride() int

Stride is the distance in bytes between two rows of a frame.

func (*Screen) String

func (s *Screen) String() string

String renders the screen for logs.

type ScreenOptions

type ScreenOptions struct {
	// Width and Height are the size of the frames the application will draw,
	// in pixels. Zero means the display's own size.
	//
	// They need not match the display. The host scales the frame to fill it,
	// so a ribbon composited once at one resolution reaches panels of another —
	// and a screen rendered deliberately small is how a weak device keeps its
	// frame rate.
	Width, Height int

	// QueueDepth is how many frame slots the shared buffer holds. Zero means
	// [DefaultQueueDepth], and it is bounded by [MinQueueDepth] and
	// [MaxQueueDepth] exactly as a capture's is: the floor leaves room for the
	// frame being drawn, the one in flight and the one being blitted.
	QueueDepth int
}

ScreenOptions says what frames an application will put on a screen.

The zero value is usable and means: frames the size of the display, with DefaultQueueDepth slots.

func (ScreenOptions) String

func (o ScreenOptions) String() string

String renders the options for logs.

func (ScreenOptions) Validate

func (o ScreenOptions) Validate() error

Validate reports whether the options describe frames a host could carry, wrapping ErrInvalidOption. It does not apply defaults: a zero Width or QueueDepth is refused here and filled in by ShowOn before this is reached.

type ScreenStats

type ScreenStats struct {
	// Presented is how many frames the application handed over.
	Presented int64
	// Acknowledged is how many of those the host reported on the screen. The
	// difference is what is in flight.
	Acknowledged int64
	// Waited is how many times [Screen.Next] had to block because every slot
	// was in flight. A large number against a small QueueDepth means the
	// application is drawing faster than the headset can take it.
	Waited int64
}

ScreenStats counts what a screen has done.

func (ScreenStats) String

func (s ScreenStats) String() string

String renders the counters for logs.

type Sentinel

type Sentinel struct {
	// Label is drawn across the middle in white. An empty label draws none.
	Label string
}

Sentinel renders quadrants of exactly known colour with a label across them: green top-left, blue bottom-right, red elsewhere, white text.

It is this package's SELF-TEST content, and it is exported because a consumer needs it too. A feed that "works" and delivers a black buffer is the classic silent failure of this whole mechanism, and the only way to catch it is to draw something whose pixels are known in advance and then SAMPLE them rather than look at them. SentinelColorAt says what a correct frame must hold at a given point, and that is what this package's live proof asserts on.

func (Sentinel) Describe

func (s Sentinel) Describe() string

Describe renders the content for logs.

func (Sentinel) Validate

func (s Sentinel) Validate() error

Validate reports whether the label will fit a message.

type StartMessage

type StartMessage struct {
	// DisplayID is the display to capture. Only the default display can be
	// captured by an unprivileged app; see [ErrNotCapturable].
	DisplayID int
	// Width and Height are the requested frame size in pixels; zero means the
	// display's native size.
	Width, Height int
	// MilliFPS is the frame-rate ceiling in thousandths of a frame per second,
	// so 60 fps is 60000 and a rate below one frame a second is still exact.
	MilliFPS uint32
	// Slots is how many frame slots the shared buffer holds.
	Slots int
}

StartMessage asks the host to begin capturing.

func DecodeStart

func DecodeStart(b []byte) (StartMessage, error)

DecodeStart decodes a StartMessage.

type Stats

type Stats struct {
	// Frames is the number of frames the host delivered.
	Frames uint64
	// Superseded is the number of delivered frames replaced by a newer one
	// before the consumer ever asked for them. A large value next to Frames
	// means the consumer is slower than the capture.
	Superseded uint64
	// Last is when the most recent frame arrived.
	Last time.Time
	// Interval is the gap between the two most recent frames.
	Interval time.Duration
}

Stats reports what a stream has seen since it started.

func (Stats) FPS

func (s Stats) FPS() float64

FPS is the instantaneous rate implied by Stats.Interval, 0 when fewer than two frames have arrived.

type StoppedMessage

type StoppedMessage struct {
	// Reason is one of [StopUser], [StopSystem] or [StopApp].
	Reason uint8
	// Detail is the host's own wording, which may be empty.
	Detail string
}

StoppedMessage says a capture ended and why.

func DecodeStopped

func DecodeStopped(b []byte) (StoppedMessage, error)

DecodeStopped decodes a StoppedMessage.

func (StoppedMessage) String

func (s StoppedMessage) String() string

String renders the reason and the detail for a log or an error.

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

Stream is a running capture.

func CaptureDisplay

func CaptureDisplay(ctx context.Context, d Display, o Options) (*Stream, error)

CaptureDisplay starts streaming a display.

The user must already have consented — see RequestAuthorization — and only the default display can be captured. The returned stream must be closed.

func (*Stream) Close

func (st *Stream) Close() error

Close ends the capture and releases the shared mapping. It is idempotent, and every borrowed frame is invalid afterwards.

func (*Stream) Err

func (st *Stream) Err() error

Err reports why the SYSTEM stopped the capture, nil while it is running. The user revoking the projection arrives here wrapping ErrPermissionDenied.

func (*Stream) Format

func (st *Stream) Format() PixelFormat

Format returns the pixel layout of every frame this stream produces.

func (*Stream) Frame

func (st *Stream) Frame() (Frame, bool)

Frame returns the most recent frame and whether it is newer than the one the previous call returned.

The bytes are BORROWED from the shared mapping and stay valid until the next Frame, WaitFrame or Close. In steady state this allocates nothing.

func (*Stream) Options

func (st *Stream) Options() Options

Options returns the options actually in force, with every zero field resolved.

func (*Stream) Size

func (st *Stream) Size() (int, int)

Size returns the capture's frame size in pixels.

func (*Stream) Stats

func (st *Stream) Stats() Stats

Stats reports what the stream has seen since it started.

func (*Stream) String

func (st *Stream) String() string

String renders the stream for logs.

func (*Stream) WaitFrame

func (st *Stream) WaitFrame(ctx context.Context) (Frame, error)

WaitFrame blocks until a frame newer than the last one returned arrives, or ctx expires. It reports ErrNoFrame on expiry, which on a motionless screen is the ordinary answer rather than a malfunction.

type USBDescriptors

type USBDescriptors []byte

USBDescriptors is a device's raw descriptor bytes, as usbfs serves them: the device descriptor followed by every configuration descriptor and everything nested inside them.

They are kept RAW because the class-specific parts are what a camera is described by — UVC's format and frame descriptors live inside the interface descriptors and no general USB parser reports them — and because a byte slice read from a kernel is a thing a test can hold.

func (USBDescriptors) Device

func (d USBDescriptors) Device() (vendor, product, class, subclass, protocol int, err error)

Device reports the device descriptor's own fields: the vendor and product ids, and the device's class triple.

It reads the first 18 bytes and nothing else, so it is valid for any blob long enough to be a device at all.

func (USBDescriptors) Interfaces

func (d USBDescriptors) Interfaces() ([]USBInterface, error)

Interfaces returns every interface descriptor's identity, in the order the device reports them: number, alternate setting and class triple.

It is how a caller finds the UVC streaming interface and its alternate settings without this package growing a full USB parser.

func (USBDescriptors) UVCFormats

func (d USBDescriptors) UVCFormats(iface int) ([]UVCFormatDesc, error)

UVCFormats reads what one streaming interface offers, from the device's raw descriptors.

⛔ IT WALKS THE CLASS-SPECIFIC DESCRIPTORS, WHICH NO GENERAL PARSER REPORTS. A camera's formats and frame sizes live in type 0x24 descriptors nested after its streaming interface, and USBDescriptors.Interfaces steps straight over them. Everything needed to commit a format is in there and nowhere else.

It associates each frame descriptor with the format that preceded it, which is how UVC orders them, and stops at the next interface so a device with two streaming interfaces does not have its formats merged.

func (USBDescriptors) Walk

func (d USBDescriptors) Walk(fn func(typ int, b []byte) bool) error

Walk calls fn for every descriptor in the blob, with its type and its whole bytes including the two-byte header.

⛔ A DESCRIPTOR OF LENGTH ZERO WOULD LOOP FOREVER, and a length running past the end would read somebody else's memory. Both are refused: a kernel will not produce either, and this parses bytes that crossed a process boundary.

type USBDevice

type USBDevice struct {
	// Name is the device node the system calls it, which is what a USB
	// permission is granted against.
	Name string
	// VendorID and ProductID identify the device. VITURE is 0x35ca.
	VendorID, ProductID int
	// Manufacturer and Product are the device's own strings, when it has them.
	Manufacturer, Product string
	// Class, Subclass and Protocol are the DEVICE's class triple, which is
	// [USBClassPerInterface] for anything composite.
	Class, Subclass, Protocol int
	// Interfaces are every interface of every alternate setting.
	Interfaces []USBInterface
}

USBDevice is one device attached to the phone's USB host port.

func DecodeUSBDevices

func DecodeUSBDevices(b []byte) ([]USBDevice, error)

DecodeUSBDevices decodes a USB device list.

func USBDevices

func USBDevices(ctx context.Context) ([]USBDevice, error)

USBDevices returns every device on the phone's USB host port, down to each interface's endpoints.

The endpoints are the point: whether a headset's camera can be read through Android's Java USB API comes down to whether its UVC streaming endpoints are bulk, and nothing but a census says. See USBDevice.BulkVideo.

It needs no permission either. A USB permission is granted against a device to OPEN it; the device list is public.

func (USBDevice) BulkVideo

func (d USBDevice) BulkVideo() bool

BulkVideo reports whether any of this device's UVC streaming interfaces delivers frames over a BULK endpoint.

⛔ IT IS THE WHOLE QUESTION FOR THE USB ROUTE. Android's Java USB API submits control, bulk and interrupt transfers and NOTHING ELSE: there is no isochronous request in UsbDeviceConnection or UsbRequest, which is why every UVC library on Android carries a native libusb. A camera whose streaming endpoints are all isochronous cannot be read through that API at all, however much permission it is given.

func (USBDevice) Readable

func (d USBDevice) Readable() []USBInterface

Readable returns the interfaces of this device whose IN endpoints Android's Java USB API can actually carry — bulk and interrupt, never isochronous.

⭐ IT IS WHAT IS LEFT WHEN THE CAMERA IS CLOSED. A headset whose camera is isochronous is not a headset with nothing to read: the VITURE Beast also presents a CDC-ACM serial interface with bulk endpoints and a HID interface with interrupt ones, and both of those UsbDeviceConnection can read today. What travels on them is a separate question this package does not answer — see go-xrkit/xrkit/headflow on the Beast publishing no orientation — but "unreachable" and "reachable and silent" are different findings and a census that could not tell them apart would be worth little.

func (USBDevice) String

func (d USBDevice) String() string

String renders the device for logs.

func (USBDevice) VideoStreaming

func (d USBDevice) VideoStreaming() []USBInterface

VideoStreaming returns the UVC streaming interfaces this device has, across every alternate setting.

func (USBDevice) Viture

func (d USBDevice) Viture() bool

Viture reports whether this device is a VITURE one.

type USBEndpoint

type USBEndpoint struct {
	// Address is bEndpointAddress: the endpoint number, with bit 7 set for an
	// IN endpoint.
	Address int
	// Attributes is bmAttributes, whose low two bits are the transfer type.
	Attributes int
	// MaxPacketSize is wMaxPacketSize for this endpoint's current setting.
	MaxPacketSize int
	// Interval is bInterval, the polling period.
	Interval int
}

USBEndpoint is one endpoint of a USB interface.

func (USBEndpoint) ISOPacketSize

func (e USBEndpoint) ISOPacketSize() int

ISOPacketSize is how many bytes one isochronous service interval may carry.

⛔⛔ wMaxPacketSize IS NOT A SIZE ON A HIGH-SPEED ISOCHRONOUS ENDPOINT. Bits 10:0 are the packet size; bits 12:11 are the number of ADDITIONAL transactions per microframe, so the bytes per interval are the product. The Beast's camera advertises 5120 on its largest alternate, which is 0x1400:

size = 0x1400 & 0x7ff       = 1024
mult = ((0x1400>>11) & 3)+1 = 3
bytes per interval          = 3072

Passing the raw 5120 as a packet length had the kernel answer EMSGSIZE — "message too long" — and that was read here as the platform REFUSING the transfer. It was refusing nothing: the request was malformed. This is the trap libusb's get_max_iso_packet_size exists to hide.

func (USBEndpoint) In

func (e USBEndpoint) In() bool

In reports whether this endpoint carries data towards the host. A camera's streaming endpoint is one.

func (USBEndpoint) String

func (e USBEndpoint) String() string

String renders the endpoint for logs.

func (USBEndpoint) Type

func (e USBEndpoint) Type() USBTransferType

Type is the endpoint's transfer type.

type USBHandle

type USBHandle struct {
	// contains filtered or unexported fields
}

A USBHandle is one USB device this application was given permission to open, held as the kernel's own descriptor.

⛔ IT IS NOT A STREAM AND NOT AN API TO BUILD ON. It exists to answer whether Android lets an untrusted application drive usbfs at all — see usbfs.go. Every method here is a step of that question, and the package stops when the question is answered.

A USBHandle is safe for concurrent use; the device it names is not, and two callers claiming the same interface will have the kernel refuse the second.

func OpenUSB

func OpenUSB(ctx context.Context, d USBDevice) (*USBHandle, error)

OpenUSB asks the host to open a USB device and lends its descriptor.

⚠ IT PUTS A DIALOG IN FRONT OF THE USER, naming the device and this application, and they may say no — which is reported as ErrUSBPermissionDenied and is a decision rather than a failure. A device that is no longer attached is ErrNotFound.

Close the handle when done: it holds a socket, a descriptor, and the host's claim on the device.

func (*USBHandle) ClaimInterface

func (h *USBHandle) ClaimInterface(n int) error

ClaimInterface takes an interface from whatever kernel driver holds it, which is required before any transfer on its endpoints.

On a UVC camera the kernel's own uvcvideo driver usually holds it, and whether Android lets an application take it away is part of the question.

func (*USBHandle) Close

func (h *USBHandle) Close() error

Close releases every interface this handle claimed, closes the descriptor and drops the host connection, which lets the host release the device. It is idempotent.

func (*USBHandle) Control

func (h *USBHandle) Control(requestType, request uint8, value, index uint16, data []byte) (int, error)

Control performs one control transfer and returns how many bytes moved.

⛔⛔ data IS WRITTEN INTO BY THE KERNEL ON AN IN TRANSFER, and its length is what the device is told to send. A slice shorter than the device's answer gets a short read; one longer invites it to overrun nothing, because the length in the request is the ceiling. Either way the number returned is what actually moved, and callers must use it rather than len(data).

It is the whole of UVC's negotiation: asking a camera what it can do and telling it what to do are both control transfers on its streaming interface.

func (*USBHandle) Descriptors

func (h *USBHandle) Descriptors() (USBDescriptors, error)

Descriptors reads the device's raw descriptor blob.

⭐ IT IS A PLAIN read(2), NOT AN IOCTL. usbfs serves the descriptors as the file's contents, so this is the one thing here that works without the kernel granting anything beyond the open — which makes it the right FIRST step of the probe: a failure here is the descriptor itself being useless, and a success narrows every later failure to the ioctl that produced it.

func (*USBHandle) Detached

func (h *USBHandle) Detached() []int

Detached returns the interfaces whose kernel driver this handle took away.

⛔⛔ IT IS NOT BOOKKEEPING, IT IS A DEBT. Taking a UVC streaming interface from uvcvideo is what makes the route work at all, and USBDEVFS_CONNECT does NOT reliably undo it: on a Pixel 11 Pro Fold the reconnect ioctl SUCCEEDS and the driver still does not rebind —

before:  1-1.1:1.0 driver=uvcvideo   1-1.1:1.1 driver=uvcvideo
after:   1-1.1:1.0 driver=uvcvideo   1-1.1:1.1 driver=

/dev/video0 survives, because the control interface still carries it, but the streaming interface is left with no driver until the device is UNPLUGGED AND PLUGGED BACK. A caller that shares the camera with anything else on the phone needs to know that, which is why this is reported rather than hidden.

func (*USBHandle) Name

func (h *USBHandle) Name() string

Name is the device node this handle was opened for.

func (*USBHandle) Negotiate

func (h *USBHandle) Negotiate(iface int, f UVCFormatDesc, z UVCFrameSize) (UVCNegotiation, error)

Negotiate performs UVC's probe/commit on a streaming interface and returns what the camera agreed to.

The sequence is the specification's, and every step of it is a control transfer:

1. GET_LEN on the probe control, so the block's size is LEARNED rather than guessed from a version number. It is 26 bytes in UVC 1.0, 34 in 1.1 and 48 in 1.5, and a block of the wrong length is refused outright;
2. GET_CUR, to take the camera's own block and leave every field this package does not understand exactly as it was;
3. SET_CUR with the format, frame and interval written in;
4. GET_CUR again — THE ANSWER. This is where a camera says what it will actually do, which may not be what it was asked;
5. SET_CUR on the commit control, which is the same block accepted.

⛔ THE INTERFACE MUST ALREADY BE CLAIMED. These transfers address the streaming interface by number, and a kernel driver still holding it will answer them itself or refuse them.

func (*USBHandle) SetAltSetting

func (h *USBHandle) SetAltSetting(iface, alt int) error

SetAltSetting selects an alternate setting of an interface. For a UVC streaming interface this is how bandwidth is chosen, and setting 0 is the zero-bandwidth one every UVC device has.

func (*USBHandle) Stream

func (h *USBHandle) Stream(neg UVCNegotiation, ep USBEndpoint, w, h2 int) (*UVCStream, error)

Stream starts reading frames from a camera that has already been negotiated.

⛔ THE ORDER MATTERS AND NOTHING ENFORCES IT BUT THIS SENTENCE: the interface must be claimed, its alternate selected and the format committed before this is called. A stream started before the commit reaps packets of nothing, which looks exactly like a camera with the lens covered.

func (*USBHandle) SubmitISO

func (h *USBHandle) SubmitISO(endpoint USBEndpoint) error

type USBInterface

type USBInterface struct {
	// Number is bInterfaceNumber, and Alternate is bAlternateSetting.
	Number, Alternate int
	// Class, Subclass and Protocol are the interface's own class triple.
	Class, Subclass, Protocol int
	// Endpoints are the endpoints of this alternate setting.
	Endpoints []USBEndpoint
}

USBInterface is one interface of a USB device, in one alternate setting.

func (USBInterface) String

func (i USBInterface) String() string

String renders the interface for logs.

func (USBInterface) Video

func (i USBInterface) Video() bool

Video reports whether this interface is a UVC one.

func (USBInterface) VideoStreaming

func (i USBInterface) VideoStreaming() bool

VideoStreaming reports whether this is the UVC interface that carries frames.

type USBTransferType

type USBTransferType int

USBTransferType is an endpoint's transfer type, the low two bits of its bmAttributes.

const (
	USBTransferControl     USBTransferType = 0
	USBTransferIsochronous USBTransferType = 1
	USBTransferBulk        USBTransferType = 2
	USBTransferInterrupt   USBTransferType = 3
)

The transfer types USB defines.

func (USBTransferType) String

func (t USBTransferType) String() string

String renders the transfer type.

type UVCFormat

type UVCFormat int

UVC format identities this package names.

const (
	// FormatUnknown is a format descriptor this package did not recognise. It
	// is kept rather than dropped: a camera that offers only formats we cannot
	// name is a different finding from a camera that offers none.
	FormatUnknown UVCFormat = iota
	// FormatYUY2 is uncompressed 4:2:2, two bytes a pixel, luminance first.
	// ⭐ It is the one to want: see the note at the top of this file.
	FormatYUY2
	// FormatMJPEG is a JPEG per frame, which Go's image/jpeg decodes.
	FormatMJPEG
)

The formats a camera here is expected to offer.

func (UVCFormat) String

func (f UVCFormat) String() string

String renders the format.

type UVCFormatDesc

type UVCFormatDesc struct {
	// Index is the camera's own bFormatIndex.
	Index int
	// Format is what this package made of its GUID or subtype.
	Format UVCFormat
	// Sizes are the frame sizes it offers, in the order the camera lists them.
	Sizes []UVCFrameSize
}

A UVCFormatDesc is one format a streaming interface offers, with its sizes.

func (UVCFormatDesc) Fastest

func (d UVCFormatDesc) Fastest() (UVCFrameSize, bool)

Fastest returns the size with the HIGHEST FRAME RATE this format offers, at no less than MinTrackingWidth across, breaking ties toward the wider frame.

⛔⛔ RATE, NOT SIZE, AND THAT WAS MEASURED THE WRONG WAY ROUND FIRST. This picked the largest frame, reasoning that more pixels across means more to match. The camera's own table says what that costs:

YUY2  1920x1080 @5fps    1280x720 @10fps    800x600 @20fps    640x480 @25fps
MJPEG 1920x1080 @30fps   … 30fps at every size

So "largest" chose 1920x1080 at FIVE frames a second — 200 ms a frame, worse than the 46–107 ms the same algorithm already achieves on macOS, for a precision nothing here needs. Latency is what head tracking is bounded by; headflow's own documentation says 1:1 locking wants under 20 ms and that choosing which screen somebody faces is comfortable at 107 ms.

func (UVCFormatDesc) String

func (d UVCFormatDesc) String() string

String renders the format and its sizes.

type UVCFrameSize

type UVCFrameSize struct {
	// Index is the camera's own bFrameIndex, which is what a commit names.
	Index int
	// Width and Height are in pixels.
	Width, Height int
	// Interval is the default frame interval in 100-nanosecond units, which is
	// how UVC states frame rates: 333333 is 30 per second.
	Interval uint32
}

A UVCFrameSize is one size a format offers, with the shortest interval between frames the camera will sustain at it.

func (UVCFrameSize) FPS

func (s UVCFrameSize) FPS() float64

FPS is the frame rate the interval states, or 0 when the camera gave none.

func (UVCFrameSize) String

func (s UVCFrameSize) String() string

String renders the size for logs.

type UVCNegotiation

type UVCNegotiation struct {
	// Format and Frame are the indices the camera settled on.
	Format, Frame int
	// Interval is the frame interval it settled on, in 100ns units.
	Interval uint32
	// MaxPayload is the largest transfer it will send, which bounds how much
	// one isochronous request can usefully carry.
	MaxPayload uint32
	// Asked is what it was given, kept so a caller can SEE a change rather
	// than infer one.
	Asked UVCFrameSize
	// Kind is the format's KIND, as against Format which is the camera's own
	// index for it.
	//
	// ⛔ A NEGOTIATION RESULT SHOULD SAY WHAT WAS AGREED, and an index is a name
	// the camera chose rather than a fact about the bytes that follow. Without
	// it a stream cannot tell whether to assemble by LENGTH or by MARKERS — and
	// that distinction is what makes a compressed frame possible at all. The
	// alternative was a second argument on [USBHandle.Stream], which would have
	// put the same fact in two places and let them disagree.
	Kind UVCFormat
}

A UVCNegotiation is what a camera agreed to.

⛔ AGREED TO, NOT ASKED FOR. A UVC camera may answer a probe with something other than what it was given — a different interval, a smaller payload — and the host is expected to accept that answer. Reporting the REQUEST as though it were the outcome is how a stream comes out at a frame rate nobody chose.

⚠ IT IS DECLARED HERE, NOT BESIDE Negotiate. The type is data and travels everywhere; only the negotiation needs a device. Declaring it twice, once per platform, left String on one of them — which go vet caught as a %s against a type with no String, on the lane that could not see the other copy.

func (UVCNegotiation) Changed

func (n UVCNegotiation) Changed() bool

Changed reports whether the camera settled on something other than what it was asked for.

func (UVCNegotiation) FPS

func (n UVCNegotiation) FPS() float64

FPS is the frame rate the camera settled on.

func (UVCNegotiation) String

func (n UVCNegotiation) String() string

String renders the outcome for logs.

type UVCPayload

type UVCPayload struct {
	// Header is the payload's own bmHeaderInfo.
	Header byte
	// Data is the pixels, which is whatever followed the header.
	Data []byte
}

A UVCPayload is one isochronous packet's worth of a frame.

func ParsePayload

func ParsePayload(b []byte) (UVCPayload, error)

ParsePayload splits one isochronous packet into its header and its pixels.

⛔ bHeaderLength IS THE OFFSET AND IT IS NOT CONSTANT. Cameras use 2, 12 and more; taking a fixed two bytes would prepend ten bytes of timestamp to every frame's pixels, which does not fail — it shears the image by five YUY2 pixels and looks like a camera mounted crooked.

A packet shorter than its own header, or one whose header is longer than the packet, is refused rather than clamped: both describe a packet that is not the one the camera meant to send.

func (UVCPayload) Bad

func (p UVCPayload) Bad() bool

Bad reports whether the camera marked the payload itself as in error.

func (UVCPayload) EOF

func (p UVCPayload) EOF() bool

EOF reports whether the camera marked this the last payload of its frame.

func (UVCPayload) FID

func (p UVCPayload) FID() bool

FID is the frame this payload belongs to, as a single bit that toggles.

type UVCStream

type UVCStream struct {
	// contains filtered or unexported fields
}

A UVCStream is a running camera.

⛔ IT IS NOT A Stream. The capture and wall paths deliver frames the HOST wrote into shared memory and hand them over borrowed; this one is built from isochronous packets this process reaped itself, assembled, and decoded. They share no machinery and the names are deliberately different.

A UVCStream is NOT safe for concurrent use: the URB ring, the assembler and the decode buffer are all single-reader state, and there is no reason for two goroutines to read one camera.

func (*UVCStream) Assembler

func (s *UVCStream) Assembler() *FrameAssembler

Assembler is the frame assembler this stream feeds, for the counters that describe HOW frames ended rather than how many did.

func (*UVCStream) Close

func (s *UVCStream) Close() error

Close cancels every outstanding request. It is idempotent.

⛔ IT DISCARDS WHAT IS STILL IN FLIGHT. A request left outstanding has the kernel writing into this process's memory on the camera's schedule, and the Go heap is entitled to move on without it.

func (*UVCStream) Frame

func (s *UVCStream) Frame(ctx context.Context) (*image.RGBA, error)

Frame reads until one whole frame has been assembled, and returns it decoded.

The image is REUSED between calls, so a steady stream allocates nothing after the first frame. A caller that keeps one copies it.

It reports the context's error if the wait is abandoned, and ErrClosed after UVCStream.Close.

func (*UVCStream) Negotiation

func (s *UVCStream) Negotiation() UVCNegotiation

Negotiation is what the camera agreed to for this stream.

func (*UVCStream) Stats

func (s *UVCStream) Stats() UVCStreamStats

Stats reports what the stream has seen.

type UVCStreamStats

type UVCStreamStats struct {
	// Packets is every isochronous packet reaped, empty ones included.
	Packets int
	// Empty is how many carried nothing, which between frames is most of them.
	Empty int
	// Bytes is the payload actually delivered, headers and all.
	Bytes int
	// Frames, Dropped and Bad come from the assembler: whole frames, frames
	// abandoned for being short, and payloads the camera marked in error.
	Frames, Dropped, Bad int
	// Late is whole frames thrown away because the CALLER had not come back for
	// the ones before them.
	//
	// ⭐ IT IS SEPARATE FROM Dropped ON PURPOSE. A dropped frame is the
	// camera's or the bus's fault and says the link is lossy; a late one is
	// OURS and says the consumer is slower than the camera. They are different
	// problems with different fixes, and a single counter would make a slow
	// tracker look like a failing cable.
	Late int
	// Foreign is completions collected from this descriptor that belonged to
	// some other request.
	//
	// ⛔ IT IS COUNTED BECAUSE A COMPLETION QUEUE BELONGS TO THE DESCRIPTOR,
	// not to whoever submitted. Anything else reading the same device — or, as
	// happened here, this package's own [USBHandle.SubmitISO] cancelling a test
	// request without collecting it — puts a completion in the way. Skipping it
	// is right; skipping it silently would make a stream that is eating other
	// people's requests look merely slow.
	Foreign int
	// Decoding is how long the decodes have taken in all.
	//
	// ⛔⛔ IT IS MEASURED ON THE DEVICE BECAUSE A BENCHMARK ELSEWHERE IS AN
	// OPTIMISTIC BOUND AND NOT A MEASUREMENT. Choosing between YUY2 and MJPEG
	// turns on whether the decode fits inside the frame interval, and the
	// processor that has to fit it is the phone's. On a workstation MJPEG costs
	// 3 % of a 30 fps interval at 320x240, 15-18 % at 640x480 and 69-84 % at
	// 1920x1080 — figures that say the shape of the answer and not the answer.
	Decoding time.Duration
	// BadJPEG is frames that arrived whole, both markers present, and still
	// would not decode. See [FrameAssembler.BadJPEG] for why it is apart from
	// Dropped.
	BadJPEG int
}

UVCStreamStats counts what a stream has seen.

It is declared here rather than beside the stream because it is data and travels everywhere: declaring a type once per platform is how String ends up on only one of them, which go vet caught once already in this package.

func (UVCStreamStats) String

func (s UVCStreamStats) String() string

String renders the counters for logs.

type Wall

type Wall struct {
	// contains filtered or unexported fields
}

A Wall is a set of displays this application created and owns, and the limit on how many of them may exist at once.

Why an application may have these at all

An ordinary APK may not manufacture desktops for OTHER people's applications: launching an activity onto a display the app created fails with a SecurityException even for the app's own activity, and the flag that would lift it, VIRTUAL_DISPLAY_FLAG_TRUSTED, needs a signature permission. That half of the refusal is real and is not going anywhere.

It may, however, create a virtual display and show a [Presentation] on it — a Dialog attached to a Display, which is not an activity start and so does not meet the check that refuses the other thing. Measured on Android 15 (API 35, arm64):

Q1 ANSWER: Presentation.show() SUCCEEDED on the app's OWN virtual display 3
Q2 frame 640x480 ... black=0/307200 topLeft=#FF00FF00 bottomRight=#FF0000FF

No permission, no consent dialog and no MediaProjection are involved, which is why a Wall needs none of RequestAuthorization's machinery. So an XR ribbon on Android carries the mirrored phone — see CaptureDisplay — AND as many screens of Android-rendered content as the application asks for.

THE LIMIT IS THE SAFETY, AND IT IS NOT A FORMALITY

On the emulator this package was measured against, creating the 304th virtual display did not fail: it killed system_server with android.view.Surface$OutOfResourcesException: NO_MEMORY out of SurfaceControl.nativeCreate, and the device soft-rebooted. An unprivileged APK with no permissions at all did that to the machine it was running on.

READ THIS BEFORE RAISING THE LIMIT: the ceiling was the SAME COUNT at 640x480 and at 1920x1080 — nine times the pixels, same number, with the app's Java heap at 14 MiB of 192. It is SurfaceControl handles, not graphics memory. So MAKING THE SCREENS SMALLER DOES NOT HELP, which is exactly the mitigation somebody reaches for. 304 is one emulator's measurement and is deliberately not a constant here; a real device's number is unknown and could be lower.

A Wall therefore refuses past its limit with ErrTooManyDisplays instead of asking the platform and hoping. DefaultMaxDisplays is 8, which is more than a ribbon uses; MaxDisplays is 32, which is the most this package will let a caller ask for however explicit they are about wanting it. The Java host enforces the same 32 independently, so a second process cannot get past it either.

A Wall is safe for concurrent use.

func NewWall

func NewWall(max int) (*Wall, error)

NewWall returns a wall that will hold at most max displays at once. A max of 0 means DefaultMaxDisplays; a negative max, or one above MaxDisplays, is refused with ErrTooManyDisplays and no wall is returned.

NewWall talks to nothing and can fail for no other reason, so a caller may build one before knowing whether a host is there.

func (*Wall) Close

func (w *Wall) Close() error

Close releases every display the wall still holds and refuses further Wall.Open calls. It is idempotent, and reports the first failure a display reported while still closing the rest.

func (*Wall) Len

func (w *Wall) Len() int

Len is how many displays the wall holds open right now.

func (*Wall) Max

func (w *Wall) Max() int

Max is the limit this wall was built with.

func (*Wall) Open

func (w *Wall) Open(ctx context.Context, spec DisplaySpec) (*OwnedDisplay, error)

Open creates one display, shows a Presentation carrying spec.Content on it, and returns a feed of the pixels Android rendered there.

It refuses with ErrTooManyDisplays when the wall is already full, and it refuses BEFORE asking the host, so a full wall costs no round trip and creates nothing. It refuses with ErrWallClosed after Wall.Close, and wraps ErrInvalidOption for a spec that cannot describe a display.

type Web

type Web struct {
	// URL is what the WebView loads. http, https and data URLs are accepted.
	URL string
}

Web renders a WebView on the display.

It is the content that justifies the whole mechanism: a CGO-free Go process cannot host a web engine, and a ribbon of web panels is a real product that falls straight out of a Presentation on an owned display.

The host's WebView has JavaScript enabled and no file access. Reaching a http(s) URL needs android.permission.INTERNET in the APK, which is protectionLevel normal; a data: URL needs nothing and is what this package's own tests use.

func (Web) Describe

func (w Web) Describe() string

Describe renders the content for logs.

func (Web) Validate

func (w Web) Validate() error

Validate reports whether the URL is one the host will load.