Documentation
¶
Overview ¶
Package computer implements computer-use: reading and operating native desktop application UI.
It is the sibling of internal/browser and is deliberately shaped like it: a process-lifetime Manager owns Backends, a task-lifetime Session owns per-task state, and Session.Close never closes the Backend. Snapshots are uid-annotated accessibility text rendered by internal/uitree, the same renderer browser-use uses, so the agent reads native apps with the same vocabulary it reads web pages.
jcode is built CGO_ENABLED=0 (agent-eval finding F1: cgo SIGABRTs on subprocess fork on macOS 26), so the macOS AX / CGEvent / ScreenCaptureKit calls cannot live in this process. Production delegates them to the native macOS helper over a Unix socket. Tests and explicit jcode_eval builds may inject a scripted backend; persisted settings cannot select one.
See internal-doc/computer-use-design.md.
Index ¶
- Constants
- Variables
- func IsBrowser(bundleID string) bool
- func Preapproved(c *config.ComputerConfig, bundleID, class string) bool
- func Supported() bool
- func SupportedMacOSVersion(productVersion string) bool
- func SupportedPlatform(goos string) bool
- func UnsupportedReason() string
- type ActRequest
- type Action
- type App
- type AppPermission
- type Backend
- type Config
- type FakeBackend
- func (f *FakeBackend) Actions() []Action
- func (f *FakeBackend) Capture(_ context.Context, bundleID string) ([]byte, error)
- func (f *FakeBackend) CaptureVisual(_ context.Context, bundleID string) (Screenshot, error)
- func (f *FakeBackend) Close() error
- func (f *FakeBackend) Closed() bool
- func (f *FakeBackend) Frontmost(context.Context) (App, error)
- func (f *FakeBackend) Kind() string
- func (f *FakeBackend) Launch(_ context.Context, bundleID string) error
- func (f *FakeBackend) ListApps(context.Context) ([]App, error)
- func (f *FakeBackend) Perform(_ context.Context, act Action) error
- func (f *FakeBackend) ReadClipboard(context.Context) (string, error)
- func (f *FakeBackend) SetApps(apps ...App)
- func (f *FakeBackend) SetClipboard(text string)
- func (f *FakeBackend) SetFrontmost(a App)
- func (f *FakeBackend) SetJournal(path string)
- func (f *FakeBackend) SetShot(bundleID string, png []byte)
- func (f *FakeBackend) SetTree(bundleID string, nodes []uitree.Node)
- func (f *FakeBackend) SetVisualShot(bundleID string, shot Screenshot)
- func (f *FakeBackend) Tree(_ context.Context, bundleID string) ([]uitree.Node, error)
- type HelperPermissions
- type HelperStatus
- type Manager
- func (m *Manager) Close() error
- func (m *Manager) Enabled() bool
- func (m *Manager) GetConfig() Config
- func (m *Manager) MaxBatch() int
- func (m *Manager) OpenScreenshot(id string) (*os.File, error)
- func (m *Manager) OpenSession(ctx context.Context) (*Session, error)
- func (m *Manager) Preapproved(bundleID, class string) bool
- func (m *Manager) RequestPermissions(ctx context.Context, accessibility, screenRecording bool) (HelperPermissions, error)
- func (m *Manager) SaveScreenshot(png []byte) (string, error)
- func (m *Manager) SetConfig(cfg Config)
- func (m *Manager) SetFakeBackend(b Backend)
- func (m *Manager) Status(ctx context.Context) Status
- func (m *Manager) TierOverrides() map[string]Tier
- type NotAllowedError
- type PermissionState
- type Screenshot
- type Session
- func (s *Session) Act(ctx context.Context, steps []ActRequest) (string, error)
- func (s *Session) Apps(ctx context.Context) (string, error)
- func (s *Session) BackendKind() string
- func (s *Session) Close() error
- func (s *Session) FrontmostBundle(ctx context.Context) string
- func (s *Session) Grant(bundleIDs []string, clipRead, clipWrite, sysKeys bool)
- func (s *Session) Granted() []string
- func (s *Session) Open(ctx context.Context, bundleID string) (string, error)
- func (s *Session) Read(ctx context.Context, kind string) (string, error)
- func (s *Session) Screenshot(ctx context.Context, bundleID string) ([]byte, error)
- func (s *Session) ScreenshotVisual(ctx context.Context, bundleID string) (Screenshot, error)
- func (s *Session) SetTierOverrides(m map[string]Tier)
- func (s *Session) Snapshot(ctx context.Context, bundleID, filter string, maxLines int, disableDiff bool) (string, error)
- func (s *Session) TierFor(bundleID string) Tier
- type Status
- type Tier
- type TierError
- type VisualCaptureBackend
Constants ¶
const MaxScreenshotBytes int64 = 20 << 20
MaxScreenshotBytes bounds both native IPC handoff files and any injected backend result before the tool creates a Base64 copy for the model request.
const ( // MinimumMacOSVersion is the deployment target used by the native Swift // helper and capture worker. Keep the user-facing requirement in one place. MinimumMacOSVersion = "14.0" )
Variables ¶
var ErrControlInterrupted = errors.New("computer control interrupted")
ErrControlInterrupted reports that the human took over (moved the mouse, switched apps deliberately, hit a kill switch). The agent must stop rather than retry: if the human grabbed the mouse, they had a reason.
Mirrors browser.ErrControlInterrupted and gets the same treatment in the tool layer — swallowed into a natural-language "stopping" message.
var ErrScreenLocked = errors.New("screen is locked")
ErrScreenLocked reports that the screen is locked. Not configurable: an agent driving a machine its owner believes is secured is not a feature.
var ErrStaleUID = errors.New("stale element uid")
ErrStaleUID reports a uid minted in an earlier snapshot generation. Rejecting it is load-bearing — on a native desktop a stale uid that now resolves to a different element means clicking the wrong button in a real app.
Functions ¶
func IsBrowser ¶
IsBrowser reports whether the bundle id is a known browser, so callers can point the model at browser-use rather than just refusing.
func Preapproved ¶
func Preapproved(c *config.ComputerConfig, bundleID, class string) bool
Preapproved reports whether class ("launch"/"interact") on bundleID is pre-authorized, consulting the per-app override first and the class default second. It is the body behind ApprovalState.SetComputerPermFunc.
An empty bundle id never pre-approves: if the app cannot be named, there is no basis for claiming the user approved it. (browserSitePreapproved makes the same call for an empty origin, for the same reason.)
func Supported ¶
func Supported() bool
Supported reports whether this process can offer native computer use. Computer use deliberately has no cross-platform fallback: production builds only drive the macOS Accessibility and Screen Recording helper.
func SupportedMacOSVersion ¶
SupportedMacOSVersion reports whether productVersion satisfies the native helper's minimum deployment target. It is deliberately pure so the version policy can be covered on every CI platform.
func SupportedPlatform ¶
SupportedPlatform is the pure form used by platform-gating tests.
func UnsupportedReason ¶
func UnsupportedReason() string
UnsupportedReason returns a stable, actionable message for APIs and CLIs.
Types ¶
type ActRequest ¶
type ActRequest struct {
Action string `json:"action"`
UID string `json:"uid"`
Value string `json:"value"`
Key string `json:"key"`
Text string `json:"text"`
Name string `json:"name"`
X *float64 `json:"x"`
Y *float64 `json:"y"`
ToX *float64 `json:"to_x"`
ToY *float64 `json:"to_y"`
Direction string `json:"direction"`
Pages float64 `json:"pages"`
}
ActRequest is one action as the model expressed it.
type Action ¶
type Action struct {
Kind string // click, type, press, set_value, scroll, drag, select_text, menu, hover, dblclick, rclick
BundleID string
UID string
Ref int64 // resolved backend handle for UID
Value string
Key string
Text string
Name string // named AX secondary action for Kind=="menu"
X, Y float64
ToX, ToY float64
// Coordinate presence is separate from value because zero is a valid global
// screen coordinate. Without these bits, JSON omitempty turns x=0 into a
// missing field at the daemon boundary.
HasX, HasY, HasToX, HasToY bool
Direction string
Pages float64
}
Action is one UI interaction.
BundleID is the *resolved* target, pinned at approval time and never re-derived from a display name later. Codex hit a real TOCTOU attack here — a mutable `app` field that returned "Calculator" to the approval check and "Terminal" to the executor — and defends with input freezing. Go copies structs by value, which gets us most of that, but the discipline is the same: resolve identity once, carry the resolved value.
type AppPermission ¶
AppPermission is a per-app configuration row.
type Backend ¶
type Backend interface {
Kind() string
// ListApps returns installed/running apps. The names are attacker-
// controllable and callers must treat them as tainted data.
ListApps(ctx context.Context) ([]App, error)
// Frontmost returns the app that currently has focus. This is the identity
// the tier gate checks, and it must be read fresh immediately before every
// action — a synthesized event goes to whatever holds focus now.
Frontmost(ctx context.Context) (App, error)
// Tree returns the accessibility tree of one app's windows.
Tree(ctx context.Context, bundleID string) ([]uitree.Node, error)
// Capture returns a PNG of one app's windows. Window-scoped, not
// screen-scoped: a non-granted app is not filtered out of the capture, it
// was never in it.
Capture(ctx context.Context, bundleID string) ([]byte, error)
// Launch starts or focuses an app.
Launch(ctx context.Context, bundleID string) error
// ReadClipboard returns the clipboard text. Gated by the clipboard_read
// grant, which is deliberately separate from any app grant — the clipboard
// belongs to the user, not to the app that happens to be in front.
ReadClipboard(ctx context.Context) (string, error)
// Perform executes one action. Implementations must auto-wait for the UI to
// settle before returning.
Perform(ctx context.Context, act Action) error
Close() error
}
Backend is the platform side of computer use. Shipping binaries use helperBackend (the native macOS daemon over a Unix socket). FakeBackend is injectable only by tests and explicit jcode_eval wiring.
Every method takes a ctx with a deadline. An unanswered TCC prompt presents as a silent multi-minute hang, not an error, so a missing deadline anywhere here is a wedged agent.
type Config ¶
type Config struct {
Enabled bool
// Backend is a compatibility field for internal callers compiled against the
// old shape. Runtime selection deliberately ignores it; persisted legacy
// values are migrated in config.ComputerConfig before reaching this package.
//
// Deprecated: inject a FakeBackend explicitly with SetFakeBackend in tests and
// evals. Production always uses the native helper.
Backend string
Approval map[string]string
AppPermissions []AppPermission
MaxActionsPerBatch int
ClipboardRead bool
ClipboardWrite bool
SystemKeyCombos bool
}
Config is the process-wide computer-use configuration.
It deliberately mirrors config.ComputerConfig rather than importing it, so this package does not depend on the config package. Exactly one mapper converts between them (see internal/computer/configmap.go).
func FromConfig ¶
func FromConfig(c *config.ComputerConfig) Config
FromConfig maps the persisted config into this package's Config, applying defaults.
This is the *only* mapper between the two shapes, and it is deliberately here rather than in the command or web layer. Keeping one mapper and one default set prevents transport-specific behavior drift.
type FakeBackend ¶
type FakeBackend struct {
// Performed records every action the gate admitted. Tests assert on this;
// the point of tier-terminal-refusal is that nothing lands here.
Performed []Action
// Launched records Launch calls.
Launched []string
// PerformHook runs before an action is recorded. It can mutate the fake
// (e.g. flip the frontmost app mid-batch, to prove the per-step gate fires)
// or return an error (e.g. ErrControlInterrupted).
PerformHook func(f *FakeBackend, act Action) error
// TreeHook lets a test mutate the tree between snapshots, to age a uid.
TreeHook func(f *FakeBackend, bundleID string)
// FrontmostErr, when set, makes Frontmost fail — used to prove that a locked
// screen or a user takeover reported there reaches the tool layer as its
// sentinel rather than a generic error the agent would retry.
FrontmostErr error
// contains filtered or unexported fields
}
FakeBackend is a scripted Backend: canned trees, a settable frontmost app, and a recording of every action that reached it.
It lives in the package rather than in a _test.go file because an explicit `jcode_eval` build injects it through SetFakeBackend to drive deterministic tool oracles with no TCC, GUI, or display. Production config has no path to this type. The containment claims in the design (tier refusal, batch abort, stale uid) are only worth making if they can be graded, and this grades them.
Mirrors browser.fakeBackend / scriptedTab (browser/session_test.go).
func (*FakeBackend) Actions ¶
func (f *FakeBackend) Actions() []Action
Actions returns a copy of the recorded actions.
func (*FakeBackend) CaptureVisual ¶
func (f *FakeBackend) CaptureVisual(_ context.Context, bundleID string) (Screenshot, error)
func (*FakeBackend) Close ¶
func (f *FakeBackend) Close() error
func (*FakeBackend) Closed ¶
func (f *FakeBackend) Closed() bool
Closed reports whether Close was called — used to assert that Session.Close does not close the Manager-owned backend.
func (*FakeBackend) Kind ¶
func (f *FakeBackend) Kind() string
func (*FakeBackend) ReadClipboard ¶
func (f *FakeBackend) ReadClipboard(context.Context) (string, error)
func (*FakeBackend) SetApps ¶
func (f *FakeBackend) SetApps(apps ...App)
SetApps replaces the installed-app list.
func (*FakeBackend) SetClipboard ¶
func (f *FakeBackend) SetClipboard(text string)
SetClipboard sets the fake clipboard's contents.
func (*FakeBackend) SetFrontmost ¶
func (f *FakeBackend) SetFrontmost(a App)
SetFrontmost sets the focused app. Callable from PerformHook to simulate a focus change mid-batch.
func (*FakeBackend) SetJournal ¶
func (f *FakeBackend) SetJournal(path string)
SetJournal makes the fake append every admitted action to path as JSONL.
func (*FakeBackend) SetShot ¶
func (f *FakeBackend) SetShot(bundleID string, png []byte)
SetShot sets an app's canned PNG bytes.
func (*FakeBackend) SetTree ¶
func (f *FakeBackend) SetTree(bundleID string, nodes []uitree.Node)
SetTree sets an app's canned accessibility tree.
func (*FakeBackend) SetVisualShot ¶
func (f *FakeBackend) SetVisualShot(bundleID string, shot Screenshot)
SetVisualShot sets a canned PNG and its global-window coordinate mapping.
type HelperPermissions ¶
type HelperPermissions struct {
Accessibility PermissionState
ScreenRecording PermissionState
}
HelperPermissions is the pair of macOS grants needed by the native helper. Accessibility covers AX inspection and input; ScreenRecording covers window pixels. The states are snapshots and can be refreshed with helperBackend.RefreshPermissionStatus.
type HelperStatus ¶
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager owns Backends for the process lifetime. Sessions borrow one and never close it. (browser/manager.go makes the same split: backends are expensive to start — a Chrome launch, a daemon handshake — and are reused across tasks.)
func NewManager ¶
NewManager creates the process-wide manager.
func (*Manager) Enabled ¶
Enabled reports whether computer use is on. It defaults off, unlike browser-use: computer use can touch anything on the machine.
func (*Manager) OpenScreenshot ¶
OpenScreenshot validates and opens an immutable screenshot while holding the cross-process store lock. Returning the already-open file closes the old validate-path-then-ReadFile race: later pruning may unlink its name, but it cannot change the bytes referenced by this handle.
func (*Manager) OpenSession ¶
OpenSession returns a task-scoped Session bound to a Backend.
Production always uses the native macOS helper. Tests and eval builds may explicitly inject a deterministic Backend with SetFakeBackend; the deprecated Config.Backend value never participates in this choice.
func (*Manager) Preapproved ¶
Preapproved reads the live, mutex-protected policy used by settings hot reload. Approval callbacks can run concurrently with an HTTP config update; reading the shared config.Config pointer directly would race and could retain a stale always-allow decision.
func (*Manager) RequestPermissions ¶
func (m *Manager) RequestPermissions(ctx context.Context, accessibility, screenRecording bool) (HelperPermissions, error)
RequestPermissions surfaces the macOS consent prompt for the named grants (Settings → Computer Use → Request permission and /computer grant both ride this). It starts the helper if needed and deliberately does NOT require computer use to be enabled: the grants are a prerequisite for enabling the feature, so gating the request on enablement would deadlock the first run.
The returned states are what the helper observes immediately after asking. The system dialog is answered later, so "denied" means "not granted yet" — callers should re-poll Status rather than treat it as a refusal.
func (*Manager) SaveScreenshot ¶
SaveScreenshot writes a PNG and returns its opaque id.
func (*Manager) SetConfig ¶
SetConfig hot-swaps the configuration (the settings endpoint calls this, so no restart is needed).
func (*Manager) SetFakeBackend ¶
SetFakeBackend installs a scripted backend explicitly. Used only by tests and eval wiring; persisted user configuration cannot reach this path.
func (*Manager) TierOverrides ¶
TierOverrides builds the validated per-app tier map from config.
An override may only tighten. A config row trying to loosen a terminal to "full" is dropped with the built-in tier left in place: loosening is a deliberate act the settings UI gates behind a warning, and a hand-edited config file is not that gate. An unparseable tier is likewise dropped rather than defaulted, so a typo cannot silently weaken containment.
type NotAllowedError ¶
type NotAllowedError struct{ BundleID, AppName string }
NotAllowedError reports an app absent from the session allowlist.
func (*NotAllowedError) Error ¶
func (e *NotAllowedError) Error() string
type PermissionState ¶
type PermissionState string
PermissionState is the native helper's non-prompting view of a macOS TCC grant. Unknown is intentionally distinct from denied: an older daemon omits the additive pong fields, and treating that absence as granted would make the settings UI claim Computer Use is ready without evidence.
const ( PermissionUnknown PermissionState = "unknown" PermissionDenied PermissionState = "denied" PermissionGranted PermissionState = "granted" )
type Screenshot ¶
Screenshot is a window-scoped visual observation. Bounds are global macOS screen coordinates; PixelWidth/PixelHeight describe the attached PNG after downscaling. Together they let a model map a pixel in custom-drawn UI back to the coordinate fallback accepted by computer_act.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session is the task-lifetime state of computer use: the app allowlist granted for this task, the grant flags, and the snapshot generations that uids are minted against.
It does not own the Backend — the Manager does. Session.Close never closes it. (browser/session.go:54-58 makes the same split for the same reason: backends are expensive to start and are reused across tasks.)
func (*Session) Act ¶
Act performs one or more actions. Every step is independently gated.
Stops on the first error and reports how far it got. There is no continue-on-error: a sequence whose step 3 failed has an unknown UI state at step 4, and pressing on is how a click lands somewhere unintended.
func (*Session) Apps ¶
Apps lists apps with their grant state and tier.
The names are tainted: an app can be named anything, including "Ignore previous instructions.app". They are wrapped in an explicit data boundary so a model reading the list is told, in band, not to obey it.
func (*Session) BackendKind ¶
BackendKind reports which backend is serving this session.
func (*Session) FrontmostBundle ¶
FrontmostBundle returns the bundle id of the focused app, or "" if it cannot be determined. Feeds the approval layer, which needs the live app identity because a click carries no bundle id in its args — exactly the reason browser-use reads the origin from the live session rather than from args.
func (*Session) Grant ¶
Grant adds apps to the session allowlist. Called after the user approves an access request; never from model args directly.
func (*Session) Open ¶
Open launches or focuses an app and grants it for this session.
Approval of computer_open *is* the grant. The call is gated upstream by the approval layer (class "launch", per-app), so by the time control reaches here the user has said yes to this specific app. The session allowlist is therefore not a second mechanism to keep in sync with approvals — it is the record of what was approved, and Open is the only thing that writes to it.
Note what this does *not* grant: the clipboard and system-key flags stay off. Approving "control Notes" is not approving "read my clipboard".
func (*Session) Read ¶
Read returns text the agent asked for. kind=clipboard is the only kind so far.
The clipboard is gated by its own grant, never by an app grant: approving "control Notes" is not approving "read whatever I last copied", and what users last copied is very often a password. The approval layer additionally refuses to ever pre-approve this call (see decideComputer), so it prompts every time even under a blanket always_allow.
func (*Session) Screenshot ¶
Screenshot captures an app's windows.
func (*Session) ScreenshotVisual ¶
ScreenshotVisual captures the current app window plus the coordinate mapping required for custom-drawn UI that has no actionable AX node.
func (*Session) SetTierOverrides ¶
SetTierOverrides installs validated per-app tier overrides from config.
func (*Session) Snapshot ¶
func (s *Session) Snapshot(ctx context.Context, bundleID, filter string, maxLines int, disableDiff bool) (string, error)
Snapshot returns uid-annotated accessibility text for an app.
By default it returns a diff against the previous snapshot of the same app: a menu-open changes a handful of nodes out of hundreds, and paying full-tree tokens for that is how a 256K window disappears. disableDiff forces the full tree.
Diffing is client-side here, unlike codex (whose service holds session state and diffs server-side). Ours is worse in principle but portable: it works identically for injected test backends and the stateful native helper.
func (*Session) TierFor ¶
TierFor resolves the effective tier for an app: the built-in table, unless config tightened it.
An override may only *tighten*. A config that tries to loosen a terminal to "full" is ignored here; loosening is a deliberate per-app action that the settings UI gates behind a warning and records as an explicit override, and it is applied by the caller building the override map — not by a silently permissive lookup.
type Status ¶
type Status struct {
Enabled bool `json:"enabled"`
Backend string `json:"backend"`
BackendKind string `json:"backend_kind"`
// Available is true when a backend can actually serve a session.
Available bool `json:"available"`
// Blocker names the first shut gate: "disabled", "no_helper",
// "permissions", or "" when nothing is blocking.
Blocker string `json:"blocker"`
// Detail is a human-readable explanation of Blocker.
Detail string `json:"detail,omitempty"`
MaxBatch int `json:"max_batch"`
// Tiers exposes the built-in tier table for the apps the UI has rows for, so
// the settings page never has to reimplement the rules.
Tiers map[string]string `json:"tiers,omitempty"`
// Grant flags, so the settings switches can render their real state rather
// than guessing. Without these the UI shows them off on mount even when
// config has them on, and the next save silently revokes them.
ClipboardRead bool `json:"clipboard_read"`
ClipboardWrite bool `json:"clipboard_write"`
SystemKeyCombos bool `json:"system_key_combos"`
// Helper reports installation/connection separately from TCC. A binary on
// disk is not a ready backend: Settings actively handshakes before Connected
// becomes true.
Helper HelperStatus `json:"helper"`
AccessibilityPermission PermissionState `json:"accessibility"`
ScreenRecordingPermission PermissionState `json:"screen_recording"`
}
Status is what the settings UI needs to tell the user which gate is shut.
Three independent things must be true before computer use works: it must be enabled, a backend must exist, and macOS must have granted permission. When it looks broken it is almost always one of those, and a UI that cannot say which one is the difference between a two-second fix and an abandoned feature. So each gate reports separately, and Blocker names the first one that is shut.
type Tier ¶
type Tier int
Tier bounds what may be done to an app, independently of whether the user allowlisted it. The allowlist answers "may the agent touch this app at all"; the tier answers "and how far".
This exists because of one fact about jcode: it runs inside a terminal. An agent that can type into that terminal can run `rm -rf`, read ~/.jcode/config.json (live API keys), or drive a second jcode — routing around jcode's entire approval system by going through the GUI instead of the `execute` tool. An approval layer the agent can walk around is decorative.
See internal-doc/computer-use-design.md §4.2.
const ( // TierRead permits observation only: snapshot and screenshot. TierRead Tier = iota // TierClick additionally permits pointing: click, hover, scroll. Enough to // press a Run button or scroll test output; not enough to enter text. TierClick // TierFull permits everything, including text entry and key combinations. TierFull )
func DefaultTier ¶
DefaultTier resolves the built-in tier for a bundle id.
Unknown apps get TierFull. That is the honest default: deny-by-default on an unknown bundle id breaks every third-party app and trains users to reflexively override, which is worse than the thing it prevents. The containment for an unknown app is the allowlist (it cannot be touched until the user names and approves it), not the tier.
func ParseTier ¶
ParseTier maps a config string to a Tier. Unknown values return (TierFull, false) so callers can reject rather than silently mis-apply a typo: a typo'd tier must never quietly become a *weaker* restriction than intended.
type TierError ¶
TierError reports an action refused by the tier gate. It carries enough for the tool layer to explain the refusal usefully rather than just failing.
type VisualCaptureBackend ¶
type VisualCaptureBackend interface {
CaptureVisual(ctx context.Context, bundleID string) (Screenshot, error)
}
VisualCaptureBackend is an optional richer capture contract. Backends that only implement Capture remain valid; native helpers implement this interface so screenshot coordinates are explicit rather than guessed.