vision

Documentation

Overview

Package vision implements the Pando Desktop Controller's vision-fallback path: when a region of the screen exposes no usable accessibility semantics (a canvas app, a remote desktop window, a game, an app with a broken accessibility implementation), the agent still needs a way to act. The model itself is the vision engine -- this package never inspects pixels or does any image analysis. Its job is to make the screenshot -> model -> coordinates -> physical input loop safe and explicit:

• DrawGrid overlays a light coordinate grid on a screenshot so a vision-capable model can estimate pixel coordinates more accurately (internal/llm/tools/desktop_screenshot.go's optional "grid" param).
• ValidateCoordinates checks a proposed (x,y) against real display bounds before any physical input is sent (internal/uiauto.Manager. ClickAt).

Every action performed through this path must be marked source="vision" in the tool's structured response (see internal/llm/tools/desktop_click_at.go), so the agent and the user can always tell a semantic (accessibility-tree) action from a blind, guessed-coordinate one. Guardrails (permission prompt, Options.AllowPhysicalInput gating) live in internal/uiauto.Manager, which is the only caller of this package's coordinate helpers.

Index

• Constants
• func DrawGrid(img image.Image, opts GridOptions) image.Image
• func ValidateCoordinates(x, y int, bounds []core.Bounds) error
• type GridOptions

Constants

const DefaultGridStep = 100

DefaultGridStep is the pixel spacing DrawGrid uses between grid lines (in both axes) when GridOptions.Step is <= 0.

Functions

func DrawGrid

func DrawGrid(img image.Image, opts GridOptions) image.Image

DrawGrid returns a copy of img with a light coordinate grid and pixel-coordinate axis labels overlaid every Step pixels, to help a vision-capable model translate what it sees into the (x,y) arguments desktop_click_at expects. img itself is never mutated. The overlay is drawn in real (unscaled) image coordinates, so labels always read the same pixel coordinates a subsequent desktop_click_at call should use -- draw it before any resize step in the screenshot pipeline.

func ValidateCoordinates

func ValidateCoordinates(x, y int, bounds []core.Bounds) error

ValidateCoordinates checks that (x,y) falls within at least one of bounds (typically the set of currently active display bounds). An empty bounds slice means the caller could not determine real display bounds (e.g. the screen package could not enumerate displays); validation is then skipped -- it is on the caller to decide whether that is acceptable rather than silently rejecting every coordinate action. Coordinates outside every given Bounds return an INVALID_ARGS core.DesktopError.

Types

type GridOptions

type GridOptions struct {
	// Step is the pixel spacing between grid lines, in both axes. <= 0
	// uses DefaultGridStep.
	Step int
	// LineColor overrides the default grid line color (a semi-transparent
	// red is used when unset, i.e. the zero color.RGBA).
	LineColor color.RGBA
	// LabelColor overrides the default axis label color (yellow when
	// unset).
	LabelColor color.RGBA
}

GridOptions controls DrawGrid.