textshot

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

README

go-textshot

Text and UI elements from screenshots, in Go. Runs the PP-OCRv6 detection and recognition models, and on demand the OmniParser icon_detect_v3 element detector, through ONNX Runtime; the runtime library and the models are downloaded on first use, pinned by SHA-256, and cached. No system packages.

eng, err := textshot.New(ctx, textshot.Options{})
if err != nil { ... }           // errors.Is(err, textshot.ErrUnavailable): no OCR on this build/host
defer eng.Close()

results, err := eng.DetectText(ctx, img)   // []TextResult{Box, Text, Score}, reading order
for _, line := range textshot.Lines(results) {
    fmt.Println(line)
}

elements, err := eng.DetectElements(ctx, img)   // []Element{ID, Key, Kind, Box, Score, Text}
for _, e := range elements {
    fmt.Println(e.ID, e.Key, e.Box)             // 32 button:Continue (1160,554)-(1243,575)
}

DetectElements fuses OCR with the element detector: an interactable region with text inside is a button (push button, menu item, list row, link), one without is an icon (toolbar icon, window control, checkbox), and text outside any interactable is text. Each element carries two identifiers: ID, its index in reading order for that frame (a set-of-marks number), and Key, a role+name style locator (button:Continue, text:Disk Utility, icon@700,310, #2 on repeats) that stays the same across frames while the layout does not move. Find an element by Key, confirm it by Box, then act. DetectInteractables runs only the detector, and MergeElements does the fusion for callers that already have OCR results. The element model (232 MB) is downloaded and loaded on the first elements call, so engines that only read text never pay for it.

go install github.com/devcell-sh/go-textshot/cmd/textshot@latest
textshot screenshot.png            # one text line per output line
textshot -json screenshot.png      # JSON array of elements: id, key, kind, box, score, text
textshot -prefetch                 # warm the cache (runtime + all models) without running

Options

Option Env Default
CacheDir TEXTSHOT_CACHE_DIR os.UserCacheDir()/textshot
LibraryPath TEXTSHOT_ONNXRUNTIME_LIB downloaded into the cache
MaxSideLen 1920 (downscale only)
Threads runtime.GOMAXPROCS(0)
MinScore 0.5
ElementMaxSideLen 1280 (downscale only)
ElementMinScore 0.1
Logger discard

Platforms: linux/amd64, linux/arm64, darwin/arm64 (ONNX Runtime 1.30.0), darwin/amd64 (1.23.2, the last Intel macOS release). Other platforms can point LibraryPath at a local libonnxruntime.

Per 1080p frame on two CPUs: OCR about 650 ms at full resolution, 430 ms at MaxSideLen: 1280; element detection about 4 s at ElementMaxSideLen: 1280 (the YOLOv9-E detector is heavy, so call it on demand rather than in a polling loop). The element session tries the platform GPU execution provider (CoreML on macOS, CUDA elsewhere) and falls back to the CPU when the loaded runtime does not have it.

Development

task test           # hermetic
task test:e2e       # TEXTSHOT_E2E=1, downloads runtime + models once
task bench
task models:export  # re-export icon_detect_v3 to ONNX (needs uv; one-off, see below)
task models:check   # verify dist/models/*.sha256 are pinned in models.go

The element model ships upstream only as TorchScript. task models:export converts it with a pinned PyTorch environment (tools/export-icon-detect, driven by uv) and writes dist/models/icon_detect_v3.onnx plus its SHA-256. Pushing a models-v* tag runs the same export in CI (.github/workflows/release.yaml), checks the digest against models.go, and publishes the file as a release asset, which is the URL ElementModel downloads from.

See AGENTS.md for the package map, pinned artifacts and cache layout.

License

Apache-2.0. The embedded ppocrv6_dict.txt and the PP-OCRv6 models are Apache-2.0 from PaddlePaddle/PaddleOCR. The pipeline is a port of lib-x/ppocr-v6-go. The element detector is icon_detect_v3 from microsoft/OmniParser-v2.0, MIT (the YOLOv9-E checkpoint; the older YOLOv8 icon_detect is AGPL and is not used).

Documentation

Overview

Package textshot extracts text lines and UI elements from screenshots.

It runs the PP-OCRv6 detection and recognition models, and on demand the OmniParser icon_detect_v3 element detector, through ONNX Runtime. The runtime shared library and the models are fetched on first use into a cache directory, verified against pinned SHA-256 digests, and reused afterwards. No system packages are required beyond a C compiler at build time (the ONNX Runtime binding is a thin cgo shim that dlopens the library).

Typical use: 

eng, err := textshot.New(ctx, textshot.Options{})
...
defer eng.Close()
results, err := eng.DetectText(ctx, img)
for _, line := range textshot.Lines(results) { ... }

elements, err := eng.DetectElements(ctx, img)   // buttons, icons and text with boxes

Index

Constants

View Source 
const (
	// EnvCacheDir overrides the cache directory.
	EnvCacheDir = "TEXTSHOT_CACHE_DIR"
	// EnvLibraryPath points at a libonnxruntime shared library (file or the
	// directory containing it) to use instead of downloading one.
	EnvLibraryPath = "TEXTSHOT_ONNXRUNTIME_LIB"
)

Environment variables honoured when the matching Options field is empty.

Variables

View Source 
var (
	// PP-OCRv6 text detection and recognition (Apache-2.0, official PaddlePaddle exports).
	DetModel = hfModel("PaddlePaddle/PP-OCRv6_small_det_onnx", "28fe5895c24fd108c19eb3e8479f4ab385fbfc62", "inference.onnx",
		"d73e0058b7a8086bbd57f3d10b8bcd4ff95363f67e06e2762b5e814fe9c9410e", 9880512)
	RecModel = hfModel("PaddlePaddle/PP-OCRv6_small_rec_onnx", "b8f84f0b80c529de40b4fbb3544b84fa7233a513", "inference.onnx",
		"5435fd747c9e0efe15a96d0b378d5bd157e9492ed8fd80edf08f30d02fa24634", 21159378)

	// OmniParser icon_detect_v3 (MIT, YOLOv9-E interactable-region detector),
	// exported to ONNX by tools/export-icon-detect and published by the release workflow.
	ElementModel = releaseModel("models-v1", "icon_detect_v3.onnx",
		"a34d6ff3bd7448d4dad6db70df0cdd80b42440f5b53eab2d4469dc6694fc004a", 232306145)
)
View Source 
var ErrUnavailable = errors.New("textshot: OCR unavailable")

ErrUnavailable is returned (wrapped) when OCR cannot run on this build or platform: the binary was built without cgo, or no ONNX Runtime release exists for the host OS/architecture.

Functions

func DefaultCacheDir 

func DefaultCacheDir() string

DefaultCacheDir is os.UserCacheDir()/textshot (for example ~/.cache/textshot on Linux, ~/Library/Caches/textshot on macOS), falling back to the temp directory when the user cache dir is unknown.

func Lines 

func Lines(rs []TextResult) []string

Lines returns the text of each result, in order.

func Prefetch 

func Prefetch(ctx context.Context, o Options) error

Prefetch downloads the runtime library and the OCR models into the cache without loading them. It needs no cgo, so it can warm a cache from any build. The element model is fetched separately by PrefetchElements.

func PrefetchElements 

func PrefetchElements(ctx context.Context, o Options) error

PrefetchElements downloads the element detection model (232 MB) into the cache without loading it.

Types

type Element 

type Element struct {
	// ID is the element's 0-based index in reading order within the list
	// it was returned in (a set-of-marks number). It is only meaningful
	// for that frame.
	ID int
	// Key names the element the way a role+name locator would:
	// "button:Continue", "text:Disk Utility", "icon@700,310" (top-left
	// corner for elements without text), with "#2", "#3" appended to
	// repeats in reading order. Keys are deterministic for a given frame
	// and stay the same across frames while the layout does not move, so
	// callers can find an element again by Key and confirm it by Box.
	Key  string
	Kind Kind
	// Box is the element's bounding box in the source image's coordinates.
	Box image.Rectangle
	// Score is the detector's confidence for interactables and the
	// recognition confidence for text, in [0,1].
	Score float32
	// Text is the label inside the element (reading order, space-joined)
	// or the text line itself; empty for icons.
	Text string
}

Element is one UI element found on a screenshot.

func MergeElements 

func MergeElements(interactables []Element, lines []TextResult) []Element

MergeElements labels interactables with the text lines inside them and returns interactables and leftover lines as one list in reading order, with fresh IDs and Keys. A line belongs to the smallest interactable that covers at least half of it and is not much taller than it, so panels do not swallow their rows; an interactable with text becomes KindButton, one without stays KindIcon, and unassigned lines become KindText.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a loaded OCR pipeline. It is safe for concurrent use.

func New 

func New(ctx context.Context, o Options) (*Engine, error)

New prepares an Engine: it resolves or downloads the ONNX Runtime library and the models, then loads them. The first call on a machine downloads roughly 40 MB; later calls reuse the cache. Errors wrapping ErrUnavailable mean OCR cannot work on this build or host at all.

func (*Engine) Close 

func (e *Engine) Close() error

Close releases the model sessions. The ONNX Runtime library itself stays loaded for the life of the process.

func (*Engine) DetectElements 

func (e *Engine) DetectElements(ctx context.Context, img image.Image) ([]Element, error)

DetectElements runs OCR and the element detector on img and merges them into one typed list in reading order: buttons (interactables with their label), icons (interactables without text) and free text.

func (*Engine) DetectInteractables 

func (e *Engine) DetectInteractables(ctx context.Context, img image.Image) ([]Element, error)

DetectInteractables finds the interactable regions (buttons, icons, menu items, fields) in img with the OmniParser icon_detect_v3 model, in reading order. Every result has KindIcon and no Text; see MergeElements. The first call downloads the model when it is not cached and loads it.

func (*Engine) DetectText 

func (e *Engine) DetectText(ctx context.Context, img image.Image) ([]TextResult, error)

DetectText finds and recognizes every text line in img, returned in reading order (top to bottom, left to right).

type Kind 

type Kind string

Kind classifies an Element. 

const (
	// KindButton is an interactable region with a text label inside it:
	// a push button, menu item, list row, link, labelled field.
	KindButton Kind = "button"
	// KindIcon is an interactable region without text: a toolbar icon,
	// window control, checkbox, unlabelled field.
	KindIcon Kind = "icon"
	// KindText is a text line that is not part of any interactable region:
	// a title, description, status message.
	KindText Kind = "text"
)

type Model 

type Model struct {
	Name   string // file name under <cache>/models
	URL    string
	SHA256 string
	Size   int64
}

Model is a pinned ONNX model download.

func (Model) CachePath 

func (m Model) CachePath(cacheDir string) string

CachePath is where the model lives under cacheDir.

type Options 

type Options struct {
	// CacheDir holds the downloaded runtime and models. Defaults to
	// $TEXTSHOT_CACHE_DIR, then os.UserCacheDir()/textshot.
	CacheDir string
	// LibraryPath is an explicit libonnxruntime shared library (file or the
	// directory containing it). Defaults to $TEXTSHOT_ONNXRUNTIME_LIB, then
	// the cached download. When set, nothing is downloaded for the runtime
	// and a missing file is an error.
	LibraryPath string
	// MaxSideLen caps the longer image side fed to the detector (default
	// 1920). Images are only downscaled, never upscaled. Lower values are
	// faster but shrink small UI text.
	MaxSideLen int
	// Threads is the intra-op thread count for inference (default:
	// runtime.GOMAXPROCS(0), which respects container CPU limits).
	Threads int
	// MinScore drops lines whose recognition confidence is below it
	// (default 0.5; negative keeps everything). Icon glyphs and decorations
	// typically decode as low-confidence junk.
	MinScore float32
	// ElementMaxSideLen caps the longer image side fed to the element
	// detector (default 1280, downscale only). The detector is far heavier
	// than OCR: on two CPUs a 1080p frame takes ~5 s at 1280 and ~12 s at
	// 1920, while 640 misses most UI controls.
	ElementMaxSideLen int
	// ElementMinScore drops detected elements whose confidence is below it
	// (default 0.1). Icons on busy screens often score 0.1-0.3.
	ElementMinScore float32
	// Logger receives download progress and diagnostics; nil discards.
	Logger *slog.Logger
}

Options configures New. The zero value downloads everything into the default cache and uses all CPUs.

type RuntimeAsset 

type RuntimeAsset struct {
	Version string
	Name    string // tarball file name in the GitHub release
	SHA256  string
}

RuntimeAsset is a pinned ONNX Runtime release tarball for one platform.

func RuntimeAssetFor 

func RuntimeAssetFor(goos, goarch string) (RuntimeAsset, bool)

RuntimeAssetFor returns the pinned ONNX Runtime release for a platform.

func (RuntimeAsset) URL 

func (a RuntimeAsset) URL() string

URL is the GitHub release download URL.

type TextResult 

type TextResult struct {
	// Box is the line's bounding box in the source image's coordinates.
	Box image.Rectangle
	// Text is the recognized string.
	Text string
	// Score is the mean per-character confidence in [0,1].
	Score float32
}

TextResult is one recognized line.

Source Files

View all Source files

Directories

Path Synopsis
cmd
textshot command
Command textshot prints the text lines or the UI elements found in screenshots.
 Command textshot prints the text lines or the UI elements found in screenshots.
internal
fetch
Package fetch downloads files atomically with optional SHA-256 verification and coarse progress logging.
 Package fetch downloads files atomically with optional SHA-256 verification and coarse progress logging.
imgutil
Package imgutil holds the pure-Go image helpers shared by the model pipelines: RGBA conversion, bilinear resizing and cropping.
 Package imgutil holds the pure-Go image helpers shared by the model pipelines: RGBA conversion, bilinear resizing and cropping.
ppocr
Package ppocr implements the PP-OCRv6 text detection and recognition pipeline on top of an abstract model runner, so the image math is pure Go and testable without ONNX Runtime.
 Package ppocr implements the PP-OCRv6 text detection and recognition pipeline on top of an abstract model runner, so the image math is pure Go and testable without ONNX Runtime.
testutil
Package testutil holds helpers for e2e tests.
 Package testutil holds helpers for e2e tests.
uidet
Package uidet finds interactable UI elements (buttons, icons, menu items, fields) with the OmniParser icon_detect_v3 model, a single-class YOLOv9-E detector, on top of an abstract model runner so the image math is pure Go and testable without ONNX Runtime.
 Package uidet finds interactable UI elements (buttons, icons, menu items, fields) with the OmniParser icon_detect_v3 model, a single-class YOLOv9-E detector, on top of an abstract model runner so the image math is pure Go and testable without ONNX Runtime.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.