browser

package
v0.0.0-...-2835ce2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 19, 2026 License: MIT Imports: 55 Imported by: 0

README

browser - a pure-Go headless browser

browser is a headless browser for scraping and automation, written entirely in Go, in the spirit of the Zig Lightpanda browser but built only from portable parts.

The point of the port is portability. Lightpanda embeds V8, BoringSSL, curl, brotli, nghttp2, sqlite and PCRE2, so it needs a Zig toolchain and per-platform native builds. This package uses only pure-Go dependencies, so it cross-compiles anywhere Go does, including Android/Termux on arm64.

How it is built

Concern Choice Why
HTML5 parse + serialize golang.org/x/net/html conformant, pure Go
CSS selectors github.com/andybalholm/cascadia querySelector/querySelectorAll
JavaScript github.com/dop251/goja ECMAScript engine, pure Go
Network requests reuses the impersonating TLS/HTTP transport
DOM the x/net/html node tree itself small, and the parsed nodes are the ones scripts mutate

The DOM is the *html.Node tree. There is no separate wrapper model, so html.Parse, script mutation, CSS matching and serialization all operate on the same nodes. Compiled CSS selectors are cached process-wide, because pages issue thousands of querySelector calls and compilation dominates the cost.

Network requests go through requests, so the document, external scripts, fetch() and XMLHttpRequest all carry the selected browser's TLS/JA3, HTTP/2 fingerprint and default headers.

Usage

One page takes one call. The package re-exports the impersonation targets, so a caller who only drives the browser needs a single import, and importing it dotted needs no prefix either:

import (
    "fmt"

    . "github.com/HashShin/shade/browser"
)

// Get gives the page a Browser of its own, which Close releases.
p, err := Get("https://quotes.toscrape.com/js/", Chrome131)
if err != nil {
    panic(err)
}
defer p.Close()

fmt.Println(p.Title())
fmt.Println(p.Text())     // the rendered text, not the raw HTML
fmt.Println(p.Markdown()) // the same document as Markdown
for _, l := range p.Links() {
    fmt.Println(l.Href, l.Text)
}

The rest of this page writes the package name out, browser.New and browser.Page, because it is documenting that package rather than a caller. Either spelling works; see the root README for the dotted form.

Lower-level control, for a page you already have or one you want to drive:

off := false
b := browser.New(browser.Options{RunScripts: &off})
p := b.NewPage("https://example.com/")
_ = p.SetContent(html, "https://example.com/") // no network
_ = p.Load("https://example.com/other")
v, _ := p.Eval("document.querySelectorAll('.quote').length")
fmt.Println(v.ToInteger()) // goja.Value

png, err := p.Screenshot(browser.ScreenshotOptions{Width: 1280})
if err != nil {
    panic(err)
}
_ = os.WriteFile("page.png", png, 0o644)

Page.Eval and Page.CallOn hand back a goja.Value, so a caller that evaluates JavaScript imports github.com/dop251/goja for the result type.

CLI:

make build
./bin/shade get https://react.dev/ --render -f markdown -i chrome131
./bin/shade get https://quotes.toscrape.com/js/ --render --eval 'document.title'
./bin/shade open https://example.com/ -f text --status

open is shorthand for get --render. Every flag is listed in ../docs/cli.md, or run shade help open.

What works

  • CSS: <style> blocks and <link rel=stylesheet> sheets are fetched and cascaded (specificity, !important, source order, inheritance, @import, @media, custom properties, a user-agent default sheet), and drive both getComputedStyle and the screenshot renderer. Loading is lazy: loading a page or extracting text never touches stylesheets; the first style-dependent operation fetches them once and caches them.

  • The page's own stylesheets, and nothing else: every <style> element and every <link rel=stylesheet> the document contains, in document order, with relative and root-relative (/assets/main.min.<hash>.css) hrefs resolved against the page, whatever integrity or crossorigin attributes they carry. A sheet added by a script is picked up too, because the style caches are invalidated when the DOM changes. Page.StyleSheets and --sheets report each one with its size and rule count, or the reason it was not applied:

    ./bin/shade get --render https://brave.com/ --sheets
applied      https://brave.com/static-assets/css/main.min.25ad059cc...css  2332 rules, 415249 bytes
applied      https://brave.com/static-assets/css/fonts-latin.min.e...css     2 rules, 16677 bytes
applied      inline <style>                                                  1 rules, 69 bytes

    The report goes to stderr and does not stop the command, so it composes: get brave.com --screenshot page.png --width 900 --sheets writes the PNG and lists the sheets it was rendered with.

  • document.styleSheets (with href, media, ownerNode and cssRules, including each rule's selectorText and style.getPropertyValue), so a page can inspect its own CSS. Reading a sheet's URL never fetches it; reading its rules does.

  • CSS custom properties and var(--name, fallback), including inheritance and nested references. Modern sites theme everything this way: Wikipedia's body colour is color: var(--color-base, #202122) and Bootstrap 5 styles every component with --bs-*, so without var() such a page falls back to the user-agent defaults.

  • @media is evaluated against the width being rendered: media types, not/only, and/or chains, min-width/max-width (and the height family, orientation, the device-* aliases, and the (width >= 600px) range form). A query the parser cannot understand does not match, because treating it as matching applies every mobile rule to a desktop page.

  • Selectors written for real build tools: CSS escapes outside strings (".font-\[\'Poppins\'\2c sans\]", ".\!text-\[14px\]", ".lg\:font-semibold"), including class names with escapes cascadia compiles but never matches, which are matched by name instead. Without this a single escaped quote in brave.com's stylesheet made the parser swallow the remaining 117KB: 1110 rules instead of 2332.

  • The font shorthand in its real grammar, where the weight, style and variant precede the size and the size must carry a unit: font: 600 14px/22px system-ui sets weight 600, not a 600px font. This is how Leo (brave.com's design system) sets its buttons, through a custom property.

  • font-weight keeps its number, so font-semibold reports 600 and font-medium 500, and bolder/lighter step through the CSS weight table.

  • Blockification for flex and grid items as well as for floats and positioned boxes, so a <span> inside a flex container reports display: block.

  • User-agent form controls by type: a text field and textarea are white on black, a button and select keep the platform face (#efefef) and centre their label, a checkbox or radio is transparent.

  • HTML presentational attributes (bgcolor, color, align) act as author-origin hints below the cascade, which is how old table layouts paint themselves, and a page without a doctype is styled in quirks mode (tables take the medium font size instead of inheriting).

  • Document decoding: the Content-Type charset, a BOM or <meta charset> is honoured (via x/net/html/charset), so non-UTF-8 pages are not mojibake. A <base href> sets the base for relative URLs, used by script src, href/src properties, fetch, XMLHttpRequest, links and markdown.

  • A response that is not markup - text/plain, application/json, a binary type - is shown as its text inside a <pre>, the way a browser's plain-text viewer does, and document.contentType reports the served type. Nothing in such a body is parsed or run: a text/plain body containing <script> stays text, where parsing it as HTML would execute it. text/html, XHTML and XML keep the parsed-and-scripted path.

  • HTML5 parsing and a mutable DOM: createElement, appendChild, insertBefore, innerHTML/outerHTML, textContent, attributes, classList, dataset, style, cloneNode.

  • CSS queries: querySelector(All), getElementById, getElementsBy*, matches, closest.

  • Events: addEventListener/removeEventListener/dispatchEvent with bubbling, DOMContentLoaded and load lifecycle, click().

  • Timers: setTimeout, setInterval, requestAnimationFrame, drained after load (bounded by a 2s budget).

  • fetch() and XMLHttpRequest, both through the impersonating transport, returning Promises that resolve after a synchronous request.

  • Web platform globals: URL/URLSearchParams, TextEncoder/TextDecoder, AbortController/AbortSignal, Event/CustomEvent, IntersectionObserver (reports observed elements as intersecting after load, so lazy content materialises), MutationObserver/ResizeObserver (inert stubs), performance, crypto.getRandomValues/randomUUID, structuredClone, DOMParser, requestIdleCallback, customElements (registry stub).

  • document.implementation.createHTMLDocument/createDocument, Range (createContextualFragment), createEvent, importNode/adoptNode, document.fonts, attachShadow (returns the host). Sub-documents created through DOMParser or implementation keep their own tree: document methods operate on the receiver, not the page document.

  • <template> content is inert, as in a browser: parsed children live in a detached fragment, so they never appear in queries, Text(), Links() or Markdown(), while template.content exposes them for cloning and template.innerHTML reads and writes that content. Serialization (HTML(), outerHTML) still includes template contents.

  • TreeWalker/NodeIterator (createTreeWalker, NodeFilter constants) with working nextNode, previousNode, parentNode and sibling/first/last navigation, which is what text-extraction loops use.

  • window.postMessage (delivers a message event to window listeners), document.domain, document.open/close/hasFocus, getSelection, and webkitMatchesSelector-style aliases.

  • window, document (including document.currentScript, which bundlers use to resolve chunk paths), navigator, location, console, history, localStorage, matchMedia, atob/btoa.

  • Script-driven navigation: location.assign, location.replace, location.reload, assigning location.href, and assigning window.location all load the new document and swap in a fresh JavaScript environment. A relative URL is resolved against the current document. The load is deferred until the running script phase has finished, so the statement after the call still runs against the old document, and the chain is bounded so a page that redirects on every load cannot spin. History entries are not modelled, so assign and replace behave alike. This matters for handoff pages that finish a JavaScript challenge and then send the browser on with location.replace(...).

  • Classic scripts that use top-level await are retried wrapped in an async function, since some bundlers ship them as classic scripts.

  • Recursion is capped like a browser engine (Options.LoadTimeout and a ~10000-frame call stack), so a polyfill loop throws RangeError instead of spinning forever, and a slow subresource cannot stall the load past the budget.

  • Extraction helpers: HTML(), Text(), Links(), Markdown().

  • Element geometry: getBoundingClientRect, getClientRects, offsetWidth/ offsetHeight/offsetTop/offsetLeft, clientWidth/clientHeight and document.elementFromPoint answer from the layout, not zeroed stubs. The layout is computed at most once per document revision and reused, so a script that measures many elements pays for one layout.

  • XPath: Page.QueryXPath/QueryXPathFirst/XPath and document.evaluate with XPathResult (node iterators, snapshots, number, string and boolean types). Compiled expressions are cached process-wide.

  • Page actions: Click, ClickPoint, Type, Fill, Select, Check, Focus, Press and Scroll, dispatching the events a browser would. They are the shared layer the CLI, the CDP Input domain and the BiDi input module call, and clicks use the geometry centre. A click then means what it means in a browser: it follows the enclosing <a href> and submits the form a submit control belongs to, unless a handler calls preventDefault.

  • iframes: each <iframe> is loaded as a child page with its own document and JavaScript environment, sharing the browser's cookies. Page.Frames(), iframe.contentDocument/contentWindow and extraction across the frame tree (Text, Links, Markdown) all work. Nesting is bounded.

  • Network interception and proxy: Options.Intercept is offered every request (document, script, stylesheet, image, font, fetch, XHR) and may continue, block or fulfil it. Options.Proxy routes every request through a proxy.

  • robots.txt: Options.ObeyRobots fetches and caches each origin's robots.txt and skips a disallowed URL with a synthetic 403.

  • CORS (Options.CORS, off by default): cross-origin fetch/XHR carries an Origin header, runs a preflight when it is not simple, and rejects a response without a matching Access-Control-Allow-Origin.

  • ES modules: <script type="module">, static import/export (default, named and namespace forms), re-exports, side-effect imports, an import map for bare specifiers, and dynamic import(), all loaded and evaluated in dependency order.

  • A Chrome DevTools Protocol server (shade serve) and WebDriver BiDi on the same port, so Puppeteer, Playwright and chromedp can drive it.

  • An MCP (Model Context Protocol) tool server (shade mcp) over stdio or HTTP, exposing navigate, get_content, evaluate, screenshot, click, type and structured_data to an AI agent.

  • Structured data: Page.StructuredData() returns every JSON-LD object the document declares, and Page.Microdata() the itemscope/itemprop graph.

  • An ad and tracker blocker (Options.Adblock): well-known advertising and analytics hosts and paths are dropped, with the caller's Intercept hook still able to override it.

  • PDF output: Page.PDF renders the page and embeds the image in a one-page PDF, the same class of output the screenshot produces.

  • Canvas: getContext('2d') implements the drawing calls pages use (fills, strokes, text, paths, drawImage, clearRect) over an image.RGBA, and toDataURL returns a PNG. Charts and generated images come out drawable.

  • WebSocket: a real client over the same pure-Go library the CDP server uses. Incoming messages are delivered as message events while the page's task queue is drained, so a handler sees a reply sent in onopen.

  • Web Workers: new Worker(url) runs the script in its own goja runtime on its own goroutine, with postMessage both ways and importScripts.

  • IndexedDB: an in-memory indexedDB with open and an upgrade, object stores (keyPath, autoIncrement), put/add/get/getAll/delete/clear/count, simple indexes, openCursor, and transactions whose requests resolve on a later turn.

  • Bodies and streams: Blob, File, FileReader, FormData, Headers, Request, Response, ReadableStream, WritableStream and TransformStream are real implementations over byte slices, not just names. new Blob(['hi']).size is 2, Array.from(new FormData(...)) iterates, new FormData(form) collects a form's successful controls, a FormData holding a file is sent as multipart and one without is urlencoded, and fetch(...).body.getReader() drains the response.

  • MutationObserver actually fires: childList, attributes (with attributeOldValue), characterData and subtree are recorded at every DOM mutation point, batched, and delivered at the microtask checkpoint -- after a script, a timer callback, an event or an eval. disconnect and takeRecords behave, so a framework that mounts into a mutation callback works.

  • IntersectionObserver and ResizeObserver report each observed target once at the next checkpoint, with rectangles read from the real layout. Because there is no viewport to scroll, an observed target is reported as intersecting; that is what reveals lazy-loaded content.

  • Shadow DOM: attachShadow returns a real ShadowRoot (with innerHTML, querySelector, appendChild, getElementById), element.shadowRoot returns it for open roots and null for closed, and getRootNode() answers the shadow root a node lives in. Declarative shadow DOM (<template shadowrootmode="open">) is attached at parse time, so a server-rendered component is visible even with --no-js. Extraction reads through the shadow boundary -- Text, Markdown, Links, HTML and the Go query helpers include shadow content, substituting a <slot> with the host's light children so nothing is duplicated. The DOM's own document.querySelector still stops at the boundary, as the spec requires.

  • history.pushState/replaceState rewrite the document URL in place, so location.pathname and relative URL resolution follow an SPA route, and back/forward/go walk the entries those calls created and fire popstate. location reads through to the live URL rather than a copy taken at load.

  • Custom elements work. customElements.define upgrades the elements already in the document, every later createElement/parse/insert upgrades too, and constructor, connectedCallback, disconnectedCallback and attributeChangedCallback (driven by the static observedAttributes) all fire. get, upgrade and whenDefined behave. The element object the page receives is the constructed instance, so a component's own methods are callable on it, and an element the page already held keeps its identity across the upgrade. A constructor may call this.attachShadow(...): document.querySelector still stops at the boundary, but extraction reads through it, so a component's content reaches Text, Markdown and HTML. Two limits: callback timing is not modelled, so lifecycle and attribute callbacks run synchronously instead of being queued on a microtask, and only autonomous custom elements are supported (define(name, ctor, {extends:...}) for a customized built-in is ignored), with no :defined CSS.

  • AbortSignal is real. abort() updates aborted/reason, fires the abort event to addEventListener and onabort listeners, and propagates to signals derived with AbortSignal.any; AbortSignal.timeout schedules an abort on the page's own timer queue. A Request carries the controller's own signal, so request.signal.aborted follows controller.abort(), and fetch rejects an already-aborted signal before sending. A request in flight cannot be interrupted, because the loader is synchronous.

  • DOMParser honours its type argument. text/xml, application/xml, application/xhtml+xml and image/svg+xml parse as XML through encoding/xml and yield an XMLDocument whose documentElement is the XML root, with case preserved (tagName, localName) as XML requires; anything else parses as HTML. Malformed XML returns a <parsererror> document rather than throwing, as a browser does.

  • structuredClone is a real structured clone rather than a JSON round-trip: Map, Set, Date, RegExp, ArrayBuffer, typed arrays and cycles all survive, and a nested Map no longer comes back as a plain object.

  • The Service Worker surface is stubbed for feature detection (register resolves, controller is null); there is no persistent worker.

MCP server

shade mcp speaks MCP JSON-RPC 2.0 over stdio, and over HTTP with --port. Each HTTP session gets its own browsing context -- its own page, cookies and memory -- so several agents can share one server without clobbering each other. A client that initializes without an Mcp-Session-Id header is assigned one, and the id comes back in the response; send it on later requests to stay on that session, or send the same id from two clients to deliberately share one page. DELETE /mcp with the header closes a session, and the session_new, session_list and session_close tools manage them explicitly. Over stdio there is one implicit session, because the client owns the process.

Point an MCP client at it:

{
  "mcpServers": {
    "shade": { "command": "/path/to/shade", "args": ["mcp"] }
  }
}

The tools are navigate, get_content (text/markdown/html/links), evaluate, screenshot, click, type and structured_data. Each session keeps one current page; a tool that needs a page and has none says so rather than failing.

  • Screenshot, a text-layout PNG renderer (see below).

Verified live:

Site Result
quotes.toscrape.com/js/ all 10 JS-rendered quotes materialise
react.dev React hydrates and renders; no console errors
www.gocomics.com Next.js bundle runs; full page renders
www.foodnetwork.com 200, ~330 KB rendered
bsky.app shell renders (~408 chars); its main chunk uses for await and stops

Rendered through the default impersonation targets, via scripts/check_sites.sh -B:

Site Target Result
www.marriott.com custom 200, ~1.1 MB rendered
www.marriott.com chrome131 403
www.ritzcarlton.com any 200, ~330 KB
mstdn.social any 200, ~53 KB
ubiqueros.com any 200, ~10 KB

Marriott behaves the same way in the browser as it does with the plain HTTP client: custom is accepted where a plain Chrome fingerprint is challenged. That is expected, because the browser renders through the same impersonating transport, and it is a good check that the two stay consistent.

Speed against a real Chromium

scripts/bench_browser.sh loads the same pages with shade and with a Chromium CLI (--headless=new --dump-dom) and reports wall time and rendered bytes. Measured on Termux/arm64 against Chromium 149:

Page Chromium shade Chromium bytes shade bytes
about:blank ~0.95 s - 40 -
quotes.toscrape.com/js/ 3.7 / 2.6 s 2.0 / 1.9 s 8 987 9 005
react.dev 3.6 / 2.2 s 3.6 / 3.8 s 272 527 272 744
gocomics.com 26.1 / 8.0 s 8.7 / 8.2 s 3 078 797 1 575 337
foodnetwork.com 23.5 / 23.5 s 3.5 / 3.7 s 625 985 340 452

Reading it honestly:

  • shade is faster on every page tested, and several times faster on heavy pages, because there is no browser startup (~1 s before Chromium even starts loading) and no per-page process to launch.
  • On react.dev the two are level: that much JavaScript interpreted without a JIT costs about what Chromium spends starting up and compiling.
  • Chromium renders more bytes on heavy pages (roughly 2x on gocomics). It is a real browser: it runs more scripts to completion and normalizes the DOM. So shade being faster is partly "does less".

Caveats, so the numbers are not over-read:

  • Chromium needs a warmed --user-data-dir here; its first run with a cold profile hangs. The warmup is untimed, and timed runs hit a URL the profile has not cached, which is the closest match to shade's always-cold behaviour.
  • Chromium is bounded with --virtual-time-budget because --dump-dom never returns on pages that do not reach network idle. That budget is virtual, not wall-clock, so its time is a floor rather than a full time-to-interactive.
  • Chromium's launcher ignores SIGTERM, so the script bounds it with timeout -k.
bash scripts/bench_browser.sh -n 3
bash scripts/bench_browser.sh https://your-site.example/
SHADE_ARGS="--timer-budget 0s" bash scripts/bench_browser.sh

Screenshots

Page.Screenshot renders the page to a PNG, mirroring what Lightpanda's screenshot does: it applies the page's CSS first, then flows the document at a given width and draws headings, paragraphs, lists with markers, preformatted blocks, blockquotes, rules, and styled runs (bold, italic, monospace, links, underlines, inline colors), with flat block background colours and text-align. The page's own web fonts are used too: a run whose computed font-family matches an @font-face is drawn with the page's file, at the closest weight and slope, instead of an embedded one.

Verified against a live page rather than by eye: on quotes.toscrape.com/js/ the cascade reproduces that site's own CSS exactly - .quote span.text at 19.2px italic (font-size: large), .quote small.author at weight 700 in #3677E8, .quote with 30px bottom margin and 10px padding, and body in sans-serif. Applying it grows the render from 1100px to 1496px, which is that CSS's margin and padding taking effect.

Rendering model

A browser parses HTML into a DOM and CSS into a rule set, cascades the rules to a computed style per element, builds a box tree, lays it out (block and inline formatting contexts, replaced elements, margin collapsing) and paints. This renderer follows that order for the parts it implements: the cascade produces a computed style, the collector turns the DOM into an ordered list of blocks (element boxes, text runs and replaced boxes), the layout flows them at a width, and the painter rasterizes the draw list. getComputedStyle answers from the cascade, so what it reports is what a render uses.

The user-agent defaults follow the HTML rendering spec: em-relative heading sizes and margins, 1em paragraph/list/blockquote margins, 40px list indentation, monospace for pre/code, mark, s/del, sub/sup, form-control faces and so on, all overridable by author CSS through the cascade. Vertical margins do not stack: adjacent margins collapse to the larger of the two, and padding never collapses, as in a browser.

Checking the cascade against a browser

tools/cssdiff compares this package's computed styles with a real Chromium, element by element, on the same page: Chromium answers over CDP, the Go browser answers through shade get --render --eval, and the two are matched by DOM position. It is a separate module, so its one dependency (chromedp) never reaches the browser package.

cd tools/cssdiff && go mod download && go run . https://news.ycombinator.com/

Every cascade fix here was found with it. Current agreement, 7 properties (font-size, colour, background, weight, style, display, alignment) over every element both engines produce:

page elements compared agree
news.ycombinator.com 817 7/7 properties, 0 differences
quotes.toscrape.com/js/ 109 7/7 properties, 0 differences
brave.com 959 font-size 1.1%, colour 1.6%, weight 0.0%, style 0.0%, display 1.1%, alignment 0.0%, background 5.0%
en.wikipedia.org (Go article) 6703 font-size 0.3%, colour 0.0%, background 0.5%, weight 0.5%, style 0.0%, display 2.5%, alignment 0.3%

The remaining Wikipedia differences are mostly display: flex/flow-root style rules behind :is()/:has() selectors that cascadia cannot compile, and grid layout the renderer does not use.

png, err := p.Screenshot(browser.ScreenshotOptions{Width: 1280, Scale: 2})

Scale multiplies the image (2 is a HiDPI capture: the boxes and the glyphs both come out twice the pixels). FontScale instead multiplies only the text, the way a browser's text-only zoom does: every rendered font size and the line height that goes with it are scaled, while boxes, padding and margins keep the page's sizes and getComputedStyle still reports them. Text then wraps against the smaller boxes, exactly as it does in a browser zoomed that way. The CLI renders text at 2x by default, so a screenshot is readable at a glance; pass --font-scale 1 for the page's own sizes.

./bin/shade get --render https://quotes.toscrape.com/js/ --screenshot page.png --width 900

--debug reports what the styling actually did, which is the first thing to check when a page looks unstyled: every stylesheet it fetched and its size, any sheet it could not fetch, the rules each one produced, and the unsupported selector samples.

[browser] stylesheet: 125934 bytes from https://quotes.toscrape.com/static/bootstrap.min.css
[browser] stylesheet: 1368 bytes from https://quotes.toscrape.com/static/main.css
[browser] page stylesheets: 2 declared, 2 applied
[browser] stylesheet 0: 1883 rules, 345/2228 selectors unsupported
[browser] style engine: 5356 rules at width 900
[browser] page fonts: 8 @font-face rules

Only what the page itself declares is used: every <style> element and every <link rel=stylesheet>, in document order, fetched from the URL the page gives. Nothing is injected, and nothing is substituted - when a line above names bootstrap.min.css that is a sheet the site itself links. Every declaration is accounted for: applied with its size, or skipped with the reason (disabled, media that does not match, a <link> without href) or reported as a failed fetch.

Option Default Meaning
Width 1280 layout width in CSS px, clamped to 320..4096
Scale 1 output multiplier (2 gives a 2x image)
MaxHeight 20000 height cap; longer content is cropped

It is deliberately the same class of renderer as the Zig reference, with the same honest limits:

  • The page's pictures are drawn: <img> and srcset (the largest candidate), PNG, JPEG, GIF and WebP, scaled to their box with a high-quality filter. Bounded and optional: at most 48 images and 16MB per render, fetched six at a time, and NoImages / --no-images turns it off. An image with a CSS width or height is drawn at that size; otherwise its natural size, scaled down to the column. background-image and gradients are not painted, so a page that is mostly CSS art still renders as text. On brave.com a 900px render goes from 3s and 2,072 distinct colours without images to 9s and 97,802 with them, which is what real photographs look like.
  • Inline SVG icons are rasterized with a pure-Go renderer (oksvg + rasterx) and drawn at the element's CSS size, with currentColor following the computed text colour. Icons and images flow as inline replaced boxes, so one can sit next to text on a line rather than forcing a line break.
  • Native form controls are drawn, not left blank: checkbox and radio (with their checked state), range, and color swatch show a graphic, while text inputs, textareas, buttons, file and select show their value, placeholder or selected option inside a box. A control's uniform border, min-height and declared height are drawn, so an empty textarea is still the tall box the page asked for and a "height: 40px" field is 40px. A width: 100% field is as wide as its containing block, padding and box-sizing included, and a submit control is inline-block: it shrinks to its label and is rounded by its border-radius, like a <button>, unless it declares a width (.btn-block). Text is centered inside a button, and a column flex container's align-items: center centers its children.
  • Positioning: position: relative/sticky keeps the element in flow (its inset offsets shift it), and absolute/fixed take it out of flow. An absolute/fixed box is placed from its nearest positioned ancestor using left/top/right/bottom (and the inset shorthand), sized by its width or shrink-to-fit, and painted above the flow. z-index values are recorded but stacking is approximated by document order.
  • Still no float and no full containing-block chain: an absolute element with no positioned ancestor uses the page origin, and fixed is placed like absolute (a full-page screenshot has no scroll). CSS is otherwise cascaded for typography, colour, display, spacing, alignment, backgrounds and borders.
  • Element decoration: a block with a background-color, border or border-radius is drawn once as a rectangle (rounded when it has a radius, with anti-aliased corners) rather than per line, and nested boxes paint parent-first. An inline element has no box of its own, so its background is drawn behind each of its runs as their rounded rectangle, grown by its padding and with the horizontal padding added to the run's width: that is a tag pill. opacity scales the element's own colours, and box-shadow (the first shadow) draws as a soft rectangle behind the box. transform and filter are not drawn.
  • Block sizing: width, max-width, max-height, height (as a minimum) and box-sizing: border-box are applied, plus right margins and padding, and margin: 0 auto centers a sized block. max-height clips the overflowing lines, like overflow on a scroll box. A sizing context is inherited by the subtree, so max-width:1100px;margin:0 auto centers a whole page column, not just the element's own text.
  • display: flex rows are laid out: children sit side by side, sized from flex-grow, flex-basis (including calc(50% - 7px)) or their content, with gap, flex-wrap, justify-content, align-items and nested rows. A column flex container stacks its children (and honors align-items horizontally); display: grid still stacks, and a row is not a full flexbox: baseline alignment, margins on items and ordered/reverse directions are not modelled.
  • Lengths include clamp(), min() and max(), and the vw unit, resolved against the layout width. Vertical and left margins and padding are applied as flow space; padding-top/padding-bottom push the content down and add space after it, even inside a block whose first child is another block.
  • text-transform (uppercase, lowercase, capitalize) and letter-spacing are applied to the drawn text and its measured width. rgba() backgrounds and text are composited over what is behind them, so a translucent surface on a dark page stays dark instead of coming out near-white, and the PNG is opaque.
  • Selectors cascadia cannot compile are skipped, and the engine counts them: --debug prints how many rules each sheet produced and lists the first unsupported selectors. On a real site most of those are ::before/::after and vendor pseudo-elements, which carry no element styling, so the number overstates the loss. One of them is not a loss at all: an ::after clearfix (content plus clear: both, Bootstrap's .row:after) generates a box whose only visible job is to end the floats inside its element, so the clear is applied to that element and the next block starts below the floats instead of over them. :is(), :where() and :has() are not supported.
  • Web fonts are applied from the page's own @font-face rules, including font providers reached through <link> or @import: a matching run is drawn with the page's file rather than an embedded one, at the closest weight and slope. Only raw sfnt (TTF and OTF) is decoded. A face offered only as WOFF or WOFF2 is recorded and reported under --debug, then falls back to the embedded Go fonts (Go regular, Go bold and Go Mono). A webfont is used per glyph, not per run: a rune the page's face has no shape for is drawn from the embedded font, which is what a browser's own font fallback does. Raleway carries no "→", so the pager button on quotes.toscrape.com drew a missing-glyph box until this landed. Italic runs of an embedded font are sheared from the upright face rather than set in Go Italic, which is a serif design: a "sans-serif" page in italic came out with serifs and ran about 2% wider than the browser's own oblique, so text wrapped a word early. There is still no synthetic bolding.
  • The machine's own fonts are used before the embedded ones, because "the text does not look like the browser" is mostly a question of which face was drawn. The families are indexed from the usual font directories on first use - names only; a face is read and parsed the first time it is drawn with - and a run's whole font-family list is tried in order, so Georgia, "Times New Roman", Times, serif reaches the serif the machine actually has. The generic families stand for what a browser's own default settings resolve to: sans-serif is Liberation Sans (what fontconfig answers Arial with), serif is Liberation Serif and monospace is DejaVu Sans Mono. On this machine that took the quotes.toscrape render from every quote wrapping a word early to every card landing on the same pixel row as Chromium's. Where a machine has no fonts at all - a scratch container, a phone without font packages - the embedded Go fonts are still what is drawn, so the renderer keeps working with none installed.
  • Not pixel-identical to a browser: glyph metrics differ from the browser's own text shaping, and layout is still the text-flow model below.

Cost, measured on Termux/arm64: loading is unaffected, because font parsing is behind a sync.Once, faces are built lazily and nothing runs unless Screenshot is called. The system font directories are walked once, on the first run that needs a family the page does not provide, and only the name table of each file is read then: a face is parsed when it is first drawn with. A quotes.toscrape.com render at 900px adds only a few milliseconds; a tall 900x3500 page takes ~115 ms, most of it PNG encoding and pixel work rather than layout.

Measuring layout against a live page

Computed styles are half of it; where a box ends is the other half, and both engines have to be asked the same question.

  • Page.RenderOutline prints the lines of text, then the boxes the layout drew as box x= y= w= h= id=, carrying the same rectangle a getBoundingClientRect would. Box lines only cover elements that paint something, so give an element a background to see its rectangle.
  • Chromium answers through tools/cssdiff/probe -js EXPRESSION, which runs the expression in the page and prints the result.

For a page-level check, tools/cssdiff/accept.html is a fixture with a grid of named areas, a padded flex row, a float and wrapping text. Rendering it in both engines and comparing the PNGs row by row is the acceptance test:

./bin/shade get --render http://127.0.0.1:8000/accept.html --screenshot ours.png --width 800
cd tools/cssdiff && go run ./probe -url http://127.0.0.1:8000/accept.html -width 800 -out chrome.png -js '0'

As of this session the two agree pixel for pixel down to the paragraph beside the float; the only differing rows are the free text, where the embedded face is about 10% narrower than Chromium's DejaVu for lowercase (see "What works").

Two traps, both of which cost real time here:

  • A live page's total height drifts. bootstrap.com moves about 180 pixels between runs on its own, so comparing a sweep with one from an hour ago is not evidence. Compare the same pages in one session, or better, in one run: a change was once reverted on a 178-pixel "regression" that turned out to be the page moving, not the change.
  • A fixture that copies a live page must rewrite its stylesheet hrefs to absolute URLs, or neither engine loads any CSS and the comparison is meaningless. A fixture that measures an element's declared size must also put a block before it: as a container's first block, the container's own height lands on the element as a minimum and its declaration cannot be seen.

Driving it as a browser (CDP and WebDriver BiDi)

shade serve starts a Chrome DevTools Protocol server on --host/--port (default 127.0.0.1:9222), so an existing automation client drives this browser the way it drives Chrome:

./bin/shade serve --port 9222
# then, from Puppeteer:
#   puppeteer.connect({ browserWSEndpoint: "ws://127.0.0.1:9222" })

It serves the discovery endpoints (/json/version, /json/list, /json/new) and a browser-level WebSocket (/devtools/browser/<id>) plus a per-page one (/devtools/page/<id>). The implemented domains are Browser, Target, Page, Runtime, DOM, Network (enable/cookies) and Input, enough for navigation, Runtime.evaluate, a DOM query, an interaction and Page.captureScreenshot. A client that asks for a method outside that set gets a protocol error rather than a hang.

Page.addScriptToEvaluateOnNewDocument and Network.setUserAgentOverride are implemented, not accepted-and-ignored. An init script is evaluated in the page's JavaScript environment once the document exists but before any of the page's own scripts run, so an injected patch is in place before page code can read navigator; Page.removeScriptToEvaluateOnNewDocument revokes one from the next document on. This matters because Playwright's addInitScript and Puppeteer's evaluateOnNewDocument are built on that method, and a client whose setup script is silently dropped does not fail - it just runs unpatched.

WebDriver BiDi is served on the same port at /session, sharing the target and page layer: session.new/status, browsingContext.create/navigate/getTree/ close/captureScreenshot, script.evaluate/callFunction and input.performActions. Both protocols are always served together.

The server only costs anything when it runs. A plain Open and extract never touches it. The routes, the implemented domains and the MCP tools are documented in ../server/README.md.

Verified against a real client: scripts/check_puppeteer.sh starts the server and a local page, then runs scripts/check_puppeteer.mjs, which connects with puppeteer-core, opens a page, navigates, reads document.title, evaluates an expression and takes a screenshot. It prints:

{"title":"Puppeteer Check","h1":"hello cdp","screenshotBytes":4356}

Checking sites

scripts/check_sites.sh -B runs the same site/target matrix with the headless browser instead of the plain HTTP client, and shows the rendered size per cell. Comparing the two modes shows which sites actually need JavaScript:

bash scripts/check_sites.sh -B -i chrome131,custom
bash scripts/check_sites.sh -B -i chrome131 https://bsky.app/

Not implemented

This is a browsing core with a document renderer, not a full web rendering engine. Screenshot applies a large subset of CSS (see the limits above) but is not a browser, and the gaps below remain. Specifically absent:

  • The JavaScript engine has no async generators or for await (... of ...) (goja reports "Async generators are not supported yet"). Bundles that rely on them, such as the Bluesky web app's main chunk, stop at that point even though the page still loads. Async functions and top-level await work.
  • A frame's contentWindow is a thin window shape over the shared DOM nodes, because a frame runs in its own JavaScript runtime; a script cannot call a function the frame defined on its own window.
  • There is no native agent mode (an LLM driving the browser in-process); the MCP server lets an external agent drive it instead.
  • No WebAssembly (no engine). Service Workers are feature-detection only: there is no persistent, background worker.
  • Form submission from a script is not carried out. A click navigates a link and submits a form (see Page actions above), but HTMLFormElement.submit and requestSubmit are absent, and a form submitted by an element.click() from inside a running script is refused with a warning rather than queued - only a link's scripted click is queued, and it lands when the script phase ends.
  • <template> contents are moved out of the element at parse time, so appending a node directly to a template element puts it in the element (visible) rather than in content. Use template.content to build content.

Known approximation: when a container holds a single block child, the engine merges the container's padding onto that one block, so the parent and child can report the same rectangle from getBoundingClientRect; the parent's own height is still correct. Hit testing and a click's centre are unaffected in the common case (an element with its own content, or its own background box).

Sites whose content is gated by a heavy framework and many chained dynamic imports may not fully render, and absolute JS-engine parity with V8 is out of scope. For most server-sent-then-rendered pages, and for probing JS challenges, it is enough.

Notes

  • Scripts run in the same goroutine as Open. A runaway script is bounded by Options.JavaScriptTimeout (default 10s); the whole script-loading phase is bounded by Options.LoadTimeout (default 30s). After DOMContentLoaded and again after load the loader waits up to Options.TimerBudget (default 2s) for pending timers, so a timer-driven page can finish rendering.
  • Rendering is much slower than the plain HTTP client by design: it issues one request per script (a large site can be 30-40), executes them in a pure-Go interpreter with no JIT, and re-serializes the DOM. Use shade get when the HTML is server-rendered and shade get --render only when scripts are required: they are the same binary, so the fast path is a flag away, but linking the browser is what makes the binary about 37 MB instead of 18 MB.
  • All network traffic, including fetch, XMLHttpRequest and external scripts, goes through one shared session, so a browser-like flow works across hosts. Every request carries the impersonation target selected at New, not just the first one.
  • One Browser may be used from several goroutines: the session and its cookie jar are synchronized, so pages can be crawled in parallel. A single Page is not safe for concurrent use. The race detector is unavailable on android/arm64, so the Termux workflow covers this by construction plus a concurrency test (TestConcurrentPages) run repeatedly. Where the detector does work, make test-race runs the whole suite under it and is clean. It is worth running after touching the WebSocket or worker code: a reader goroutine used to schedule the close event onto the page's timer queue while the page was draining it.
  • The parity features are opt-in and lazy, so they do not slow the plain Open -> extract path: geometry, XPath and the actions only run when called (and a geometry layout is cached until the DOM changes); interception, proxy, robots and CORS only act when their option is set; serve only costs while it runs; iframes and ES modules only cost on pages that contain them.

Waiting for a page

A load runs scripts, fires DOMContentLoaded and load, then waits up to --timer-budget (default 2s) for pending timers. That covers most pages, and the waits below are for the rest:

  • --wait-until load (default) is the behaviour above.
  • --wait-until domcontentloaded stops right after the DOMContentLoaded event, before the timer budget and before child frames. A handler that renders on that event still runs; a page that needs a later turn does not. On a timer-driven page this roughly halves the load time, which is the point.
  • --wait-until networkidle0 keeps draining after load until nothing has been pending for 500ms. A page that holds a socket open legitimately never goes idle, so the timeout ends the wait without it being an error.
  • --wait-ms 500ms advances the page's clock, running timers as their turn comes; sleeping alone would not run the callback.
  • --wait-script 'window.__ready' evaluates the expression repeatedly until it is truthy, up to --wait-timeout. A syntax error fails immediately rather than polling until the timeout, since it will never become true.
  • --wait SELECTOR waits for an element, as before.

The same primitives are on Page: WaitForTime, WaitForScript, WaitForNetworkIdle and WaitForSelector.

CLI flags

Every flag is documented in ../docs/cli.md. The generated list is the source of truth:

shade help open     # the browser path
shade help get      # the fast path
shade help serve    # CDP + WebDriver BiDi
shade help mcp      # MCP

The flags most specific to this package, in one place:

Flag Effect
--wait-until MODE Stop the load after load (default), domcontentloaded or networkidle0.
--wait-ms D Keep running the page's clock for D, so timer callbacks fire.
--wait-script JS Poll an expression until it is truthy.
--wait SELECTOR Wait for an element before extracting.
--timer-budget D How long a load waits for pending timers. Lower is faster.
--screenshot FILE Render the page to a PNG. --width, --scale, --font-scale (2x by default), --max-height, --no-images control it.
--pdf FILE Render the page to a PDF.
--sheets Report the page's own stylesheets, and whether each was applied.
--block GLOB Block requests matching a glob. Repeatable.
--click, --type, --fill, --select Drive the page. Repeatable, and applied in the order given.

Documentation

Overview

Package browser is a pure-Go, dependency-light headless browser for scraping and automation.

It is a Go answer to the Zig Lightpanda browser, but deliberately built only from portable, pure-Go parts so it cross-compiles and runs everywhere Go does, including Android/Termux and musl systems:

  • HTML5 parsing + serialization: golang.org/x/net/html
  • CSS selectors: github.com/andybalholm/cascadia
  • JavaScript: github.com/dop251/goja (an ECMAScript engine in pure Go)
  • Network: shade/requests, so pages are fetched with real browser TLS/JA3 and HTTP/2 fingerprints.

The DOM is the x/net/html node tree itself. That keeps the port small and correct: the same nodes that were parsed are the ones scripts mutate.

Index

Constants

View Source 
const (
	Chrome100            = impersonate.Chrome100
	Chrome101            = impersonate.Chrome101
	Chrome104            = impersonate.Chrome104
	Chrome107            = impersonate.Chrome107
	Chrome110            = impersonate.Chrome110
	Chrome116            = impersonate.Chrome116
	Chrome119            = impersonate.Chrome119
	Chrome120            = impersonate.Chrome120
	Chrome123            = impersonate.Chrome123
	Chrome124            = impersonate.Chrome124
	Chrome131            = impersonate.Chrome131
	Chrome131Android     = impersonate.Chrome131Android
	Chrome133a           = impersonate.Chrome133a
	Chrome136            = impersonate.Chrome136
	Chrome142            = impersonate.Chrome142
	Chrome145            = impersonate.Chrome145
	Chrome146            = impersonate.Chrome146
	Chrome150            = impersonate.Chrome150
	Chrome99             = impersonate.Chrome99
	Chrome99Android      = impersonate.Chrome99Android
	Custom               = impersonate.Custom
	Edge101              = impersonate.Edge101
	Edge99               = impersonate.Edge99
	Firefox133           = impersonate.Firefox133
	Firefox135           = impersonate.Firefox135
	Firefox144           = impersonate.Firefox144
	Firefox147           = impersonate.Firefox147
	OkHTTP4Android       = impersonate.OkHTTP4Android
	Safari153            = impersonate.Safari153
	Safari155            = impersonate.Safari155
	Safari170            = impersonate.Safari170
	Safari172Ios         = impersonate.Safari172Ios
	Safari180            = impersonate.Safari180
	Safari180Ios         = impersonate.Safari180Ios
	Safari184            = impersonate.Safari184
	Safari184Ios         = impersonate.Safari184Ios
	Safari260            = impersonate.Safari260
	Safari2601           = impersonate.Safari2601
	Safari260Ios         = impersonate.Safari260Ios
	Tor145               = impersonate.Tor145
	DefaultChrome        = impersonate.DefaultChrome
	DefaultEdge          = impersonate.DefaultEdge
	DefaultSafari        = impersonate.DefaultSafari
	DefaultSafariIOS     = impersonate.DefaultSafariIOS
	DefaultSafariBeta    = impersonate.DefaultSafariBeta
	DefaultSafariIOSBeta = impersonate.DefaultSafariIOSBeta
	DefaultChromeAndroid = impersonate.DefaultChromeAndroid
	DefaultFirefox       = impersonate.DefaultFirefox
	DefaultTor           = impersonate.DefaultTor
)

Impersonation targets, re-exported from the impersonate package so that a caller who only drives the browser needs one import rather than two: 

import . "github.com/HashShin/shade/browser"

p, err := Get("https://quotes.toscrape.com/js/", Chrome131)

They are aliases of the constants in impersonate/targets.go, not a second copy of the values. A raw string works too, since Get takes a plain string, but a misspelt constant does not compile where a misspelt string only fails when the page is loaded.

Variables

View Source 
var WaitUntilNames = []string{"load", "domcontentloaded", "networkidle0"}

WaitUntilNames are the accepted Options.WaitUntil values.

Functions

func Attr 

func Attr(n *html.Node, key string) string

Attr returns an element attribute value.

func UserAgent 

func UserAgent() string

UserAgent returns the default User-Agent the browser reports when no impersonation preset overrides it.

Types

type Browser 

type Browser struct {
	// contains filtered or unexported fields
}

Browser is a reusable browsing context: it owns cookies and an impersonating HTTP session so multiple pages share state.

func New 

func New(opts Options) *Browser

New creates a Browser with the given options.

func (*Browser) Close 

func (b *Browser) Close()

Close releases the underlying HTTP session, unless it was handed in through Options.Session: that one belongs to its creator.

func (*Browser) Fork 

func (b *Browser) Fork() *Browser

Fork returns a new Browser with the same options and its own cookie jar, robots cache and page set. Two forks can hold independent logins, which is what gives each MCP HTTP session its own cookies and memory. They still share nothing else, so a fork is cheap: no request is made until it loads a page.

func (*Browser) NewPage 

func (b *Browser) NewPage(url string) *Page

NewPage creates a page without loading anything. Use SetContent or Load.

func (*Browser) Open 

func (b *Browser) Open(rawURL string) (*Page, error)

Open loads a URL, parses it, and runs its scripts. It is the main entry point for a one-shot page.

func (*Browser) Options 

func (b *Browser) Options() Options

Options returns a copy of the browser options.

type ConsoleEntry 

type ConsoleEntry struct {
	Level   string
	Message string
	Time    time.Time
}

ConsoleEntry is a captured console.* message. 

type Link struct {
	Text string
	Href string
}

Link is an anchor found in a rendered page.

type MicrodataItem 

type MicrodataItem struct {
	Type       string
	Properties map[string]string
}

MicrodataItem is one item in a microdata graph: an element with itemscope and its itemtype and named properties.

type Options 

type Options struct {
	// Impersonate selects the TLS/HTTP fingerprint target (see
	// shade/impersonate). Empty means the requests default.
	Impersonate string

	// UserAgent overrides navigator.userAgent and the request UA. When empty
	// the impersonation preset's UA is used.
	UserAgent string

	// Headers are added to every request.
	Headers map[string]string

	// Timeout per network request. Zero means no explicit timeout.
	Timeout time.Duration

	// MaxRedirects for subresource and document requests.
	MaxRedirects int

	// RunScripts controls JavaScript execution. Default true.
	RunScripts *bool

	// LoadImages is currently ignored (no image decoding); kept for API parity.
	LoadImages bool

	// JavaScriptTimeout bounds a single script evaluation. Default 10s.
	JavaScriptTimeout time.Duration

	// LoadTimeout bounds the whole script-loading phase (document scripts and
	// external bundles). Default 30s. A slow or hanging subresource is dropped
	// once the budget is spent, so a page always finishes loading.
	LoadTimeout time.Duration

	// TimerBudget bounds how long the loader waits for pending timers after
	// DOMContentLoaded and again after load. Default 2s each. Lower it to
	// finish sooner at the cost of missing content rendered by timers.
	TimerBudget time.Duration

	// Console receives console.* output. nil discards it.
	Console func(level, message string)

	// Insecure skips TLS verification when true.
	Insecure bool

	// Proxy routes every request (document, scripts, fetch, XHR) through the
	// given proxy URL, for example "http://127.0.0.1:8080". Empty means a
	// direct connection. It is passed straight to the transport, so socks5://
	// and authenticated http:// proxies work.
	Proxy string

	// LoadOptions are passed to every request a page makes, the document and
	// its subresources alike. They are how a fingerprint asked for on one
	// request reaches the whole load, and they are applied per request, so a
	// browser can share its session and still honour them.
	LoadOptions []requests.Option

	// Session is the requests session the browser fetches through, which is
	// what lets a page and a plain request share cookies: it is how
	// Request.Browser reuses the session its request was sent on. Nil creates
	// one, which Close then closes; a session set here belongs to its creator
	// and is left open. The options that shape the session itself -
	// Impersonate, Timeout, Insecure, MaxRedirects and Proxy - apply only when
	// the browser creates its own.
	Session *requests.Session

	// ObeyRobots makes the browser honour robots.txt: before any request it
	// fetches and caches the origin's robots.txt and skips a disallowed URL,
	// yielding a synthetic 403 that does not stop the load. Off by default,
	// so the fast path pays nothing.
	ObeyRobots bool

	// Intercept, when set, is offered every request before it is sent (document,
	// scripts, stylesheets, images, fonts, fetch and XHR). Returning nil lets
	// the request continue; returning Block drops it; returning Fulfill answers
	// it from the hook. The hook runs synchronously on the request path.
	Intercept func(*Request) *Response

	// CORS, when set, enforces cross-origin rules on fetch and XHR: a
	// cross-origin request carries an Origin header and a preflight when it is
	// not simple, and a response without a matching Access-Control-Allow-Origin
	// is rejected. Off by default, matching the original's experimental flag,
	// and off means fetch behaves as before.
	CORS bool

	// Adblock drops requests to well-known advertising and analytics hosts and
	// paths, so a page renders without the noise and loads faster. Off by
	// default. It runs after Intercept, so a hook can override it by
	// fulfilling a request the blocker would have dropped.
	Adblock bool

	// WaitUntil decides how far a load goes before Open returns:
	// "load" (the default) runs scripts, DOMContentLoaded, load and the timer
	// budget; "domcontentloaded" stops after DOMContentLoaded; "networkidle0"
	// additionally keeps draining until nothing is pending. An unknown value
	// means "load". See wait.go.
	WaitUntil string

	// Debug logs page-load phases to stderr.
	Debug bool
}

Options configure a Browser or a one-off Page.

type Page 

type Page struct {
	URL      string
	Referrer string
	// contains filtered or unexported fields
}

Page is a single loaded document plus its JavaScript environment.

func Get 

func Get(rawURL string, target ...string) (*Page, error)

Get opens one page in a Browser of its own, so a single page takes one call rather than a Browser and an Open: 

p, err := browser.Get("https://example.com/", impersonate.Chrome131)
if err != nil {
	panic(err)
}
defer p.Close()

The target is optional; without it the default impersonation is used. Close the page when done, which closes the Browser behind it. Use New and Open instead when several pages should share cookies, or when more than the impersonation target has to be set.

func Open 

func Open(req requests.RequestTypes, opts ...requests.Option) (*Page, error)

Open loads a request's URL in a browser of its own and returns the page object, so the same Request literal Browse takes can be read and driven: Browse answers with the rendered bytes, Open with the page they came from.

The page owns the browser, so Page.Close releases both. Only a GET of a URL has a browser path, which is the same limit Browse has.

func (*Page) CallOn 

func (p *Page) CallOn(decl string, this goja.Value, args []goja.Value) (goja.Value, error)

CallOn calls a function declaration with `this` bound to obj and the given arguments, in the page's JavaScript context. It is the primitive CDP's Runtime.callFunctionOn builds on.

func (*Page) Check 

func (p *Page) Check(sel string, on bool) error

Check sets a checkbox or radio's checked state and fires change.

func (*Page) Click 

func (p *Page) Click(sel string) error

Click dispatches a full mouse press and release on the first element matching the selector, then a click, as a browser does, and follows the click's own meaning: a link navigates and a submit control submits its form.

func (*Page) ClickNode 

func (p *Page) ClickNode(n *html.Node) error

ClickNode clicks an element node directly.

func (*Page) ClickPoint 

func (p *Page) ClickPoint(x, y float64) error

ClickPoint clicks whatever element is at the page coordinates.

func (*Page) Close 

func (p *Page) Close()

Close releases the Browser this page was loaded in. A page from Get owns its Browser, so this is the whole cleanup a one-shot needs. A page from New shares its Browser with the other pages in it, so this closes those too; use Browser.Close directly when that lifetime is the clearer one.

func (*Page) Console 

func (p *Page) Console() []ConsoleEntry

Console returns captured console output.

func (*Page) Document 

func (p *Page) Document() *html.Node

Document returns the root DOM node.

func (*Page) ElementFromPoint 

func (p *Page) ElementFromPoint(x, y float64) *html.Node

ElementFromPoint returns the deepest element whose box contains the point, which is what document.elementFromPoint and a CDP hit test need. It reads the same cached layout as ElementRect.

func (*Page) ElementRect 

func (p *Page) ElementRect(n *html.Node) Rect

ElementRect returns the element's border box in CSS pixels. An element the layout did not place (a text node, or one inside display:none) returns the zero rectangle, as a browser's getBoundingClientRect returns for a detached node's parent chain.

func (*Page) Eval 

func (p *Page) Eval(script string) (goja.Value, error)

Eval runs additional JavaScript in the page context.

func (*Page) Fill 

func (p *Page) Fill(sel, value string) error

Fill sets a control's value in one step and fires input and change, the way a paste or a programmatic fill does.

func (*Page) Focus 

func (p *Page) Focus(sel string) error

Focus dispatches the focus event on the element and records it as active.

func (*Page) FrameFor 

func (p *Page) FrameFor(el *html.Node) *Page

FrameFor returns the child page an <iframe> element loaded, or nil.

func (*Page) Frames 

func (p *Page) Frames() []*Page

Frames returns the child pages of this document, in document order.

func (*Page) GetElementByID 

func (p *Page) GetElementByID(idv string) *html.Node

GetElementByID returns the element with the given id, looking inside shadow roots as well.

func (*Page) HTML 

func (p *Page) HTML() string

HTML returns the serialized rendered DOM.

func (*Page) InitScripts 

func (p *Page) InitScripts() []string

InitScripts returns a copy of the scripts installed with SetInitScripts. 

func (p *Page) Links() []Link

Links returns every anchor with a resolvable href, including those inside child frames.

func (*Page) Load 

func (p *Page) Load(rawURL string) error

Load fetches a URL into an existing page and runs its scripts.

func (*Page) Log 

func (p *Page) Log(level, message string)

Log appends a message to the page console (used by scripts and the CLI).

func (*Page) Markdown 

func (p *Page) Markdown() string

Markdown renders the current DOM as markdown, useful for feeding an LLM. A child frame's content is appended in document order, so framed content is not lost the way a plain text extraction of the top document would lose it.

func (*Page) Microdata 

func (p *Page) Microdata() []MicrodataItem

Microdata extracts itemscope elements and their itemprop values, which is the older alternative to JSON-LD (schema.org's itemscope/itemprop markup).

func (*Page) NodeValue 

func (p *Page) NodeValue(n *html.Node) goja.Value

NodeValue wraps a DOM node as a JavaScript value in the page's context, so a CDP DOM.resolveNode can hand it to a client.

func (*Page) OuterHTMLOf 

func (p *Page) OuterHTMLOf(n *html.Node) string

OuterHTMLOf serializes a node, including the element itself.

func (*Page) PDF 

func (p *Page) PDF(opts ScreenshotOptions) ([]byte, error)

PDF renders the page and returns a PDF document containing it. The options are the same as Screenshot's; Width, Scale and MaxHeight control the layout and the image, and NoImages turns off drawing the page's pictures.

func (*Page) Press 

func (p *Page) Press(key string) error

Press dispatches a key press on the focused element, or the body when nothing is focused.

func (*Page) Query 

func (p *Page) Query(sel string) *html.Node

Query returns the first element matching a CSS selector. Query returns the first element matching a CSS selector. It reaches into shadow roots, which the DOM's own document.querySelector does not: for a scraper, content hidden behind a web component is content that must be reachable.

func (*Page) QueryAll 

func (p *Page) QueryAll(sel string) []*html.Node

QueryAll returns all elements matching a CSS selector, shadow roots included.

func (*Page) QuerySelectorAllFrom 

func (p *Page) QuerySelectorAllFrom(n *html.Node, sel string) []*html.Node

QuerySelectorAllFrom returns every descendant of n matching sel.

func (*Page) QuerySelectorFrom 

func (p *Page) QuerySelectorFrom(n *html.Node, sel string) *html.Node

QuerySelectorFrom returns the first descendant of n matching sel.

func (*Page) QueryXPath 

func (p *Page) QueryXPath(expr string) []*html.Node

QueryXPath evaluates an XPath expression against the document and returns the matched nodes. A compile error or a scalar result yields no nodes; use Page.XPath when the type of the result matters.

func (*Page) QueryXPathFirst 

func (p *Page) QueryXPathFirst(expr string, from *html.Node) *html.Node

QueryXPathFirst returns the first node matching expr, searching from `from` (the document when from is nil).

func (*Page) QueryXPathFrom 

func (p *Page) QueryXPathFrom(expr string, from *html.Node) []*html.Node

QueryXPathFrom evaluates expr with `from` as the context node, so a relative expression such as ".//a" searches only that subtree.

func (*Page) ReadyState 

func (p *Page) ReadyState() string

ReadyState returns the current document readyState.

func (*Page) RenderOutline 

func (p *Page) RenderOutline(opts ScreenshotOptions, limit int) ([]string, error)

RenderOutline lays the page out exactly as Screenshot would and returns one line per drawn block, without rasterizing anything. It is how a render is compared against a browser's geometry: Chromium answers with getBoundingClientRect (tools/cssdiff/geom) and this answers with the boxes this renderer used.

func (*Page) Response 

func (p *Page) Response() *requests.Response

Response returns the document response, if loaded over the network.

func (*Page) Runtime 

func (p *Page) Runtime() *goja.Runtime

Runtime returns the page's JavaScript runtime, or nil when scripts are disabled. It backs CDP's Runtime domain.

func (*Page) Screenshot 

func (p *Page) Screenshot(opts ScreenshotOptions) ([]byte, error)

Screenshot renders the page to a PNG.

The page's CSS is applied first: <style> blocks and <link rel=stylesheet> sheets are fetched, parsed and cascaded (specificity, !important, source order, inheritance, simple media queries), including the user-agent defaults. The renderer then flows the document at the requested width: block flow with margins, padding and borders, flex rows, grid columns, tables, absolute positioning, images and backgrounds, and draws the text runs that result.

It is still a document renderer, not a web renderer: there are no floats, no multi-column layout, and no transforms or animations. Layout, stylesheet loading, font parsing and rasterization all happen inside this call, so loading pages is unaffected.

func (*Page) Scroll 

func (p *Page) Scroll(sel string) error

Scroll dispatches a scroll event on the element; a full-page render has no scrolling viewport, so this only informs the page.

func (*Page) Select 

func (p *Page) Select(sel, value string) error

Select chooses an <option> by value (or by label when no value matches) and fires change.

func (*Page) SetContent 

func (p *Page) SetContent(source, url string) error

SetContent loads HTML from a string, running scripts unless disabled.

func (*Page) SetInitScripts 

func (p *Page) SetInitScripts(scripts []string)

SetInitScripts installs scripts evaluated on every document load before the page's own scripts run. The slice is copied, so the caller keeps ownership.

func (*Page) SetUserAgent 

func (p *Page) SetUserAgent(ua string)

SetUserAgent overrides navigator.userAgent and the User-Agent sent on subsequent requests. It is the hook behind CDP Network.setUserAgentOverride. An empty value is ignored so a misconfigured driver cannot blank the UA.

func (*Page) SetViewportWidth 

func (p *Page) SetViewportWidth(w float64)

SetViewportWidth sets the width media queries and the layout resolve against, which is what a CDP Emulation.setDeviceMetricsOverride changes.

func (*Page) ShadowRoots 

func (p *Page) ShadowRoots() []*shadowRoot

ShadowRoots returns every attached shadow root in document order, so a caller can inspect the trees extraction walked through.

func (*Page) StructuredData 

func (p *Page) StructuredData() []any

StructuredData returns every JSON-LD object the document declares, in document order, including those in child frames. A block that fails to parse is skipped rather than failing the whole call.

func (*Page) StyleSheets 

func (p *Page) StyleSheets() []StyleSheet

StyleSheets lists the stylesheets the page itself declares, in document order, fetching the linked ones the first time it is called. Nothing is injected, and nothing the page declares is left out: a sheet that failed or was skipped appears with Err set, so "the site's CSS did not load" can be told apart from "the page has no CSS to load".

func (*Page) StyleSheetsAt 

func (p *Page) StyleSheetsAt(width float64) []StyleSheet

StyleSheetsAt is StyleSheets counted at a given layout width, so a report can name the same width a screenshot renders at. The counts differ because @media is evaluated while parsing.

func (*Page) Text 

func (p *Page) Text() string

Text returns the visible text of the rendered DOM, skipping script and style content, with child frame text appended so framed content is not lost.

func (*Page) TextOf 

func (p *Page) TextOf(n *html.Node) string

TextOf returns the visible text of a node's subtree, which is how a matched XPath or selector node is turned into a string.

func (*Page) Title 

func (p *Page) Title() string

Title returns the document title, trimmed.

func (*Page) Type 

func (p *Page) Type(sel, text string) error

Type focuses the element and types text into it, firing the key and input events a user's keystrokes would.

func (*Page) UserAgent 

func (p *Page) UserAgent() string

UserAgent returns the User-Agent this page reports and sends.

func (*Page) WaitForNetworkIdle 

func (p *Page) WaitForNetworkIdle(quiet, timeout time.Duration)

WaitForNetworkIdle drains the page until nothing has been pending for the quiet period, or the timeout expires. It is --wait-until networkidle0. The timeout is not an error: a page holding a socket open legitimately never goes idle, and the caller still wants whatever it rendered.

func (*Page) WaitForScript 

func (p *Page) WaitForScript(script string, timeout time.Duration) error

WaitForScript evaluates script repeatedly until it returns a truthy value, which is how a page signals that it has finished rendering. It is --wait-script. The timeout bounds the wait; an error from the script is returned rather than retried, because a broken expression never becomes true.

func (*Page) WaitForSelector 

func (p *Page) WaitForSelector(sel string, timeout time.Duration) *html.Node

WaitForSelector polls until a selector matches or the timeout expires.

func (*Page) WaitForTime 

func (p *Page) WaitForTime(d time.Duration)

WaitForTime advances the page by at least d, running timers as their turn comes. It is --wait-ms: a page whose content appears on a setTimeout needs the clock moved, and sleeping alone would not run the callback.

func (*Page) XPath 

func (p *Page) XPath(expr string) (XPathResult, error)

XPath evaluates expr as a general expression, returning its typed result.

type Rect 

type Rect struct {
	X, Y, Width, Height float64
}

Rect is an element's border box in CSS pixels, measured from the page's top-left, the same rectangle getBoundingClientRect reports.

func (Rect) Bottom 

func (r Rect) Bottom() float64

Bottom is the box's bottom edge.

func (Rect) Right 

func (r Rect) Right() float64

Right is the box's right edge.

type Request 

type Request struct {
	URL          string
	Method       string
	ResourceType string // document, script, stylesheet, image, font, fetch, xhr, other
	Headers      map[string]string
}

Request is offered to an Intercept hook before it is sent. Its fields are a copy, so a hook may modify the returned request through the pointer it receives but cannot change the page's own state.

type Response 

type Response struct {
	Status  int
	Headers map[string]string
	Body    []byte
	// contains filtered or unexported fields
}

Response is what an Intercept hook returns: nil to continue to the network, Block to drop the request, or Fulfill to answer without the network.

func Block 

func Block() *Response

Block drops a request. The caller sees a synthetic 403 with no body, and the load continues rather than failing.

func Fulfill 

func Fulfill(status int, contentType string, body []byte) *Response

Fulfill answers a request from the caller instead of the network.

type ScreenshotOptions 

type ScreenshotOptions struct {
	// Width is the layout width in CSS px. Default 1280, clamped to 320..4096.
	Width int

	// Scale multiplies the output size (2 gives a 2x image). Default 1.
	Scale float64

	// FontScale multiplies every rendered font size, and the line height that
	// goes with it, the way a browser's text-only zoom does. Boxes, padding
	// and margins keep the page's own sizes, and getComputedStyle still
	// reports them, so this changes only how large the text is drawn.
	// Default 1, the page's own sizes.
	FontScale float64

	// MaxHeight bounds the output height in CSS px, so a very long page cannot
	// produce a huge image. Default 20000. The content is cropped.
	MaxHeight int

	// NoImages turns off drawing the page's pictures, which is the only part
	// of a render that fetches anything beyond the page's own stylesheets.
	// Images are on by default, bounded by MaxImages and MaxImageBytes.
	NoImages bool

	// MaxImages caps how many images are fetched for one render. Default 48.
	MaxImages int

	// MaxImageBytes caps the total image bytes kept for drawing. Default 16MB.
	MaxImageBytes int
}

ScreenshotOptions configures Page.Screenshot.

type StyleSheet 

type StyleSheet struct {
	// Href is the absolute URL, empty for an inline <style> element.
	Href string
	// Media is the media attribute, or the <style> element's.
	Media string
	// Inline is true for a <style> element.
	Inline bool
	// Bytes is the size of the CSS text.
	Bytes int
	// Rules is the number of rules parsed from it. Media queries are evaluated
	// while parsing, so this number is only meaningful with Width.
	Rules int
	// Fonts is the number of @font-face rules in the sheet. They are not style
	// rules, so a font provider's sheet can be all fonts and no rules.
	Fonts int
	// Width is the layout width the rules were counted at.
	Width float64
	// Err is why the sheet is not applied, empty when it is.
	Err string
}

StyleSheet describes one stylesheet the document declares. It is the Go answer to document.styleSheets, and the way to check that a page's own CSS was loaded rather than guessed at.

type XPathResult 

type XPathResult struct {
	Nodes   []*html.Node
	Number  float64
	String  string
	Boolean bool
	Kind    string // "nodes", "number", "string", "boolean"
}

XPathResult is the outcome of evaluating an expression: a node set, a number, a string or a boolean.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.