README
¶
gosearch/browser — optional real-browser engine
Drives an unmodified Chromium-family browser over CDP (via
chromedp) to run searches and
extract page content where plain HTTP cannot: pages that only render behind
JavaScript, and search endpoints that answer non-JS clients with a JS
challenge.
This is a separate Go module. The core gosearch module does not import
it, and installing the core never pulls this in:
go get github.com/BugraAkdemir/gosearch # core only (zero extra deps)
Installing the browser module:
# Until the first prefixed tag is published, pull the tip of main:
go get github.com/BugraAkdemir/gosearch/browser@main
# After a browser/vX.Y.Z tag exists (multi-module repos require prefixed tags):
go get github.com/BugraAkdemir/gosearch/browser
Usage
package main
import (
"context"
"fmt"
"log"
browser "github.com/BugraAkdemir/gosearch/browser"
)
func main() {
ctx := context.Background()
e, err := browser.New(ctx) // discovers Chrome/Edge/Chromium on the system
if err != nil {
log.Fatal(err)
}
defer e.Close()
results, err := e.Search(ctx, "facebook") // rendered DOM, post-JS
if err != nil {
log.Fatal(err)
}
for _, r := range results {
fmt.Println(r.Title, r.URL)
}
page, err := e.Fetch(ctx, "https://example.com/") // same shape as gosearch.Fetch
if err != nil {
log.Fatal(err)
}
fmt.Println(page.Title, len(page.Content))
}
One Engine = one long-lived browser process with one reused tab. Keep it
alive for the lifetime of your program; steady-state memory stays at roughly
one page's worth (~100–250 MB idle + page), not a fresh browser per request.
Images are disabled and the GPU process is off by default to keep consumption
down.
Deployment helpers: browser.Install(ctx, opts...) performs the same
discovery-or-download without building an Engine — call it in a Dockerfile
RUN step or setup script so the one-time download never lands on a live
user request. browser.WithCacheDir(path) points storage somewhere writable
when the default user-cache location is read-only (locked-down containers).
When Google serves its /sorry CAPTCHA
Search reports exactly where the tab landed when it fails, e.g.
landed on: https://www.google.com/sorry/... — that is Google's
IP-reputation CAPTCHA, which no tool here will auto-solve. The legitimate
escape hatch is you solving it once, in a persistent profile:
// 1) once, headed — click the CAPTCHA yourself, then close:
e, _ := browser.New(ctx,
browser.AllowDownload(true),
browser.WithProfileDir("/home/me/.gosearch-profile"),
browser.WithHeadless(false),
)
e.Search(ctx, "warm up") // solve by hand in the window; Ctrl+C after
// 2) from now on, headless with the same profile reuses that session:
e2, _ := browser.New(ctx,
browser.AllowDownload(true),
browser.WithProfileDir("/home/me/.gosearch-profile"),
)
defer e2.Close()
Related knobs: WithUserAgent(ua) overrides the declared identity string —
by default the engine declares a standard desktop Chrome UA (the same
realistic-identity policy as the core HTTP client) instead of
chrome-headless-shell's HeadlessChrome; webdriver flags and fingerprints
stay untouched. Searches also warm up on the homepage first and use
Google's default locale/result-count (no bot-ish num=20), because ordinary
flow is precisely what keeps the wall away.
Where the executable comes from
Resolution order in New:
browser.WithExecutable(path)— force a specific binary.- Embedded archive — if this module was built with
-tags gosearch_embed_engine, the binary carries chrome-headless-shell inside and self-extracts to the OS cache on first use. Produce the archive first:
The build fails without the archive present — an engine-less "embedded" binary can never ship by accident. Expect roughly +110 MB compressed / +250 MB extracted binary size.cd browser go run ./tools/fetch-engine -out engine/chrome-headless-shell.zip go build -tags gosearch_embed_engine ... - System discovery — stable-channel names first (
google-chrome-stable,google-chrome,chromium,chromium-browser,microsoft-edge,chrome-headless-shell) plus standard install paths per OS. - Opt-in download — with
browser.AllowDownload(true), fetches chrome-headless-shell from Google's official chrome-for-testing CDN into<user-cache>/gosearch/browser/<version>/. Never downloads without that explicit flag; otherwise returnsErrNoBrowserFoundnaming every path probed.
Honest limitations
The browser is driven unmodified: no stealth patches, no
navigator.webdriver masking, no fingerprint spoofing. Consequences:
- It clears JavaScript-gated pages (e.g. Google's enablejs wall).
- It does not defeat IP-reputation blocks or interactive CAPTCHAs, and headless/automation signals remain detectable. From a datacenter IP, expect challenges regardless of engine.
- Search extraction is heuristic (h3-in-anchor titles, container text as snippet) because Google A/B-tests its rendered DOM without notice.
- Yandex is not targeted here: its SmartCaptcha is interactive by design; plain-HTTP results from a trusted IP remain the supported path for it.
When a page renders but no recognizable results appear, Search wraps
gosearch.ErrChallenge; genuinely empty extraction surfaces as
gosearch.ErrNoResults — same sentinel vocabulary as the core module.
Tests
go test -race ./... # offline/deterministic, never launches a browser
go test -race -tags integration ./... # live: skips itself if no browser is installed
Documentation
¶
Overview ¶
Package browser is the OPTIONAL real-browser engine for gosearch: it drives an unmodified Chromium-family browser (Chrome, Edge, Chromium, or Google's chrome-headless-shell) over CDP to run searches and extract page content where plain HTTP cannot — most notably pages that only render or unlock behind JavaScript.
It is a separate Go module on purpose: depending on it pulls in chromedp and, potentially, a ~100–300 MB browser runtime. The core gosearch module stays dependency-light; nothing here is imported unless you ask for it.
Honest limitations (the line this project will not cross): the browser is driven UNMODIFIED — no stealth patches, no navigator.webdriver masking, no fingerprint spoofing. The only identity adjustment is a standard desktop Chrome User-Agent string (same policy as the core HTTP client); webdriver stays on and everything else stays stock. That means the engine clears JavaScript-gated pages but does NOT defeat IP-reputation blocks or interactive CAPTCHAs. When an engine still refuses to serve results, this package reports ErrChallenge/ErrBlocked-wrapped errors like the rest of gosearch instead of pretending otherwise.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrDownloadDisabled = errors.New("browser: download disabled and no executable available")
ErrDownloadDisabled is returned when the caller explicitly opted OUT of downloading (AllowDownload(false)) but no system browser was found either, or when an embed-mode build was requested but the engine archive is absent.
var ErrNoBrowserFound = errors.New("browser: no chromium-family executable found")
ErrNoBrowserFound is returned by New when no usable Chromium-family executable was discovered on the system, none was supplied via WithExecutable, and downloads are not enabled. The error text names every location that was probed so the fix (install Chrome/Chromium/Edge, pass a path, or enable downloads) is obvious.
Functions ¶
func CurrentPlatform ¶
func CurrentPlatform() string
CurrentPlatform names the chrome-for-testing platform slug for this host (linux64, mac-arm64, mac-x64, win64), or "" when unsupported.
func DownloadFile ¶
DownloadFile streams url into dstFile atomically (.part then rename).
func Install ¶
Install performs the same discovery-or-download resolution as New without constructing an Engine: use it during deployment or image building (Dockerfile RUN step, setup script) so the one-time engine download is not paid by a live user request later. It is safe to call repeatedly — an already-present executable or cached engine short-circuits to nil.
Types ¶
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is one long-lived, lazily started browser instance shared across calls: a single process with a single tab keeps steady-state memory at roughly one page's worth instead of paying full startup per request.
func New ¶
New resolves which executable to drive (explicit path > embedded archive > system discovery > opt-in download) and returns an Engine. The browser process starts lazily on first use, not in New, so constructing one never pays startup cost. Call Close when done.
func (*Engine) Close ¶
Close shuts the browser down. Throwaway profile directories are removed; user-supplied persistent ones (WithProfileDir) are kept — that is the whole point of them. Idempotent.
func (*Engine) Executable ¶
Executable reports the resolved chromium-family binary path. Valid before first use because resolution happens in New.
func (*Engine) Fetch ¶
Fetch retrieves url in the shared tab, waits for JavaScript to render it, and extracts the readable main content into a gosearch.Page — a drop-in swap for gosearch.Fetch for callers whose target pages need JS.
The extractor prefers <article>/<main>, falls back to the text-heaviest container heuristic, strips script/style/nav/header/footer/aside/form noise, and caps output at ~20k characters. It is deliberately simpler than the core module's DOM-based readability extractor: rendered innerText has already discarded most chrome, so heavy scoring buys little here.
func (*Engine) Search ¶
Search renders Google for query in the shared tab and extracts results from the DOM AFTER JavaScript has run — the case plain HTTP cannot handle.
The heuristic is best-effort exactly like the core providers: every anchor containing an <h3> is treated as a result candidate, titles come from the h3, snippets from the surrounding result container's text with the title line removed. Non-http(s) destinations, Google-internal plumbing (search pagination, accounts, support, /url wrappers) and duplicates are skipped. When no h3 ever appears (consent wall, captcha, unusual layout), Search returns an error wrapping gosearch.ErrChallenge rather than empty results, so callers can distinguish "no answers" from "engine refused".
type Option ¶
type Option func(*engineConfig)
Option configures an Engine. Options are applied in order.
func AllowDownload ¶
AllowDownload permits New to download chrome-headless-shell — a small, official, automation-only build of Chromium published on Google's chrome-for-testing CDN — into the OS user cache directory the first time it is needed. Nothing is ever downloaded without this explicit opt-in.
func WithCacheDir ¶
WithCacheDir overrides where the downloaded or embedded engine is stored and extracted (default: the OS user cache directory, e.g. ~/.cache/gosearch/browser on Linux). Useful when the default location is read-only, as in some locked-down containers.
func WithExecutable ¶
WithExecutable bypasses all discovery and uses the given chromium-family executable (Chrome, Chromium, Edge, or chrome-headless-shell) as-is.
func WithHeadless ¶
WithHeadless controls whether the browser runs headlessly (default true, i.e. headless). WithHeadless(false) opens a visible window — useful once, with WithProfileDir, to manually clear an interactive challenge so the saved session carries over to later headless runs.
func WithProfileDir ¶
WithProfileDir uses a persistent browser profile at path: cookies survive Close() and across runs. The directory is created if missing and is NOT deleted by Close.
func WithUserAgent ¶
WithUserAgent sets the exact User-Agent string the browser declares. By default chrome-headless-shell advertises "HeadlessChrome", which search engines treat as an automation-only build and answer differently; the default here is a standard desktop Chrome UA instead — the same realistic-identity policy the core HTTP client already applies. Nothing else about the browser is altered: webdriver stays on, fingerprints stay stock, CAPTCHAs are never solved.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
headed-once
command
Command headed-once opens a VISIBLE browser window with a persistent profile so YOU can manually clear an interactive challenge (CAPTCHA / consent) one time; the saved session then carries over to headless runs of examples/search using the same profile directory.
|
Command headed-once opens a VISIBLE browser window with a persistent profile so YOU can manually clear an interactive challenge (CAPTCHA / consent) one time; the saved session then carries over to headless runs of examples/search using the same profile directory. |
|
search
command
Command search runs a headless Google search through the browser engine and prints results.
|
Command search runs a headless Google search through the browser engine and prints results. |
|
tools
|
|
|
fetch-engine
command
Command fetch-engine downloads the stable chrome-headless-shell archive for the current platform into engine/chrome-headless-shell.zip so a build with -tags gosearch_embed_engine can embed it.
|
Command fetch-engine downloads the stable chrome-headless-shell archive for the current platform into engine/chrome-headless-shell.zip so a build with -tags gosearch_embed_engine can embed it. |