gosearch

README

gosearch

Web search and page-content extraction for Go — no API key, no SDK, no account.

gosearch queries real search engines (DuckDuckGo, Bing, Google, Yandex) by fetching their public HTML result pages and parsing them directly, and extracts the readable main content of any URL. It is built for local-first programs — LLM agents, CLI tools, self-hosted services — that cannot or will not depend on a hosted search API.

results, _ := gosearch.Search(ctx, "golang html parser", gosearch.DuckDuckGo)

page, _ := gosearch.Fetch(ctx, results[0].URL,
    gosearch.WithMarkdown(), // headings, lists, links, code fences preserved
)
fmt.Println(page.Content) // ready to feed an LLM

Contents

• Why
• Features
• Quick start
• Usage patterns
• Choosing an engine
• Options
• Reliability, honestly
• Explicit non-goals
• Real-browser rendering (gosearch/browser)
• Versioning & stability
• Documentation
• Contributing
• License

Why

Most "web search for my agent/tool" solutions assume you'll pay for and depend on a hosted search API (SerpAPI, Google Custom Search, Bing Search API…). That is a real dependency: an account, a key, a bill, a third party on your critical path. Projects built to run fully locally need none of those things — so gosearch talks to the engines' own public result pages directly, the same way a web browser would, and parses what comes back.

The core module's entire third-party dependency surface is golang.org/x/net/html. That is a deliberate architectural bet: you can drop this into almost any Go program without dragging in an SDK.

Features

• Four engines, one interface — DuckDuckGo, Bing, Google, Yandex through a single Search(ctx, query, engine, ...Option) call returning structured []Result{Title, URL, Snippet}.
• Readable content extraction — Fetch(ctx, url, ...Option) returns a page's main content with navigation, ads, scripts, and boilerplate removed — never raw HTML.
• Markdown output — WithMarkdown() renders extracted content as GitHub-flavored Markdown: headings, lists, tables, fenced code, links, and emphasis survive. Built for LLM context, where structure carries meaning.
• Ordered fallback chain — WithFallback(...) moves to the next engine only when the current one reports being blocked or challenged, instead of failing outright.
• Honest failure signaling — anti-bot interventions surface as typed sentinel errors (ErrBlocked, ErrChallenge) via errors.Is, never as a silent empty result or a parse panic on a captcha page.
• Near-duplicate collapsing — engines often list one page under several URL spellings (percent-encoded vs literal İ/I, reordered parameters); these collapse into one result, original spelling preserved.
• Opt-in freshness dates — WithDates() fills Result.Date from each engine's own metadata when provided; off by default so timeless use cases are never polluted by accident.
• Caller-side domain policy — WithBlockedDomains(...) / WithAllowedDomains(...) enforce your spam and quality rules on the result list. The library judges no site on its own.
• Resilience built in — transient-failure retries with exponential backoff, realistic browser headers, persistent cookie jar, per-host rate limiting.

Quick start

Requirements: Go 1.25+. No keys, no accounts, no config files.

go get github.com/BugraAkdemir/gosearch@latest

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/BugraAkdemir/gosearch"
)

func main() {
	ctx := context.Background()

	// Search (DuckDuckGo is the most reliable default).
	results, err := gosearch.Search(ctx, "facebook", gosearch.DuckDuckGo,
		gosearch.WithMaxResults(5),
	)
	if err != nil {
		log.Fatal(err)
	}
	for _, r := range results {
		fmt.Println(r.Title, "->", r.URL)
	}

	// Extract any page's readable content.
	page, err := gosearch.Fetch(ctx, "https://en.wikipedia.org/wiki/Facebook")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(page.Title)
	fmt.Println(page.Content)
}

Usage patterns

Resilient searches with fallback

If the primary engine blocks or challenges the request, the same query continues down the chain automatically. First success wins.

results, err := gosearch.Search(ctx, "facebook", gosearch.Google,
	gosearch.WithFallback(gosearch.Bing, gosearch.DuckDuckGo),
)

Page content as Markdown (LLM-ready)

page, err := gosearch.Fetch(ctx, url, gosearch.WithMarkdown())
// page.Content now contains "# Heading", "- list items", "[links](…)", "```code```"

The LLM-agent pipeline

The two calls compose into the standard retrieval flow:

// 1. Find candidate sources.
results, err := gosearch.Search(ctx, question, gosearch.DuckDuckGo,
	gosearch.WithFallback(gosearch.Bing),
	gosearch.WithMaxResults(5),
	gosearch.WithBlockedDomains("pinterest.com"), // your quality policy
)

// 2. Read the top hits as Markdown context.
var context strings.Builder
for _, r := range results[:min(3, len(results))] {
	page, err := gosearch.Fetch(ctx, r.URL, gosearch.WithMarkdown())
	if err != nil {
		continue // one unreadable source ≠ abort
	}
	fmt.Fprintf(&context, "## %s\n%s\n\n", page.Title, page.Content)
}
// context.String() → model input

Freshness-aware results

results, _ := gosearch.Search(ctx, q, gosearch.Bing,
	gosearch.WithDates(), // Result.Date: "1 day ago", "2026-08-20", …
)

Full option semantics: docs/API.md. Task-oriented copy-paste recipes: docs/RECIPES.md.

Choosing an engine

Anti-bot strictness differs per engine and per network — IP reputation is often the deciding factor, not client behavior. Rough expectations from a normal residential connection, based on live testing of this library:

Engine	Expected reliability	Notes
DuckDuckGo	Highest	Official no-JS HTML endpoint; parser validated against a real captured page.
Bing	High	Served clean organic results even to a flagged datacenter IP; titles arrive behind a click-tracker unwrapped best-effort.
Google	Moderate	No official no-JS endpoint; DOM changes without notice.
Yandex	Lowest	Aggressive geo/IP-based captcha gating, especially outside Russia.

"Best-effort heuristic" status: the DuckDuckGo parser is validated against a real captured response; the other three parsers match current observed markup and are tested against synthetic fixtures, but engines rotate their DOM without notice — treat a parse miss as a signal to capture fresh HTML, not necessarily a bug.

Options

One variadic ...Option applies to both Search and Fetch; scope notes below. Full table with semantics: docs/API.md.

Option	Scope	Purpose
WithTimeout(d)	both	Request deadline (default 15s)
WithMaxResults(n)	Search	Cap result count
WithFallback(engines...)	Search	Ordered block/challenge fallback
WithDates()	Search	Populate Result.Date (default off)
WithBlockedDomains(ds...) / WithAllowedDomains(ds...)	Search	Your domain policy
WithRetries(n)	both	Transient-failure retries (default 2, 0 disables)
WithProxy(rawURL)	both	Route through your own egress
WithCookies(cs...)	both	Seed session cookies
WithHeader(k, v) / WithUserAgent(ua)	both	Override request headers
WithHTTPClient(c)	both	Bring your own *http.Client (escape hatch)
WithMarkdown()	Fetch	Markdown instead of plain-text content

Reliability, honestly

Search engines run anti-bot systems. A 200 OK does not mean success — engines serve captcha/challenge pages with status 200, which is why every response passes through block detection before parsing, and why failures are typed errors rather than empty slices. From datacenter/cloud IPs every engine may challenge or block you regardless of politeness; residential networks fare far better. When every engine in a fallback chain refuses you, Search returns an errors.Join of each engine's error so you can see exactly what happened.

Explicit non-goals

This library behaves like an ordinary visitor — and stops there. It will never solve CAPTCHAs, execute JS challenges to disguise itself, mask automation flags, or rotate identities to defeat a security control. Those cross from "look like a normal visitor" into "defeat a security control," which is out of scope on principle, not merely as a technical limitation.

Real-browser rendering (gosearch/browser)

For pages whose content only exists after JavaScript runs — which defeats plain-HTTP Fetch() by definition — an opt-in separate Go module drives a real, unmodified Chromium-family browser:

• Discovers Chrome/Edge/Chromium on the system; optionally downloads Google's official chrome-headless-shell with explicit permission; or embeds the engine at compile time via -tags gosearch_embed_engine.
• One long-lived process and one reused tab: steady-state cost is about one page's RAM, not a browser per request.
• Same honest line: it executes JavaScript; it does not solve CAPTCHAs or mask automation.

Because it lives in its own module, go get github.com/BugraAkdemir/gosearch never pulls browser dependencies into your project. See browser/README.md for installation and trade-offs.

Versioning & stability

Releases follow semantic versioning as annotated vX.Y.Z tags; the module proxy distributes them automatically. During v0.x, the public API may still evolve between minor versions — pin an exact version if that matters to you. The browser/ directory is a separate Go module and will carry its own browser/vX.Y.Z tags when published.

Documentation

Document	Contents
docs/GETTING_STARTED.md	From zero to first search and first extraction
docs/RECIPES.md	Copy-paste solutions: fallback chains, error handling, LLM pipelines, batch searching, the browser engine, troubleshooting
docs/API.md	Human-readable reference for every function, type, option, and error (go doc -all . remains the source of truth)
docs/ARCHITECTURE.md	Package graph, request flow, provider status, and the reasoning behind the internal/ split

Contributing

See CONTRIBUTING.md. Development conventions, verification commands, and known pitfalls live in AGENTS.md; the phased roadmap and exit criteria live in plan.md.

License

MIT

Documentation

Overview

Package gosearch performs web search and page-content extraction without an API key. It works by fetching each search engine's public HTML result page (DuckDuckGo, Google, Yandex, Bing) and parsing it directly, plus a Fetch function that extracts the readable content of any URL.

It is built for local-first / zero-dependency Go programs (for example an LLM agent's web-search tool) that cannot or will not depend on a hosted search API. The only third-party dependency is golang.org/x/net/html.

Reliability

Search engines run anti-bot systems (image captchas, JavaScript challenges, IP-reputation blocks). gosearch behaves like an ordinary browser — realistic headers, a persistent cookie jar, self-imposed rate limiting, and the lowest-friction endpoint each engine offers — but it never attempts to solve a CAPTCHA, execute a JavaScript challenge, or spoof its identity to defeat a security control. When an engine blocks a request, Search returns a typed error (ErrBlocked or ErrChallenge) rather than an empty result, so callers can react (for example, fall back to another engine via WithFallback).

Anti-bot strictness varies by engine and by network. DuckDuckGo is the most reliable (it is the only engine with an official no-JavaScript HTML endpoint); Google is moderate; Yandex is the most likely to block; Bing is the most tolerant of automated clients after DuckDuckGo. Requests from datacenter/cloud IPs are far more likely to be blocked than requests from a residential network.

Index

• Variables
• type Engine
  • func (e Engine) String() string
• type Option
  • func WithAllowedDomains(domains ...string) Option
  • func WithBlockedDomains(domains ...string) Option
  • func WithCookies(cookies ...*http.Cookie) Option
  • func WithDates() Option
  • func WithFallback(engines ...Engine) Option
  • func WithHTTPClient(client *http.Client) Option
  • func WithHeader(key, value string) Option
  • func WithMarkdown() Option
  • func WithMaxResults(n int) Option
  • func WithProxy(rawURL string) Option
  • func WithRetries(n int) Option
  • func WithTimeout(d time.Duration) Option
  • func WithUserAgent(ua string) Option
• type Page
  • func Fetch(ctx context.Context, url string, opts ...Option) (*Page, error)
• type Result
  • func Search(ctx context.Context, query string, engine Engine, opts ...Option) ([]Result, error)

Variables

var (
	// ErrBlocked means the engine refused the request because its anti-bot
	// system flagged it (for example an HTTP 429, an IP-reputation block, or a
	// redirect to a "you look like a bot" page). This is not a bug in the
	// query; it typically means the current IP/network is distrusted. Callers
	// using WithFallback will advance to the next engine on this error.
	ErrBlocked = serrors.ErrBlocked

	// ErrChallenge means the engine served an interactive challenge (an image
	// CAPTCHA or a JavaScript anti-bot challenge) instead of results. It is a
	// specific, common form of being blocked. gosearch never attempts to solve
	// such challenges by design. Callers using WithFallback will advance to the
	// next engine on this error.
	ErrChallenge = serrors.ErrChallenge

	// ErrNoResults means the request succeeded and was parsed successfully, but
	// the engine genuinely returned no results for the query. This is a valid
	// answer, not a failure: WithFallback does NOT advance to another engine on
	// this error, because a different engine is no more likely to have results
	// for a query that legitimately has none.
	ErrNoResults = serrors.ErrNoResults

	// ErrUnsupportedEngine means Search was called with an Engine value that is
	// not one of the defined constants (DuckDuckGo, Google, Yandex, Bing).
	ErrUnsupportedEngine = serrors.ErrUnsupportedEngine
)

Sentinel errors returned by Search and Fetch. Callers should test for these with errors.Is, never by string-matching an error message. Providers wrap these with additional context using fmt.Errorf("%w: ..."), so errors.Is continues to work through wrapping and through the errors.Join that the fallback chain uses.

These are re-exported from an internal package so that internal packages can return the same values without an import cycle; the identity is unchanged, so errors.Is works across the whole library.

Types

type Engine

type Engine int

Engine identifies which search engine Search should query. Pass one of the exported constants (DuckDuckGo, Google, Yandex, Bing) as the primary engine, and optionally more via WithFallback.

const (
	// DuckDuckGo queries html.duckduckgo.com, DuckDuckGo's no-JavaScript HTML
	// endpoint. It is the most reliable engine for this library because it is
	// explicitly designed to work without JavaScript.
	DuckDuckGo Engine = iota

	// Google queries Google Search. Google has no official no-JavaScript
	// endpoint and its result markup is regionally A/B tested, so parsing is
	// best-effort and more likely to break or be blocked than DuckDuckGo.
	Google

	// Yandex queries Yandex Search. Yandex applies aggressive, geo/IP-based
	// anti-bot gating, so it is the most likely of these engines to return
	// ErrBlocked or ErrChallenge.
	Yandex

	// Bing queries Microsoft Bing Search. Bing's plain-HTML endpoint is the
	// least aggressive of the four against automated clients — it served
	// clean organic results to a flagged datacenter IP during testing where
	// Google and Yandex challenged or blocked. Titles arrive wrapped in
	// Bing's click-tracker; the real destination is recovered from the
	// result's visible citation URL (best-effort, see the provider docs).
	Bing
)

func (Engine) String

func (e Engine) String() string

String returns the engine's human-readable name, suitable for logs and error messages.

type Option

type Option func(*config)

Option configures a Search or Fetch call. Options are applied in order.

Most options (timeout, proxy, headers, cookies, custom client) apply to both Search and Fetch. A few are meaningful only for Search — WithFallback and WithMaxResults — and are ignored by Fetch; each such option documents this.

func WithAllowedDomains

func WithAllowedDomains(domains ...string) Option

WithAllowedDomains keeps only Search results whose host matches one of the given domains or a subdomain of one; everything else is dropped. Results whose URL carries no parsable host cannot be proven allowed and drop too. An empty list means no allowlisting. Search-only; ignored by Fetch.

func WithBlockedDomains

func WithBlockedDomains(domains ...string) Option

WithBlockedDomains drops Search results whose host is one of the given domains or a subdomain of one ("spam.example.net" also kills "www.spam.example.net" but not "notspam.example.net"). This is the caller's SEO-spam / quality policy — the library itself makes no judgments about which sites deserve to exist. Combined with WithAllowedDomains, deny is applied first. Search-only; ignored by Fetch.

func WithCookies

func WithCookies(cookies ...*http.Cookie) Option

WithCookies seeds cookies into the client's cookie jar before the first request. This lets you reuse a session (for example cookies exported from your own logged-in browser) so requests look like a returning visitor.

func WithDates

func WithDates() Option

WithDates makes Search populate Result.Date with each engine's own freshness stamp (best-effort, often absent). Off by default so callers who need timeless results — historical queries, stable snapshots — never see date metadata by accident; enabling it does not change what the engine is asked or returns, it only surfaces metadata already on the page. This option is Search-only and is ignored by Fetch.

func WithFallback

func WithFallback(engines ...Engine) Option

WithFallback sets the ordered list of engines to try if the primary engine returns ErrBlocked or ErrChallenge. Engines are tried in exactly the order given; the first that succeeds wins. Fallback is not triggered by ErrNoResults (an empty result set is a valid answer). This option is Search-only and is ignored by Fetch.

func WithHTTPClient

func WithHTTPClient(client *http.Client) Option

WithHTTPClient uses the supplied *http.Client as-is instead of the client the library would otherwise build. This is an advanced escape hatch: it bypasses the library's default headers, cookie jar, and rate limiting, so you take responsibility for configuring those yourself. Timeout, proxy, and header options are not applied on top of a client supplied this way.

func WithHeader

func WithHeader(key, value string) Option

WithHeader adds or overrides a single request header. Call it multiple times to set multiple headers.

func WithMarkdown

func WithMarkdown() Option

WithMarkdown makes Fetch return its extracted main content as GitHub-flavored Markdown in Page.Content — headings as # levels, lists as bullets, code blocks fenced, links as [text](href), emphasis preserved — instead of plain text. Structure is what makes page content useful to LLM consumers; callers who want bare text simply omit this option (the default). This option is Fetch-only and is ignored by Search.

func WithMaxResults

func WithMaxResults(n int) Option

WithMaxResults caps the number of results Search returns. Zero (the default) means no cap. This option is Search-only and is ignored by Fetch.

func WithProxy

func WithProxy(rawURL string) Option

WithProxy routes all requests through the given proxy URL (for example "http://user:pass@host:port" or "socks5://host:port"). This is a legitimate way to use your own network egress; it is not used to rotate identities to defeat anti-bot systems.

func WithRetries

func WithRetries(n int) Option

WithRetries sets how many times a transiently failing request — transport error or HTTP 408/5xx — is retried with exponential backoff before the failure is final. The default is 2; passing 0 or a negative number disables retrying entirely. Blocks and challenges (ErrBlocked/ErrChallenge) are never retried — they are deterministic for the caller's IP reputation, and WithFallback exists for them.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout sets the timeout for the operation. A non-positive duration resets to the default. The default is 15 seconds.

func WithUserAgent

func WithUserAgent(ua string) Option

WithUserAgent overrides the default browser User-Agent header. Supplying a realistic browser User-Agent is part of how this library avoids looking like a bot; override only if you have a specific reason.

type Page

type Page struct {
	// URL is the page that was fetched. If the request followed redirects,
	// this is the final URL.
	URL string
	// Title is the page's title (from <title>, an <h1>, or article metadata,
	// whichever the extractor judged best). It may be empty.
	Title string
	// Content is the extracted main text of the page. It may be empty if the
	// extractor could not identify a main content region (for example, a page
	// whose content is rendered entirely by JavaScript).
	Content string
}

Page is the extracted, readable content of a URL returned by Fetch. It holds the main article/body text with navigation, ads, scripts, and boilerplate removed — not the raw HTML of the page.

func Fetch

func Fetch(ctx context.Context, url string, opts ...Option) (*Page, error)

Fetch retrieves url and extracts its main readable content into a Page. It does not run JavaScript, so pages whose content is rendered client-side may yield an empty Page.Content. If the server responds with an anti-bot page (for example HTTP 403/429), Fetch returns ErrBlocked/ErrChallenge.

With Markdown opted in (WithMarkdown), Page.Content carries GitHub-flavored Markdown with headings, lists, code fences, links, and emphasis preserved; the default remains plain text.

Search-only options (WithFallback, WithMaxResults) are ignored by Fetch.

type Result

type Result struct {
	// Title is the result's display title (the clickable heading).
	Title string
	// URL is the destination link the result points to.
	URL string
	// Snippet is the short description/excerpt shown under the title. It may
	// be empty for some engines or result types.
	Snippet string
	// Date is the result's freshness stamp exactly as the engine rendered it
	// (for example "2026-08-20" or "3 days ago") when the engine exposed one.
	// It stays "" unless Search was called with WithDates, and even then many
	// engines simply do not provide dates on their no-JavaScript pages — an
	// empty Date is normal and must not be treated as an error.
	Date string
}

Result is a single web-search result returned by Search.

func Search

func Search(ctx context.Context, query string, engine Engine, opts ...Option) ([]Result, error)

Search queries the given engine for query and returns the parsed results.

If WithFallback supplied additional engines, they are tried, in order, only when the current engine returns ErrBlocked or ErrChallenge; the first engine that succeeds wins. Fallback is NOT triggered by ErrNoResults (an empty result set is a valid answer) nor by other errors (network failures, etc.), which are returned immediately. If every engine in the chain is blocked/challenged, Search returns the errors.Join of each engine's error, so errors.Is(err, ErrBlocked) / errors.Is(err, ErrChallenge) still report true.

The same underlying HTTP client (with its cookie jar and rate limiter) is reused across the fallback chain.

All four engines are implemented. The DuckDuckGo parser is validated against a real captured success page; the Google and Yandex parsers are best-effort heuristics written against documented markup until a real capture lands — see plan.md's exit criteria and AGENTS.md Known Pitfalls. Bing served clean organic results even to a flagged datacenter IP during the 2026-08-24 probe, but its parser is likewise best-effort until a real capture lands.