provider

Documentation

Overview

Package provider holds the shared type that search-engine providers return.

It lives in an internal package (rather than the public gosearch package) so providers can be written without importing gosearch, avoiding an import cycle (gosearch imports the providers). The public gosearch package converts these into its own documented Result type.

Index

• func ExtractDate(container *html.Node) string
• func NormalizeURL(raw string) string
• type Result

Functions

func ExtractDate

func ExtractDate(container *html.Node) string

ExtractDate returns the freshness/publication date a search-result container carries, if any: a <time datetime="..."> attribute wins, then Bing's news_dt span text. It returns "" when the container carries neither — dates are best-effort metadata, never required, and engines differ wildly in whether they expose them on the no-JavaScript pages this library parses.

The value passes through verbatim: ISO instants ("2026-08-20") and human relative stamps ("1 day ago") both occur. Whether the caller sees the value at all is decided one level up by the WithDates option, so callers who need timeless results keep getting them by default.

func NormalizeURL

func NormalizeURL(raw string) string

NormalizeURL returns a comparison key for URL-based deduplication. Search engines routinely list one page under several spellings — percent-encoded vs literal non-ASCII query values, dotted vs undotted capital I (a live Bing duplicate: ?il=Istanbul vs ?il=%C4%B0stanbul), differing parameter order — and callers should see one result, not near-identical twins.

The key folds what provably does not change the resource: scheme and host case, default ports, fragments, percent-encoding, and Unicode simple case (so İ and I collapse; deliberate — engines treat them as interchangeable). Everything else stays significant: path and trailing slash, query presence, http vs https. The returned key is NEVER a display URL — providers keep the original string in Result.URL.

Types

type Result

type Result struct {
	Title   string
	URL     string
	Snippet string

	// Date is the result's freshness stamp as the engine rendered it (for
	// example "2026-08-20" or "1 day ago"), extracted best-effort from the
	// result container. Engines often omit it entirely; "" is normal. The
	// root package strips this unless the caller opted in via WithDates.
	Date string
}

Result is one parsed search result, produced by a provider's Search function. It mirrors the public gosearch.Result field-for-field; gosearch copies these across so the public type's documentation and identity stay in the public package.