botdetect

Documentation

Overview

Package botdetect classifies a User-Agent as a client that will not run the boundary runtime.

It is here rather than in pw because both runtimes ask the same question and there is only one right answer to it. A second copy of the token list would drift, and the way it would drift is silent: a crawler added on one transport and not the other changes which render branch runs, and both branches produce a correct-looking page. Nothing would fail, and the difference would show up as a search result rather than as a test.

Nothing here touches a request. The caller reads the header, on whichever transport it has, and hands over the string.

Index

• Variables
• func Classify(agent string, extra []string) bool
• func Normalize(values []string) []string

Variables

var UserAgents = []string{

	"googlebot", "google-inspectiontool", "storebot-google", "adsbot-google",
	"mediapartners-google", "bingbot", "bingpreview", "msnbot", "slurp",
	"duckduckbot", "duckassistbot", "baiduspider", "yandexbot",
	"yandexrenderresourcesbot", "sogou", "seznambot", "qwantbot", "petalbot",
	"applebot", "naver", "yeti/",

	"gptbot", "oai-searchbot", "chatgpt-user", "claudebot", "claude-web",
	"claude-user", "claude-searchbot", "anthropic-ai", "perplexitybot",
	"perplexity-user", "ccbot", "bytespider", "amazonbot",
	"meta-externalagent", "cohere-ai", "diffbot", "youbot", "timpibot",
	"omgili",

	"facebookexternalhit", "facebookcatalog", "meta-externalfetcher",
	"twitterbot", "slackbot", "slack-imgproxy", "discordbot", "telegrambot",
	"whatsapp", "linkedinbot", "pinterest", "redditbot", "skypeuripreview",
	"embedly", "iframely", "vkshare", "mastodon", "bitlybot", "nuzzel",
	"quora link preview",

	"ahrefsbot", "semrushbot", "mj12bot", "dotbot", "dataforseobot",
	"screaming frog", "w3c_validator", "pingdom", "uptimerobot", "statuscake",

	"winhttp", "wininet", "ms-office", "microsoft office", "microsoft outlook",

	"+http", "crawler", "spider", "feedfetcher", "archive.org_bot",
}

UserAgents are the clients that present a browser-shaped User-Agent and still run no script. Everything that names itself honestly — curl, Wget, python-requests, Go-http-client — is caught by the Mozilla prefix rule in Classify instead, so this list only has to carry the bots that disguise themselves.

Every token here must be absent from every shipping browser User-Agent. That is why the bare substring "bot" is not a member: an Android phone reporting CUBOT would become a crawler. The browser list in pw's tests guards the rule.

Functions

func Classify

func Classify(agent string, extra []string) bool

Classify reports whether the agent belongs to a non-interactive client. The two tests answer different populations: the token scan catches bots wearing a browser costume, and the prefix rule catches every CLI and client library at once, including ones released after this list was written.

func Normalize

func Normalize(values []string) []string

Normalize lowercases the configured additions once, so classification stays a plain scan.