botdetect

README

botdetect

Bot, crawler, and malicious-traffic detection for Go HTTP services.

Pure, request-agnostic functions you can call from middleware, ingestion pipelines, or batch jobs — no store, no DB, no goroutines.

go get github.com/dracory/botdetect

Detection functions

import "github.com/dracory/botdetect"

// User-agent based bot/crawler detection (browsers, scrapers, AI crawlers,
// CLI HTTP clients, scanners)
botdetect.IsBot(r.UserAgent())

// Richer verdict: identity + class for purpose-aware policy
d := botdetect.Detect(r.UserAgent())
// d.Bot   → true/false
// d.Name  → "googlebot", "gptbot", "chatgpt-user", "sqlmap", ...
// d.Class → ClassSearch | ClassSocial | ClassAI | ClassAIUser |
//           ClassSEOTool | ClassMonitor | ClassScraper | ClassScanner | ClassOther

// Same, from an *http.Request (nil-safe)
botdetect.DetectRequest(r)

// Structured bot detection with normalized identity and BotClass
// (ClassSearch, ClassSocial, ClassAI, ClassAIUser, ClassSEOTool,
// ClassMonitor, ClassScraper, ClassScanner, ClassOther)
det := botdetect.Detect(r.UserAgent())
// det.Bot   true
// det.Name  "googlebot" | "gptbot" | "chatgpt-user" | "sqlmap" | ...
// det.Class botdetect.ClassSearch | botdetect.ClassAI | ...
det = botdetect.DetectRequest(r)

// Bot-only file probes: robots.txt, ads.txt, sitemap.xml, llms.txt, ...
botdetect.IsBotPath(r.URL.Path)

// Universally malicious endpoints: .env, .git/, .ssh/, shell.php, ...
// A single hit is strong evidence of a vulnerability scanner.
botdetect.IsMaliciousPath(r.URL.Path)

// Known referrer-spam domains (exact and subdomain matching)
botdetect.IsReferrerSpam(r.Referer())

// Cloud/data-center CIDR ranges (AWS, GCP, Azure, Hetzner, OVH, ...)
botdetect.IsDataCenterIP(clientIP)

All functions return plain bool and are safe to call with empty strings.

User-agent parsing

info := botdetect.ParseUserAgent(r.UserAgent())
// info.Browser        "edge" | "chrome" | "firefox" | ...
// info.BrowserVersion "120.0"
// info.Os             "macos" | "windows" | "ios" | ...
// info.OsVersion      "17.2"
// info.Device         "iphone" | "ipad" | "ipod" | ""
// info.DeviceType     "desktop" | "mobile" | "tablet" | "bot"

All name fields are normalized to lowercase — compare against the exported constants (botdetect.DeviceTypeMobile, botdetect.OsIOS, botdetect.BrowserEdge, botdetect.Bot, ...).

Blocking middleware example

func botBlock(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if botdetect.IsMaliciousPath(r.URL.Path) {
            http.Error(w, "forbidden", http.StatusForbidden)
            return
        }
        if botdetect.IsBot(r.UserAgent()) {
            http.Error(w, "forbidden", http.StatusForbidden)
            return
        }
        next.ServeHTTP(w, r)
    })
}

Mount it early in the chain, before routing — bots then never reach your handlers or analytics.

Note: blocking on IsBot also rejects legitimate crawlers (Googlebot, Bingbot). If SEO matters, flag rather than block, or allowlist known crawlers first.

Pattern lists

The raw pattern tables are exported for consumers that want to drive their own matching (e.g. SQL LIKE scans over stored logs):

botdetect.BotPathPatterns()       // bot-only filenames
botdetect.MaliciousPathPatterns() // malicious path patterns

Design notes

• IsBot("") returns false — a missing UA is not, by itself, evidence of a bot.
• Malicious-path patterns are stack-agnostic by design (*.php, wp-admin are excluded — they're malicious on some stacks, legitimate on others). Add your own stack-specific checks on top.
• Data-center CIDR matching uses coarse published ranges; some ranges may produce false positives on non-datacenter traffic. See docs/maintaining-data.md for how to update the detection lists and refresh the CIDR ranges.

License

AGPL-3.0 — see LICENSE.

Documentation

Overview

Package botdetect provides bot, crawler, and malicious-traffic detection for HTTP requests, plus an optional blocking middleware.

Detection is split into pure, request-agnostic functions (Detect, DetectRequest, IsBot, IsBotPath, IsMaliciousPath, IsReferrerSpam, IsDataCenterIP) so the verdicts can be reused from middleware, ingestion pipelines, or batch jobs.

Index

• Constants
• func BotPathPatterns() []string
• func IsBot(userAgent string) bool
• func IsBotPath(path string) bool
• func IsDataCenterIP(ip string) bool
• func IsMaliciousPath(path string) bool
• func IsReferrerSpam(referrer string) bool
• func MaliciousPathPatterns() []string
• type BotClass
• type Detection
  • func Detect(userAgent string) Detection
  • func DetectRequest(r *http.Request) Detection
• type UserAgentInfo
  • func ParseUserAgent(ua string) UserAgentInfo

Constants

const (
	OsMacOS = "macos"
	OsIOS   = "ios"
)

OS names returned by ParseUserAgent.

const (
	DeviceIPhone = "iphone"
	DeviceIPad   = "ipad"
	DeviceIPod   = "ipod"
)

Device names returned by ParseUserAgent.

const (
	DeviceTypeDesktop = "desktop"
	DeviceTypeMobile  = "mobile"
	DeviceTypeTablet  = "tablet"
	DeviceTypeTV      = "tv"
)

Device types returned by ParseUserAgent in UserAgentInfo.DeviceType.

const Bot = "bot"

Bot is the value uasurfer-derived fields (DeviceType, Browser, Device) take when the user-agent is classified as a bot.

const (
	BrowserEdge = "edge"
)

Browser names returned by ParseUserAgent. All values are lowercase — display formatting (e.g. "macos" → "macOS") is left to consumers.

Functions

func BotPathPatterns

func BotPathPatterns() []string

BotPathPatterns returns the list of path substrings for bot-only files (robots.txt, ads.txt, sitemap.xml, etc.) that legitimate crawlers request but no human browser does. Consumers can use this list to drive SQL LIKE scans or similar bulk queries.

func IsBot

func IsBot(userAgent string) bool

IsBot checks whether a user-agent string belongs to a bot, crawler, or automated tool. The primary verdict comes from uasurfer's IsBot() (conventional bot tokens, contact-URL convention, headless browsers, named crawler fleets). Our own pattern lists then act as a supplementary pass for scanners and crawlers uasurfer deliberately skips. An empty user-agent returns false — a missing UA is not, by itself, evidence of a bot.

func IsBotPath

func IsBotPath(path string) bool

IsBotPath reports whether a request path targets a bot-only file (robots.txt, ads.txt, sitemap.xml, etc.) that no human browser requests. Matching is case-insensitive. The pattern is matched as a suffix of the last path segment (filename), so "/app-ads.txt" matches "ads.txt" but "/ads.txt.backup" does not.

func IsDataCenterIP

func IsDataCenterIP(ip string) bool

IsDataCenterIP checks whether an IP address falls within known data center CIDR ranges (AWS, GCP, Azure, DigitalOcean, Oracle Cloud, Hetzner, OVH, Linode/Akamai, Vultr, Alibaba, Tencent, Contabo, Scaleway/Online.net, IBM Cloud, UpCloud).

func IsMaliciousPath

func IsMaliciousPath(path string) bool

IsMaliciousPath reports whether a request path targets a universally malicious endpoint (.env, .git/, .svn/, .htpasswd, shell.php) that is never legitimate on any properly-configured site. A single hit is strong evidence of a vulnerability scanner. Stack-dependent patterns are not included — consumers should add their own.

File patterns (e.g. shell.php) are matched as a suffix of the last path segment. Dotfile patterns starting with '.' (e.g. .env, .htpasswd) match the segment exactly, as a suffix (e.g. "backup.env"), or as a prefix followed by a separator — capturing variants like .env.bak, .env_old, and .env.local while avoiding false positives on .envrc or .envelope.yml. Directory patterns (e.g. .git/, .svn/) are matched against individual path segments, so "/.git/config" matches but "/.github/workflows" does not.

func IsReferrerSpam

func IsReferrerSpam(referrer string) bool

IsReferrerSpam checks whether a referrer URL host matches a known spam domain. The referrer can be a full URL or just a domain. Matching is case-insensitive.

func MaliciousPathPatterns

func MaliciousPathPatterns() []string

MaliciousPathPatterns returns the list of path substrings for endpoints that are never legitimate on any properly-configured website (e.g. .env, .git/, shell.php). Stack-dependent patterns like .php or wp-admin are excluded — consumers should add their own based on their tech stack.

Types

type BotClass

type BotClass string

BotClass categorizes a detected bot so consumers can apply purpose-aware policy (allow search engines, block scanners, observe AI crawlers) instead of a flat allow/block.

const (
	ClassSearch  BotClass = "search"   // Googlebot, Bingbot, DuckDuckBot, ...
	ClassSocial  BotClass = "social"   // Twitterbot, FacebookExternalHit, LinkedInBot, ...
	ClassAI      BotClass = "ai"       // GPTBot, ClaudeBot, Google-Extended, Bytespider, ...
	ClassAIUser  BotClass = "ai-user"  // ChatGPT-User, Perplexity-User, Claude-Web (human-initiated fetch)
	ClassSEOTool BotClass = "seo-tool" // AhrefsBot, SemrushBot, PetalBot, ...
	ClassMonitor BotClass = "monitor"  // Pingdom, Datadog, Site24x7, UptimeRobot, ...
	ClassScraper BotClass = "scraper"  // curl, python-requests, scrapy, headless, ...
	ClassScanner BotClass = "scanner"  // sqlmap, nikto, nmap, masscan, zgrab, cve-, ...
	ClassOther   BotClass = "other"    // generic/unclassified bots
)

type Detection

type Detection struct {
	Bot   bool     // true if identified as a bot
	Name  string   // normalized bot identifier (e.g. "googlebot", "gptbot", "sqlmap")
	Class BotClass // empty when Bot is false
}

Detection is the result of analyzing a user-agent or request: whether it is a bot, its normalized identity, and its class.

func Detect

func Detect(userAgent string) Detection

Detect analyzes a user-agent string and returns a Detection with the bot's identity and class. An empty user-agent returns a zero Detection.

Order of evaluation: the specific-name classification table runs first (preserving per-agent identity that uasurfer collapses into fleet constants — e.g. "chatgpt-user" vs "gptbot" both parse as BrowserOpenAIBot), then uasurfer's IsBot() for named fleets and generic heuristics, then our broad word-boundary pattern list for anything unclassified.

func DetectRequest

func DetectRequest(r *http.Request) Detection

DetectRequest analyzes an *http.Request — currently its User-Agent — and returns a Detection. Returns a zero Detection if r is nil. Alternate UA-carrier headers (From, X-Scanner, ...) are added in a follow-up once the vendored headers list lands.

type UserAgentInfo

type UserAgentInfo struct {
	Browser        string
	BrowserVersion string
	Os             string
	OsVersion      string
	Device         string
	DeviceType     string
}

UserAgentInfo holds parsed user-agent data.

func ParseUserAgent

func ParseUserAgent(ua string) UserAgentInfo

ParseUserAgent extracts browser, OS, device, and device-type from a user-agent string using the uasurfer library. Unknown values are left as empty strings. All name fields are normalized to lowercase so consumers can compare them without case-folding.