botdetect

README

botdetect

Bot, crawler, and malicious-traffic detection for Go HTTP services.

Pure, request-agnostic functions you can call from middleware, ingestion pipelines, or batch jobs — no store, no DB, no goroutines.

go get github.com/dracory/botdetect

Detection functions

import "github.com/dracory/botdetect"

// User-agent based bot/crawler detection (browsers, scrapers, AI crawlers,
// CLI HTTP clients, scanners)
botdetect.IsBot(r.UserAgent())

// Richer verdict: identity + class for purpose-aware policy
d := botdetect.Detect(r.UserAgent())
// d.Bot   → true/false
// d.Name  → "googlebot", "gptbot", "chatgpt-user", "sqlmap", ...
// d.Class → ClassSearch | ClassSocial | ClassAI | ClassAIUser |
//           ClassSEOTool | ClassMonitor | ClassScraper | ClassScanner | ClassOther

// Same, from an *http.Request (nil-safe)
botdetect.DetectRequest(r)

// Structured bot detection with normalized identity and BotClass
// (ClassSearch, ClassSocial, ClassAI, ClassAIUser, ClassSEOTool,
// ClassMonitor, ClassScraper, ClassScanner, ClassOther)
det := botdetect.Detect(r.UserAgent())
// det.Bot   true
// det.Name  "googlebot" | "gptbot" | "chatgpt-user" | "sqlmap" | ...
// det.Class botdetect.ClassSearch | botdetect.ClassAI | ...
det = botdetect.DetectRequest(r)

// Bot-only file probes: robots.txt, ads.txt, sitemap.xml, llms.txt, ...
botdetect.IsBotPath(r.URL.Path)

// Universally malicious endpoints: .env, .git/, .ssh/, shell.php, ...
// A single hit is strong evidence of a vulnerability scanner.
botdetect.IsMaliciousPath(r.URL.Path)

// Known referrer-spam domains (exact and subdomain matching)
botdetect.IsReferrerSpam(r.Referer())

// Cloud/data-center CIDR ranges (AWS, GCP, Azure, Hetzner, OVH, ...)
botdetect.IsDataCenterIP(clientIP)

All functions return plain bool and are safe to call with empty strings.

User-agent parsing

info := botdetect.ParseUserAgent(r.UserAgent())
// info.Browser        "edge" | "chrome" | "firefox" | ...
// info.BrowserVersion "120.0"
// info.Os             "macos" | "windows" | "ios" | ...
// info.OsVersion      "17.2"
// info.Device         "iphone" | "ipad" | "ipod" | ""
// info.DeviceType     "desktop" | "mobile" | "tablet" | "bot"

All name fields are normalized to lowercase — compare against the exported constants (botdetect.DeviceTypeMobile, botdetect.OsIOS, botdetect.BrowserEdge, botdetect.Bot, ...).

Blocking middleware

mux := http.NewServeMux()
// ... routes

handler := botdetect.Middleware(mux, botdetect.Options{
    // Let search engines and human-initiated AI fetches through;
    // block scanners, scrapers, and everything else detected.
    AllowedClasses: map[botdetect.BotClass]bool{
        botdetect.ClassSearch: true,
        botdetect.ClassAIUser: true,
    },
    // Jail repeat offenders for 5 minutes — after any block decision
    // the IP is rejected on a cheap map lookup.
    JailDuration: 5 * time.Minute,
    // Bearer-authed machine endpoints and assets bypass everything.
    ExcludePaths: []string{"/api/", "/assets*"},
    OnDetection: func(r *http.Request, d botdetect.Detection) {
        log.Printf("bot: %s class=%s path=%s", d.Name, d.Class, r.URL.Path)
    },
    OnJail: func(ip string, r *http.Request, d botdetect.Detection) {
        log.Printf("jailed %s (%s) for %s", ip, d.Name, r.URL.Path)
    },
})
http.ListenAndServe(":8080", handler)

Mount it early in the chain, before routing — bots then never reach your handlers or analytics. Details:

• Malicious paths (.env, .git/, shell.php, ...) are blocked regardless of UA — disable via DisableMaliciousPathBlock. Stack-specific probes (wp, .php, cgi-bin, backup, ...) come from PathBlocklist, which defaults to DefaultPathBlocklist() — edit a copy of the returned slice to exclude tokens (e.g. /new) or add your own.
• JailDuration jails the client IP after any block decision; GetIP customizes the jail key (default: RemoteAddr — supply an X-Forwarded-For-aware func behind a trusted proxy). Jails are in-process: they don't survive restarts or span replicas. Get the key wrong either way and it hurts: trusting raw XFF lets anyone jail arbitrary IPs, while keying on the proxy's own public egress IP (e.g. behind a CDN) lets one scanner jail every user behind it — honor XFF only from a known proxy CIDR list.
• ExcludePaths bypasses every check including jails — keep assets reachable for visitors on a jailed shared IP. IncludePaths does the inverse: when set, only matching paths are gated at all (e.g. just /account/*); on overlap, exclusions win.
• /robots.txt is never blocked (RFC 9309: a 4xx robots.txt means "no rules").
• ObserveOnly: true detects + reports via OnDetection but never blocks or jails — good for a rollout phase.
• BlockResponse customizes the rejection (default: 403 Forbidden).

Pattern lists

The raw pattern tables are exported for consumers that want to drive their own matching (e.g. SQL LIKE scans over stored logs):

botdetect.BotPathPatterns()       // bot-only filenames
botdetect.MaliciousPathPatterns() // malicious path patterns

Design notes

• IsBot("") returns false — a missing UA is not, by itself, evidence of a bot.
• Malicious-path patterns are stack-agnostic by design (*.php, wp-admin are excluded — they're malicious on some stacks, legitimate on others). Add your own stack-specific checks on top.
• Data-center CIDR matching uses coarse published ranges; some ranges may produce false positives on non-datacenter traffic. See docs/maintaining-data.md for how to update the detection lists and refresh the CIDR ranges.

License

AGPL-3.0 — see LICENSE.

Documentation

Overview

Package botdetect provides bot, crawler, and malicious-traffic detection for HTTP requests, plus an optional blocking middleware.

Detection is split into pure, request-agnostic functions (Detect, DetectRequest, IsBot, IsBotPath, IsMaliciousPath, IsReferrerSpam, IsDataCenterIP) so the verdicts can be reused from middleware, ingestion pipelines, or batch jobs.

Index

• Constants
• func BotPathPatterns() []string
• func DefaultPathBlocklist() []string
• func IsBot(userAgent string) bool
• func IsBotPath(path string) bool
• func IsBotRequest(r *http.Request) bool
• func IsDataCenterIP(ip string) bool
• func IsMaliciousPath(path string) bool
• func IsReferrerSpam(referrer string) bool
• func MaliciousPathPatterns() []string
• func MatchesPathBlocklist(path string, patterns []string) bool
• func Middleware(next http.Handler, opts Options) http.Handler
• type BotClass
• type Detection
  • func Detect(userAgent string) Detection
  • func DetectRequest(r *http.Request) Detection
• type Options
• type UserAgentInfo
  • func ParseRequest(r *http.Request) UserAgentInfo
  • func ParseUserAgent(ua string) UserAgentInfo

Constants

const (
	OsMacOS = "macos"
	OsIOS   = "ios"
)

OS names returned by ParseUserAgent.

const (
	DeviceIPhone = "iphone"
	DeviceIPad   = "ipad"
	DeviceIPod   = "ipod"
)

Device names returned by ParseUserAgent.

const (
	DeviceTypeDesktop = "desktop"
	DeviceTypeMobile  = "mobile"
	DeviceTypeTablet  = "tablet"
	DeviceTypeTV      = "tv"
)

Device types returned by ParseUserAgent in UserAgentInfo.DeviceType.

const Bot = "bot"

Bot is the value uasurfer-derived fields (DeviceType, Browser, Device) take when the user-agent is classified as a bot.

const (
	BrowserEdge = "edge"
)

Browser names returned by ParseUserAgent. All values are lowercase — display formatting (e.g. "macos" → "macOS") is left to consumers.

Functions

func BotPathPatterns

func BotPathPatterns() []string

BotPathPatterns returns the list of path substrings for bot-only files (robots.txt, ads.txt, sitemap.xml, etc.) that legitimate crawlers request but no human browser does. Consumers can use this list to drive SQL LIKE scans or similar bulk queries.

func DefaultPathBlocklist

func DefaultPathBlocklist() []string

DefaultPathBlocklist returns a fresh copy of the default stack-specific path blocklist used by Middleware. Edit the returned slice — e.g. slices.DeleteFunc(list, func(p string) bool { return p == "/new" }) — and assign it to Options.PathBlocklist to exclude or add tokens.

func IsBot

func IsBot(userAgent string) bool

IsBot checks whether a user-agent string belongs to a bot, crawler, or automated tool. The primary verdict comes from uasurfer's IsBot() (conventional bot tokens, contact-URL convention, headless browsers, named crawler fleets). Our own pattern lists then act as a supplementary pass for scanners and crawlers uasurfer deliberately skips. An empty user-agent returns false — a missing UA is not, by itself, evidence of a bot.

func IsBotPath

func IsBotPath(path string) bool

IsBotPath reports whether a request path targets a bot-only file (robots.txt, ads.txt, sitemap.xml, etc.) that no human browser requests. Matching is case-insensitive. The pattern is matched as a suffix of the last path segment (filename), so "/app-ads.txt" matches "ads.txt" but "/ads.txt.backup" does not.

func IsBotRequest

func IsBotRequest(r *http.Request) bool

IsBotRequest is the boolean shorthand for DetectRequest — reports whether the request's User-Agent or any alternate UA-carrier header identifies as a bot. Agent identity only: paths are not checked (IsBotPath/IsMaliciousPath stay separate concerns).

func IsDataCenterIP

func IsDataCenterIP(ip string) bool

IsDataCenterIP checks whether an IP address falls within known data center CIDR ranges (AWS, GCP, Azure, DigitalOcean, Oracle Cloud, Hetzner, OVH, Linode/Akamai, Vultr, Alibaba, Tencent, Contabo, Scaleway/Online.net, IBM Cloud, UpCloud).

func IsMaliciousPath

func IsMaliciousPath(path string) bool

IsMaliciousPath reports whether a request path targets a universally malicious endpoint (.env, .git/, .svn/, .htpasswd, shell.php) that is never legitimate on any properly-configured site. A single hit is strong evidence of a vulnerability scanner. Stack-dependent patterns are not included — consumers should add their own.

File patterns (e.g. shell.php) are matched as a suffix of the last path segment. Dotfile patterns starting with '.' (e.g. .env, .htpasswd) match the segment exactly, as a suffix (e.g. "backup.env"), or as a prefix followed by a separator — capturing variants like .env.bak, .env_old, and .env.local while avoiding false positives on .envrc or .envelope.yml. Directory patterns (e.g. .git/, .svn/) are matched against individual path segments, so "/.git/config" matches but "/.github/workflows" does not.

func IsReferrerSpam

func IsReferrerSpam(referrer string) bool

IsReferrerSpam checks whether a referrer URL host matches a known spam domain. The referrer can be a full URL or just a domain. Matching is case-insensitive.

func MaliciousPathPatterns

func MaliciousPathPatterns() []string

MaliciousPathPatterns returns the list of path substrings for endpoints that are never legitimate on any properly-configured website (e.g. .env, .git/, shell.php). Stack-dependent patterns like .php or wp-admin are excluded — consumers should add their own based on their tech stack.

func MatchesPathBlocklist

func MatchesPathBlocklist(path string, patterns []string) bool

MatchesPathBlocklist reports whether path matches any blocklist pattern, using the shape-based semantics documented on stackPathBlocklist. A nil or empty pattern list matches nothing; an empty path matches nothing.

func Middleware

func Middleware(next http.Handler, opts Options) http.Handler

Middleware returns HTTP middleware that blocks bots (by class policy) and malicious-path probes before they reach the wrapped handler, with optional IP jailing for repeat offenders.

/robots.txt is never blocked: RFC 9309 defines a 4xx robots.txt as "no crawl restrictions", so blocking it would tell crawlers everything is allowed.

Evaluation order: IncludePaths → ExcludePaths → robots.txt → jailed-IP check → malicious paths (IsMaliciousPath + PathBlocklist) → UA detection. Any block decision jails the client IP when JailDuration > 0; ObserveOnly never blocks or jails.

Types

type BotClass

type BotClass string

BotClass categorizes a detected bot so consumers can apply purpose-aware policy (allow search engines, block scanners, observe AI crawlers) instead of a flat allow/block.

const (
	ClassSearch  BotClass = "search"   // Googlebot, Bingbot, DuckDuckBot, ...
	ClassSocial  BotClass = "social"   // Twitterbot, FacebookExternalHit, LinkedInBot, ...
	ClassAI      BotClass = "ai"       // GPTBot, ClaudeBot, Google-Extended, Bytespider, ...
	ClassAIUser  BotClass = "ai-user"  // ChatGPT-User, Perplexity-User, Claude-Web (human-initiated fetch)
	ClassSEOTool BotClass = "seo-tool" // AhrefsBot, SemrushBot, PetalBot, ...
	ClassMonitor BotClass = "monitor"  // Pingdom, Datadog, Site24x7, UptimeRobot, ...
	ClassScraper BotClass = "scraper"  // curl, python-requests, scrapy, headless, ...
	ClassScanner BotClass = "scanner"  // sqlmap, nikto, nmap, masscan, zgrab, cve-, ...
	ClassOther   BotClass = "other"    // generic/unclassified bots
)

type Detection

type Detection struct {
	Bot   bool     // true if identified as a bot
	Name  string   // normalized bot identifier (e.g. "googlebot", "gptbot", "sqlmap")
	Class BotClass // empty when Bot is false
}

Detection is the result of analyzing a user-agent or request: whether it is a bot, its normalized identity, and its class.

func Detect

func Detect(userAgent string) Detection

func DetectRequest

func DetectRequest(r *http.Request) Detection

DetectRequest analyzes an *http.Request: the User-Agent plus alternate UA-carrier headers from the vendored list (X-Operamini-Phone-Ua, X-Device-User-Agent, X-Original-User-Agent, From, X-Scanner, ...). Returns a zero Detection if r is nil. A non-empty X-Scanner header is an immediate match — scanners announce themselves there and the value is not checked against patterns.

type Options

type Options struct {
	// AllowedClasses lets detected bots of these classes through (e.g.
	// ClassSearch for SEO). Empty means every detected bot is blocked.
	// Humans (Bot == false) are always allowed.
	AllowedClasses map[BotClass]bool

	// DisableMaliciousPathBlock rejects requests to universally malicious
	// paths (.env, .git/, shell.php, ...) regardless of user-agent.
	// Enabled unless explicitly set to true.
	DisableMaliciousPathBlock bool

	// PathBlocklist adds stack-specific probe patterns (wp, .php,
	// cgi-bin, backup, ...) checked by MatchesPathBlocklist on top of
	// the universal IsMaliciousPath set. nil means
	// DefaultPathBlocklist(); pass an empty slice to disable, or an
	// edited copy of DefaultPathBlocklist() to exclude/add tokens —
	// e.g. "/new" false-positives on routes like /news.
	// Ignored when DisableMaliciousPathBlock is true.
	PathBlocklist []string

	// IncludePaths scopes every check — detection, blocking, and
	// jails — to matching request paths only; all other paths pass
	// untouched. Empty (default) means every path is in scope. Same
	// pattern syntax as ExcludePaths. Useful when the gated set is
	// narrower than the exempt set (e.g. only "/account/*" and
	// "/checkout/*"). When a path matches both lists, ExcludePaths wins.
	IncludePaths []string

	// ExcludePaths bypasses every check — detection, blocking, and
	// jails — for matching request paths. A trailing '*' means prefix
	// match; otherwise the pattern is segment-aware (exact path or
	// subpath). Matching is case-sensitive and exact by design —
	// exclusions are explicit opt-outs, not fingerprints (unlike
	// PathBlocklist, which is matched case-insensitively).
	// Use it for bearer-authed machine endpoints and assets that must
	// keep working for visitors on a jailed shared IP.
	ExcludePaths []string

	// JailDuration > 0 enables IP jailing: after any block decision the
	// client IP is rejected for this duration on a cheap map lookup,
	// before any detection work. 0 (default) keeps per-request
	// blocking only. Jails live in process memory — they do not
	// survive restarts and are not shared across replicas.
	JailDuration time.Duration

	// GetIP resolves the client IP used as the jail key. The default
	// reads RemoteAddr only — matching the challenge package, which
	// does not trust proxy headers by default. Consumers behind a
	// trusted proxy should supply an X-Forwarded-For-aware
	// implementation; note that trusting client-set headers lets an
	// attacker jail arbitrary IPs, so only honor them from a known
	// proxy chain.
	//
	// The inverse failure exists too: if every request arrives from a
	// public-IP edge (e.g. a CDN's egress) and GetIP keys on RemoteAddr
	// or a private-peer heuristic, one scanner jails the shared egress
	// IP and every user behind it eats 403s for the jail term — a
	// self-inflicted outage. Behind such an edge, GetIP must honor XFF
	// from a configured trusted-proxy CIDR list (CDNs publish theirs),
	// not just private ranges.
	GetIP func(r *http.Request) string

	// ObserveOnly detects and reports via OnDetection but never blocks
	// and never jails.
	ObserveOnly bool

	// OnDetection is called for every request that Detect flags as a
	// bot, that hits a malicious path, or that arrives from a jailed IP
	// — before the block decision. Useful for logging or metrics.
	OnDetection func(r *http.Request, d Detection)

	// OnJail is called once when an IP is jailed — not on every jailed
	// hit. Only fires when JailDuration > 0.
	OnJail func(ip string, r *http.Request, d Detection)

	// BlockResponse writes the rejection for blocked and jailed
	// requests. Defaults to a 403 "Forbidden".
	BlockResponse http.HandlerFunc
}

Options configures Middleware.

type UserAgentInfo

type UserAgentInfo struct {
	Browser        string
	BrowserVersion string
	Os             string
	OsVersion      string
	Device         string
	DeviceType     string
}

UserAgentInfo holds parsed user-agent data.

func ParseRequest

func ParseRequest(r *http.Request) UserAgentInfo

ParseRequest parses a request's User-Agent plus Sec-CH-UA-* client hints via uasurfer's ParseWithHints — better device-type and OS-version accuracy on Chromium browsers where the consumer advertises Accept-CH. Returns a zero UserAgentInfo if r is nil or the UA is empty.

func ParseUserAgent

func ParseUserAgent(ua string) UserAgentInfo

ParseUserAgent extracts browser, OS, device, and device-type from a user-agent string using the uasurfer library. Unknown values are left as empty strings. All name fields are normalized to lowercase so consumers can compare them without case-folding.