botdetect

README

botdetect

Bot, crawler, and malicious-traffic detection for Go HTTP services.

Pure, request-agnostic functions you can call from middleware, ingestion pipelines, or batch jobs — no store, no DB, no goroutines.

go get github.com/dracory/botdetect

Detection functions

import "github.com/dracory/botdetect"

// User-agent based bot/crawler detection (browsers, scrapers, AI crawlers,
// CLI HTTP clients, scanners)
botdetect.IsBot(r.UserAgent())

// Richer verdict: identity + class for purpose-aware policy
d := botdetect.Detect(r.UserAgent())
// d.Bot   → true/false
// d.Name  → "googlebot", "gptbot", "chatgpt-user", "sqlmap", ...
// d.Class → ClassSearch | ClassSocial | ClassAI | ClassAIUser |
//           ClassSEOTool | ClassMonitor | ClassScraper | ClassScanner | ClassOther

// Same, from an *http.Request (nil-safe)
botdetect.DetectRequest(r)

// Structured bot detection with normalized identity and BotClass
// (ClassSearch, ClassSocial, ClassAI, ClassAIUser, ClassSEOTool,
// ClassMonitor, ClassScraper, ClassScanner, ClassOther)
det := botdetect.Detect(r.UserAgent())
// det.Bot   true
// det.Name  "googlebot" | "gptbot" | "chatgpt-user" | "sqlmap" | ...
// det.Class botdetect.ClassSearch | botdetect.ClassAI | ...
det = botdetect.DetectRequest(r)

// Bot-only file probes: robots.txt, ads.txt, sitemap.xml, llms.txt, ...
botdetect.IsBotPath(r.URL.Path)

// Universally malicious endpoints: .env, .git/, .ssh/, shell.php, ...
// A single hit is strong evidence of a vulnerability scanner.
botdetect.IsMaliciousPath(r.URL.Path)

// Known referrer-spam domains (exact and subdomain matching)
botdetect.IsReferrerSpam(r.Referer())

// Cloud/data-center CIDR ranges (AWS, GCP, Azure, Hetzner, OVH, ...)
botdetect.IsDataCenterIP(clientIP)

All functions return plain bool and are safe to call with empty strings.

User-agent parsing

info := botdetect.ParseUserAgent(r.UserAgent())
// info.Browser        "edge" | "chrome" | "firefox" | ...
// info.BrowserVersion "120.0"
// info.Os             "macos" | "windows" | "ios" | ...
// info.OsVersion      "17.2"
// info.Device         "iphone" | "ipad" | "ipod" | ""
// info.DeviceType     "desktop" | "mobile" | "tablet" | "bot"

All name fields are normalized to lowercase — compare against the exported constants (botdetect.DeviceTypeMobile, botdetect.OsIOS, botdetect.BrowserEdge, botdetect.Bot, ...).

Blocking middleware

mux := http.NewServeMux()
// ... routes

handler := botdetect.Middleware(mux, botdetect.Options{
    // Let search engines and human-initiated AI fetches through;
    // block scanners, scrapers, and everything else detected.
    AllowedClasses: map[botdetect.BotClass]bool{
        botdetect.ClassSearch: true,
        botdetect.ClassAIUser: true,
    },
    OnDetection: func(r *http.Request, d botdetect.Detection) {
        log.Printf("bot: %s class=%s path=%s", d.Name, d.Class, r.URL.Path)
    },
})
http.ListenAndServe(":8080", handler)

Mount it early in the chain, before routing — bots then never reach your handlers or analytics. Details:

• Malicious paths (.env, .git/, shell.php, ...) are blocked regardless of UA — disable via DisableMaliciousPathBlock.
• /robots.txt is never blocked (RFC 9309: a 4xx robots.txt means "no rules").
• ObserveOnly: true detects + reports via OnDetection but never blocks — good for a rollout phase.
• BlockResponse customizes the rejection (default: 403 Forbidden).

Pattern lists

The raw pattern tables are exported for consumers that want to drive their own matching (e.g. SQL LIKE scans over stored logs):

botdetect.BotPathPatterns()       // bot-only filenames
botdetect.MaliciousPathPatterns() // malicious path patterns

Design notes

• IsBot("") returns false — a missing UA is not, by itself, evidence of a bot.
• Malicious-path patterns are stack-agnostic by design (*.php, wp-admin are excluded — they're malicious on some stacks, legitimate on others). Add your own stack-specific checks on top.
• Data-center CIDR matching uses coarse published ranges; some ranges may produce false positives on non-datacenter traffic. See docs/maintaining-data.md for how to update the detection lists and refresh the CIDR ranges.

License

AGPL-3.0 — see LICENSE.

Documentation

Overview

Package botdetect provides bot, crawler, and malicious-traffic detection for HTTP requests, plus an optional blocking middleware.

Detection is split into pure, request-agnostic functions (Detect, DetectRequest, IsBot, IsBotPath, IsMaliciousPath, IsReferrerSpam, IsDataCenterIP) so the verdicts can be reused from middleware, ingestion pipelines, or batch jobs.

Index

• Constants
• func BotPathPatterns() []string
• func IsBot(userAgent string) bool
• func IsBotPath(path string) bool
• func IsBotRequest(r *http.Request) bool
• func IsDataCenterIP(ip string) bool
• func IsMaliciousPath(path string) bool
• func IsReferrerSpam(referrer string) bool
• func MaliciousPathPatterns() []string
• func Middleware(next http.Handler, opts Options) http.Handler
• type BotClass
• type Detection
  • func Detect(userAgent string) Detection
  • func DetectRequest(r *http.Request) Detection
• type Options
• type UserAgentInfo
  • func ParseRequest(r *http.Request) UserAgentInfo
  • func ParseUserAgent(ua string) UserAgentInfo

Constants

const (
	OsMacOS = "macos"
	OsIOS   = "ios"
)

OS names returned by ParseUserAgent.

const (
	DeviceIPhone = "iphone"
	DeviceIPad   = "ipad"
	DeviceIPod   = "ipod"
)

Device names returned by ParseUserAgent.

const (
	DeviceTypeDesktop = "desktop"
	DeviceTypeMobile  = "mobile"
	DeviceTypeTablet  = "tablet"
	DeviceTypeTV      = "tv"
)

Device types returned by ParseUserAgent in UserAgentInfo.DeviceType.

const Bot = "bot"

Bot is the value uasurfer-derived fields (DeviceType, Browser, Device) take when the user-agent is classified as a bot.

const (
	BrowserEdge = "edge"
)

Browser names returned by ParseUserAgent. All values are lowercase — display formatting (e.g. "macos" → "macOS") is left to consumers.

Functions

func BotPathPatterns

func BotPathPatterns() []string

BotPathPatterns returns the list of path substrings for bot-only files (robots.txt, ads.txt, sitemap.xml, etc.) that legitimate crawlers request but no human browser does. Consumers can use this list to drive SQL LIKE scans or similar bulk queries.

func IsBot

func IsBot(userAgent string) bool

IsBot checks whether a user-agent string belongs to a bot, crawler, or automated tool. The primary verdict comes from uasurfer's IsBot() (conventional bot tokens, contact-URL convention, headless browsers, named crawler fleets). Our own pattern lists then act as a supplementary pass for scanners and crawlers uasurfer deliberately skips. An empty user-agent returns false — a missing UA is not, by itself, evidence of a bot.

func IsBotPath

func IsBotPath(path string) bool

IsBotPath reports whether a request path targets a bot-only file (robots.txt, ads.txt, sitemap.xml, etc.) that no human browser requests. Matching is case-insensitive. The pattern is matched as a suffix of the last path segment (filename), so "/app-ads.txt" matches "ads.txt" but "/ads.txt.backup" does not.

func IsBotRequest

func IsBotRequest(r *http.Request) bool

IsBotRequest is the boolean shorthand for DetectRequest — reports whether the request's User-Agent or any alternate UA-carrier header identifies as a bot. Agent identity only: paths are not checked (IsBotPath/IsMaliciousPath stay separate concerns).

func IsDataCenterIP

func IsDataCenterIP(ip string) bool

IsDataCenterIP checks whether an IP address falls within known data center CIDR ranges (AWS, GCP, Azure, DigitalOcean, Oracle Cloud, Hetzner, OVH, Linode/Akamai, Vultr, Alibaba, Tencent, Contabo, Scaleway/Online.net, IBM Cloud, UpCloud).

func IsMaliciousPath

func IsMaliciousPath(path string) bool

IsMaliciousPath reports whether a request path targets a universally malicious endpoint (.env, .git/, .svn/, .htpasswd, shell.php) that is never legitimate on any properly-configured site. A single hit is strong evidence of a vulnerability scanner. Stack-dependent patterns are not included — consumers should add their own.

File patterns (e.g. shell.php) are matched as a suffix of the last path segment. Dotfile patterns starting with '.' (e.g. .env, .htpasswd) match the segment exactly, as a suffix (e.g. "backup.env"), or as a prefix followed by a separator — capturing variants like .env.bak, .env_old, and .env.local while avoiding false positives on .envrc or .envelope.yml. Directory patterns (e.g. .git/, .svn/) are matched against individual path segments, so "/.git/config" matches but "/.github/workflows" does not.

func IsReferrerSpam

func IsReferrerSpam(referrer string) bool

IsReferrerSpam checks whether a referrer URL host matches a known spam domain. The referrer can be a full URL or just a domain. Matching is case-insensitive.

func MaliciousPathPatterns

func MaliciousPathPatterns() []string

MaliciousPathPatterns returns the list of path substrings for endpoints that are never legitimate on any properly-configured website (e.g. .env, .git/, shell.php). Stack-dependent patterns like .php or wp-admin are excluded — consumers should add their own based on their tech stack.

func Middleware

func Middleware(next http.Handler, opts Options) http.Handler

Middleware returns HTTP middleware that blocks bots (by class policy) and malicious-path probes before they reach the wrapped handler.

/robots.txt is never blocked: RFC 9309 defines a 4xx robots.txt as "no crawl restrictions", so blocking it would tell crawlers everything is allowed.

Types

type BotClass

type BotClass string

BotClass categorizes a detected bot so consumers can apply purpose-aware policy (allow search engines, block scanners, observe AI crawlers) instead of a flat allow/block.

const (
	ClassSearch  BotClass = "search"   // Googlebot, Bingbot, DuckDuckBot, ...
	ClassSocial  BotClass = "social"   // Twitterbot, FacebookExternalHit, LinkedInBot, ...
	ClassAI      BotClass = "ai"       // GPTBot, ClaudeBot, Google-Extended, Bytespider, ...
	ClassAIUser  BotClass = "ai-user"  // ChatGPT-User, Perplexity-User, Claude-Web (human-initiated fetch)
	ClassSEOTool BotClass = "seo-tool" // AhrefsBot, SemrushBot, PetalBot, ...
	ClassMonitor BotClass = "monitor"  // Pingdom, Datadog, Site24x7, UptimeRobot, ...
	ClassScraper BotClass = "scraper"  // curl, python-requests, scrapy, headless, ...
	ClassScanner BotClass = "scanner"  // sqlmap, nikto, nmap, masscan, zgrab, cve-, ...
	ClassOther   BotClass = "other"    // generic/unclassified bots
)

type Detection

type Detection struct {
	Bot   bool     // true if identified as a bot
	Name  string   // normalized bot identifier (e.g. "googlebot", "gptbot", "sqlmap")
	Class BotClass // empty when Bot is false
}

Detection is the result of analyzing a user-agent or request: whether it is a bot, its normalized identity, and its class.

func Detect

func Detect(userAgent string) Detection

func DetectRequest

func DetectRequest(r *http.Request) Detection

DetectRequest analyzes an *http.Request: the User-Agent plus alternate UA-carrier headers from the vendored list (X-Operamini-Phone-Ua, X-Device-User-Agent, X-Original-User-Agent, From, X-Scanner, ...). Returns a zero Detection if r is nil. A non-empty X-Scanner header is an immediate match — scanners announce themselves there and the value is not checked against patterns.

type Options

type Options struct {
	// AllowedClasses lets detected bots of these classes through (e.g.
	// ClassSearch for SEO). Empty means every detected bot is blocked.
	// Humans (Bot == false) are always allowed.
	AllowedClasses map[BotClass]bool

	// BlockMaliciousPath rejects requests to universally malicious paths
	// (.env, .git/, shell.php, ...) regardless of user-agent. Enabled
	// unless explicitly set to false via DisableMaliciousPathBlock.
	DisableMaliciousPathBlock bool

	// ObserveOnly detects and reports via OnDetection but never blocks.
	ObserveOnly bool

	// OnDetection is called for every request that Detect flags as a bot
	// or that hits a malicious path — before the block decision. Useful
	// for logging or metrics.
	OnDetection func(r *http.Request, d Detection)

	// BlockResponse writes the rejection for blocked requests.
	// Defaults to a 403 "Forbidden".
	BlockResponse http.HandlerFunc
}

Options configures Middleware.

type UserAgentInfo

type UserAgentInfo struct {
	Browser        string
	BrowserVersion string
	Os             string
	OsVersion      string
	Device         string
	DeviceType     string
}

UserAgentInfo holds parsed user-agent data.

func ParseRequest

func ParseRequest(r *http.Request) UserAgentInfo

ParseRequest parses a request's User-Agent plus Sec-CH-UA-* client hints via uasurfer's ParseWithHints — better device-type and OS-version accuracy on Chromium browsers where the consumer advertises Accept-CH. Returns a zero UserAgentInfo if r is nil or the UA is empty.

func ParseUserAgent

func ParseUserAgent(ua string) UserAgentInfo

ParseUserAgent extracts browser, OS, device, and device-type from a user-agent string using the uasurfer library. Unknown values are left as empty strings. All name fields are normalized to lowercase so consumers can compare them without case-folding.