digital-dome

README

Digital Dome ✨

Opinionated Shield from AIs and Malicious Scanners

Digital Dome is a fast, minimal web application firewall that uses request information to protect a site against AI scanners. It hosts several configurable rules, along with sensible (if aggressive) defaults.

Quickstart

import github.com/benpate/digital-dome/dome
import github.com/benpate/digital-dome/dome4echo

domeConfig := dome.New(dome.RemoteAddr) // Create a new digital dome (using sensible defaults)
middleware := dome4echo.New(domeConfig)  // Create echo middleware
e.Pre(middleware)                        // Use the middleware

// easy peasy.

The first argument to dome.New() is a required ClientIPResolver — a func(*http.Request) string that returns the "real" client IP for a request. This is the address Digital Dome uses when tracking and blocking bad actors, so it must reflect the true client. Pass the built-in dome.RemoteAddr when Dome runs without a trusted proxy in front of it: it reads the TCP peer address, which cannot be spoofed by request headers. If you run behind a proxy (Cloudflare, a load balancer, etc.), supply your own resolver backed by a trusted-proxy strategy so a malicious client can't forge its IP via headers.

Block AI Scrapers

I've manually collected this list of AI scrapers that Digital Dome uses to protect your website. Any requests that contain any of the values below will be blocked before they reach your application.

In addition, since every legitimate browser includes a User-Agent value, requests with an empty value are also automatically blocked.

Amazonbot
anthropic-ai
AdsBot-Google
Applebot
Applebot-Extended
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
ChatGPT
ChatGPT-User
Claude
ClaudeBot
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
FacebookExternalHit
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
magpie-crawler
Meta-ExternalAgent
meta-externalagent
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
PerplexityBot
PetalBot
Quora-Bot
Scrapy
TurnitinBot
Twitterbot
YaK
Yandex
YouBot

Block Commonly Scanned Paths

Like the list of User-Agent strings above, Digital Dome also maintains a list of URL paths that are commonly scanned for vulnerabilities. For instance, default WordPress directories are commonly scanned as a precursor to hacking a server. Since Digital Dome is built specifically for Go applications, there's no problem blocking every directory related to WordPress.

Block Probes That Hide In The Query String

Some frameworks expose the same endpoint two ways. WordPress serves its entire REST API from ?rest_route= when a site uses plain permalinks, so a scanner blocked at /wp-json simply asks for /?rest_route=/batch/v1 instead, and the path blocklist never sees it.

Digital Dome blocks a request that carries any query parameter in the BlockedQueryParams list. Matching is by parameter name, and exactly, unlike every other list here. A query value is attacker-controlled, and a substring rule over values would fire on ordinary traffic — a link preview or an oEmbed lookup whose target URL merely mentions the pattern.

domeConfig := dome.New(                       // Create the digital dome shield
    dome.RemoteAddr,                          // Resolve the "real" client IP for each request
    dome.BlockQueryParams("rest_route", "XDEBUG_SESSION_START"), // Replace the default list
)

Block Malicious IP Addresses

Digital Dome tracks errors generated by your application and returned through the middleware.

If a specific IP address generates too many errors in a designated timespan, future requests from that IP will be blocked before they reach your application.

The length of time an IP is blocked grows with the number of bad requests they make — staying short for the first several errors, then increasing up to a maximum of two hours per request. So, honest mistakes will heal quickly and automatically, and script-kiddie scanners will ban themselves into oblivion.

By default, Digital Dome counts all StatusForbidden responses towards the quota for any IP address, and begins blocking all traffic after 5 forbidden requests within one minute. You can calibrate the kinds of responses that trigger this behavior using the BlockStatusCodes() option.

domeConfig := dome.New(             // Create the digital dome shield
    dome.RemoteAddr,             // Resolve the "real" client IP for each request
    dome.BlockStatusCodes(404),     // Choose status codes to trigger blocking behavior
)

Configuration Options

Digital Dome uses optional functional parameters to configure its behavior. You can apply these at startup when you create the dome, or afterward when your app is running.

dd := dome.New(                         // Apply options at creation time
    dome.RemoteAddr,                 // Resolve the "real" client IP for each request (REQUIRED)
    dome.BlockKnownAIBots(),            // Block AI bots only (instead of the default bad-bot list)
    dome.BlockPaths(dome.BlockedPaths...), // Block the built-in list of commonly scanned paths
    dome.BlockCache(2048),              // Expand the size of the blocked IP cache
)

dd.With(dome.BlockStatusCodes(404)) // Or apply other options later

Option	Description
Block User Agents
BlockUserAgents(strings...)	Digital Dome can block requests based on any number of provided User-Agent strings. It uses an efficient Aho-Corasick string matching algorithm from CloudFlare to perform this operation quickly.
BlockKnownAIBots()	Digital Dome maintains a list of known AI bots that it can compare against each request's User-Agent
BlockKnownBadBots()	(DEFAULT) Digital Dome maintains a list of known bad actors that it can compare against each request's User-Agent. This includes all of the AI bots listed above, plus several hundred more non-search-engine user agents that are used for scraping your website.
Block Paths
BlockPaths(strings...)	Digital Dome can block requests based on any number of provided path names. As with User-Agent blocking, it uses an efficient Aho-Corasick string matching algorithm from CloudFlare to perform this operation quickly. By default, New() blocks the built-in BlockedPaths list of commonly scanned paths.
SoftBlockPaths(strings...)	Soft-blocked paths are not blocked outright. Instead, requests are allowed through, but if they return a client (4xx) error, they count toward the requesting IP's block quota. By default, New() soft-blocks the built-in SuspiciousPaths list.
Block Query Parameters
BlockQueryParams(strings...)	Digital Dome blocks any request carrying one of these query parameter names, whatever path it is aimed at. Unlike the lists above, this one matches names exactly rather than by substring, because a query value is attacker-controlled. By default, New() blocks the built-in BlockedQueryParams list.
Log Errors
LogDatabase(data.Collection)	Digital Dome can log failed requests to a database if a data.Collection is provided to this Option (see Storage below)
LogStatusCodes(ints...)	Customize which status codes are logged using this Option. The default is StatusNotFound (404).
Block Malicious Requests
BlockStatusCodes(ints...)	Digital Dome can track when requests trigger specific errors (for example, StatusForbidden) and block all requests from that IP address. See "Block Malicious IP Addresses" above for details. The default is StatusForbidden (403).
BlockCache(capacity)	This option sets the capacity of the blocked IP address cache. Default is 1024 IP addresses.

Custom Router Middleware

Digital Dome is built to work with any Go HTTP Router library or framework. There is currently one adapter, made for labstack echo. Middleware adapters are very easy to make, so if your router is not listed below, please file an issue to get one made.

Custom Database Storage

Digital Dome uses a storage adapter to write log files to a database. Currently, I have only written an adapter for MongoDB, but adapters can be written for any database, so please file an issue to make one for your chosen database.

To begin writing log files, simply pass a data.Collection into the LogDatabase option, and Digital Dome will use it whenever it needs to log a request.

db, err := mongodb.Connect(...)          // Connect to mongodb
collection := mongodb.                   // Wrap mongo with a data.Collection adapter
    NewSession(db).
    Collection("DigitalDome_LogFiles") 

domeConfig := dome.New(                  // Create the digital dome shield
    dome.RemoteAddr,                  // Resolve the "real" client IP for each request
    dome.LogDatabase(collection),        // Use this database collection to log errors
    dome.LogStatusCodes(404),            // Choose status codes to log
)  

Banner Image Used Without Permission

Image Credits. None. The banner image was an AI-generated using https://www.freepik.com. AI images cannot be copyrighted or owned by anyone. Savor the delicious irony.

No Warranty

Digital Dome is a solid defense against many AI scanners and malicious bots. But it is not a complete Web Application Firewall, and it comes with NO WARRANTIES or GUARANTEES against any particular online threats. Be mindful.

Pull Requests Welcome

The battle against rampant AIs is ongoing, and this module will require constant upkeep to stay current. I will happily accept any and all experience reports, use cases, and contributions that will make Digital Dome better. If you have an idea for making Digital Dome better, file an issue or send in a pull request. We're all in this together! ✨