botrate

README

botrate

High-performance, SEO-friendly rate limiter for Go applications

---

Overview

BotRate is a high-performance rate limiter designed specifically for modern web applications. Unlike traditional rate limiters that blindly block high-frequency IPs, botrate intelligently distinguishes between malicious scrapers and verified bots (Search Engines, AI Crawlers, etc.).

This ensures your site remains protected from abuse without sacrificing your SEO rankings or AI knowledge base presence.

Features

• 🛡️ Smart Bot Detection - Uses knownbots library for verified bot identification (Googlebot, Bingbot, GPTBot, ClaudeBot, etc.)
• 🔒 Behavior Analysis - Asynchronous IP+URL pattern detection to identify scrapers
• ⚡ High Performance - <2μs hot path latency, only rate limits blacklisted IPs
• 💾 Memory Efficient - Only creates token buckets for blacklisted IPs
• 🎯 Flexible - HTTP callback handling is left to caller for maximum compatibility
• 🔄 Graceful Shutdown - Proper resource cleanup with Close() method

Installation

go get github.com/cnlangzi/botrate

Quick Start

Basic Usage

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"github.com/cnlangzi/botrate"
	"golang.org/x/time/rate"
)

func main() {
	limiter, err := botrate.New(
		// Rate limiting for blacklisted IPs only
		botrate.WithLimit(rate.Every(10*time.Minute)),

		// Behavior analysis
		botrate.WithAnalyzerWindow(time.Minute),
		botrate.WithAnalyzerPageThreshold(50),
		botrate.WithAnalyzerQueueCap(10000),
	)
	if err != nil {
		log.Fatalf("Failed to create limiter: %v", err)
	}
	defer limiter.Close()

	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ua := r.UserAgent()
		ip := extractIP(r)

		if !limiter.Allow(ua, ip) {
			http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
			return
		}

		w.Write([]byte("Hello, World!"))
	})

	fmt.Println("Server started on :8080")
	http.Handle("/", handler)
	http.ListenAndServe(":8080", nil)
}

// extractIP extracts the real client IP from the request.
func extractIP(r *http.Request) string {
	if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
		return strings.TrimSpace(strings.Split(xff, ",")[0])
	}
	if xri := r.Header.Get("X-Real-IP"); xri != "" {
		return xri
	}
	return r.RemoteAddr
}

Using Wait Method (Blocking)

For scenarios where you want to wait instead of immediately rejecting:

handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	if err := limiter.Wait(r.Context(), ua, ip); err != nil {
		http.Error(w, err.Error(), http.StatusTooManyRequests)
		return
	}
	w.Write([]byte("Hello!"))
})

Middleware Pattern

func BotRateMiddleware(l *botrate.Limiter) func(http.Handler) http.Handler {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			if !l.Allow(r.UserAgent(), extractIP(r)) {
				http.Error(w, "Rate limit exceeded", http.StatusTooManyRequests)
				return
			}

			next.ServeHTTP(w, r)
		})
	}
}

// Usage
http.Handle("/", BotRateMiddleware(limiter)(myHandler))

API Reference

Options

Option	Description	Default
WithLimit(rate.Limit)	Requests per second for blocked IPs	rate.Every(10*time.Minute)
WithAnalyzerWindow(time.Duration)	Analysis window duration	5*time.Minute
WithAnalyzerPageThreshold(int)	Max distinct pages threshold	50
WithAnalyzerQueueCap(int)	Event queue capacity	10000
WithKnownbots(*knownbots.Validator)	Custom knownbots validator	nil (use default)

Methods

Allow(ua, ip string) bool

Non-blocking check if the request should proceed. Returns true if allowed, false if blocked.

Bot Detection Logic:

• Verified bot (StatusVerified): ✅ Allow immediately
• RDNS lookup failed (StatusPending): ✅ Allow, retry verification next time
• Fake bot (StatusFailed): ❌ Block immediately
• Normal user: Continue to analyzer and blocklist check

allowed := limiter.Allow(ua, ip)
if !allowed {
    // Request was blocked (fake bot or blacklisted IP)
}

result := limiter.Allow(ua, ip) if !result.Allowed { // Handle denial }

#### `Wait(ctx context.Context, ua, ip string) error`

Blocks until the request is allowed or the context ends. Returns `nil` if allowed, `ErrLimit` if blocked.

**Bot Detection Logic:**
- **Verified bot** (StatusVerified): ✅ Allow immediately
- **RDNS lookup failed** (StatusPending): ✅ Allow, retry verification next time
- **Fake bot** (StatusFailed): ❌ Block immediately
- **Normal user**: Continue to analyzer and blocklist check

```go
err := limiter.Wait(ctx, ua, ip)
if err != nil {
    // Handle denial (ErrLimit) or context cancellation
}

Close()

Gracefully shuts down the limiter and releases resources. Always call this when the limiter is no longer needed.

limiter, err := botrate.New(...)
if err != nil {
    log.Fatalf("Failed to create limiter: %v", err)
}
defer limiter.Close()

How It Works

Request
  │
  ▼
┌─────────────────────────────────────┐
│ 1. KnownBots Verification           │  Hot path: <1μs
│    - Check if UA matches known bot  │
│    - Verified → Allow immediately   │
│    - RDNS failed → Allow, retry     │
│    - Fake bot → Block immediately   │
└─────────────────────────────────────┘
  │
  ▼ (only for normal users)
┌─────────────────────────────────────┐
│ 2. Blocklist Check                  │  Atomic read: <100ns
│    - Check if IP is blacklisted     │
│    - Not blocked → Record + Allow   │
│    - Blocked → Rate limit           │
└─────────────────────────────────────┘
  │
  ▼ (async)
┌─────────────────────────────────────┐
│ 3. Behavior Analysis                │  Background worker
│    - Record IP+URL combination      │
│    - Bloom filter deduplication     │
│    - Visit counter increment        │
│    - Threshold exceeded → Block     │
└─────────────────────────────────────┘

Key Design Decisions

1. Fake bots blocked immediately - Known bot UAs with failed verification are blocked without rate limiting
2. RDNS lookup failures are tolerated - Failed DNS lookups allow the request (will retry next time)
3. Verified bots bypass everything - Googlebot, Bingbot, etc. are allowed without rate limiting
4. Normal users go through analyzer - Behavior analysis only applies to regular users
5. Async behavior analysis - Request processing is never blocked by analysis

Performance

Scenario	Latency	Memory
Normal user	<1.5μs	0 bytes
Verified bot	<1μs	0 bytes
Blacklisted IP	<2μs	~200 bytes/IP
Fake bot	<1μs	0 bytes

Total memory budget: <5MB (Bloom: 1MB + Counter: 1MB + Blacklisted IPs: variable)

Benchmark Results

$ go test -run=^$ -bench=. -benchmem -cpu=1,4,8

Key metrics to monitor:

• ns/op - Nanoseconds per operation (lower is better)
• B/op - Bytes allocated per operation (should be 0 for hot path)
• allocs/op - Allocations per operation (should be 0 for hot path)

Error Handling

Errors

var ErrLimit = context.DeadlineExceeded

Check errors with:

if errors.Is(err, botrate.ErrLimit) {
    // Request was denied (fake bot or blacklisted IP)
}

Denial Reasons

Allow() returns false when:

1. Fake bot - Known bot UA (e.g., "GPTBot") but IP verification failed
2. Blacklisted IP - IP was flagged by behavior analysis

Wait() returns ErrLimit when:

1. Fake bot - Blocked immediately
2. Rate limited - Normal user on blocklist hitting rate limit

allowed := limiter.Allow(ua, ip)

if !allowed {
    // Request was denied
    // - Fake bot: blocked immediately
    // - Blacklisted IP: rate limited
}

Configuration Examples

Strict Rate Limiting

limiter, err := botrate.New(
	botrate.WithLimit(rate.Every(10*time.Minute)),
)
if err != nil {
    log.Fatalf("Failed to create limiter: %v", err)
}

Aggressive Bot Detection

limiter, err := botrate.New(
	botrate.WithAnalyzerWindow(30*time.Second),
	botrate.WithAnalyzerPageThreshold(20),
)
if err != nil {
    log.Fatalf("Failed to create limiter: %v", err)
}

High-Throughput Configuration

limiter, err := botrate.New(
	botrate.WithAnalyzerWindow(10*time.Minute),
	botrate.WithAnalyzerPageThreshold(100),
	botrate.WithAnalyzerQueueCap(50000),
)
if err != nil {
    log.Fatalf("Failed to create limiter: %v", err)
}

Custom KnownBots Validator

// Create custom validator with specific configuration
customKB, err := knownbots.New(
	knownbots.WithRoot("./custom-bots"),
	knownbots.WithSchedulerInterval(12*time.Hour),
)
if err != nil {
    log.Fatalf("Failed to create validator: %v", err)
}

// Use custom validator
limiter, err := botrate.New(
	botrate.WithKnownbots(customKB),
	botrate.WithLimit(rate.Every(5*time.Minute)),
)
if err != nil {
    log.Fatalf("Failed to create limiter: %v", err)
}

Architecture

botrate/
├── limiter.go          # Main Limiter type and API
├── botrate.go          # Error definitions
├── config.go           # Configuration struct
├── options.go          # Functional options
├── analyzer/           # Behavior analysis engine
│   ├── analyzer.go    # Core analyzer with worker
│   ├── bloom.go       # Double-buffered Bloom filter
│   └── counter.go     # LRU visit counter (O(1))
└── example/
    └── main.go        # Working example

Development

Makefile Commands

A Makefile is provided for common development tasks:

make help          # Show available commands
make test          # Run all tests (short + race)
make test-short    # Run short tests (fast)
make test-race     # Run tests with race detector
make test-coverage # Run tests with coverage report
make bench         # Run benchmarks (1 and 4 CPUs)
make bench-all     # Run all benchmarks (1, 4, 8 CPUs)
make build         # Build the project
make clean         # Clean build artifacts

Examples

# Run all tests
make test

# Run benchmarks
make bench

# Generate coverage report
make test-coverage

# Run benchmarks with detailed output
make bench-all

Contributing

Contributions are welcome! Please read our contributing guidelines before submitting PRs.

1. Fork the repository
2. Create a feature branch
3. Add tests for your changes
4. Ensure all tests pass: make test
5. Run benchmarks: make bench
6. Submit a pull request

License

MIT License - see LICENSE for details.

Acknowledgments

• knownbots - Bot detection library
• bloom - Bloom filter implementation
• golang/x/time - Token bucket rate limiter

Documentation

Index

• Variables
• type Config
• type Limiter
  • func New(opts ...Option) (*Limiter, error)
  • func (l *Limiter) Allow(ua, ip string) (allowed bool, reason Reason)
  • func (l *Limiter) Close()
  • func (l *Limiter) Wait(ctx context.Context, ua, ip string) (err error, reason Reason)
• type Option
  • func WithAnalyzerPageThreshold(threshold int) Option
  • func WithAnalyzerQueueCap(cap int) Option
  • func WithAnalyzerWindow(window time.Duration) Option
  • func WithKnownbots(kb *knownbots.Validator) Option
  • func WithLimit(limit rate.Limit) Option
• type Reason

Variables

var (
	DefaultLimit         = rate.Every(10 * time.Minute) // Very strict: 1 request per 10 min
	DefaultWindow        = 5 * time.Minute
	DefaultPageThreshold = 50
	DefaultQueueCap      = 10000
)

Default configuration values.

var ErrLimit = context.DeadlineExceeded

ErrLimit is returned when the request is rate limited.

Types

type Config

type Config struct {
	Limit         rate.Limit
	Window        time.Duration
	PageThreshold int
	QueueCap      int
}

Config holds core configuration.

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter provides bot-aware rate limiting.

func New

func New(opts ...Option) (*Limiter, error)

New creates a new rate limiter with default config and applies options.

func (*Limiter) Allow

func (l *Limiter) Allow(ua, ip string) (allowed bool, reason Reason)

Allow reports whether the request should proceed. Returns:

• allowed: true if allowed, false if blocked
• reason: the reason for blocking when allowed is false

func (*Limiter) Close

func (l *Limiter) Close()

Close gracefully shuts down the limiter and releases resources.

func (*Limiter) Wait

func (l *Limiter) Wait(ctx context.Context, ua, ip string) (err error, reason Reason)

Wait blocks until the request is allowed or the context is canceled. Returns:

• err: nil if allowed, otherwise the blocking error (context canceled/timeout or ErrLimit)
• reason: the reason for blocking (ReasonFakeBot or ReasonRateLimited)

type Option

type Option func(*Limiter)

Option is a functional option for configuring Limiter.

func WithAnalyzerPageThreshold

func WithAnalyzerPageThreshold(threshold int) Option

WithAnalyzerPageThreshold sets max distinct pages threshold.

func WithAnalyzerQueueCap

func WithAnalyzerQueueCap(cap int) Option

WithAnalyzerQueueCap sets event queue capacity.

func WithAnalyzerWindow

func WithAnalyzerWindow(window time.Duration) Option

WithAnalyzerWindow sets analysis window duration.

func WithKnownbots

func WithKnownbots(kb *knownbots.Validator) Option

WithKnownbots implants a custom knownbots.Validator.

func WithLimit

func WithLimit(limit rate.Limit) Option

WithLimit sets events per second for rate limiting.

type Reason

type Reason string

Reason represents the reason for rate limiting.

const (
	// ReasonFakeBot indicates the request was blocked because
	// the bot verification failed (fake bot or unknown status).
	ReasonFakeBot Reason = "fake_bot"

	// ReasonRateLimited indicates the request was blocked because
	// the IP was flagged by behavior analysis.
	ReasonRateLimited Reason = "rate_limited"
)