detect

Documentation

Overview

Package detect classifies a fetched page as usable, blocked, a captcha or challenge, or suspiciously empty.

The generic detectors here look at the status code, at markers common to the big bot-protection vendors, and at body size. They are heuristics: they will miss a site's own block page, and that is what custom Detectors are for. A consumer chains its site-specific detectors in front of Default().

Index

• Constants
• func NeedsJS(p Page) bool
• type Chain
  • func (c Chain) Detect(p Page) Outcome
• type Detector
  • func Default() Detector
  • func Markers() Detector
  • func MarkersFrom(m MarkerSet) Detector
  • func ShortBody(minBytes int) Detector
  • func Status() Detector
• type Func
  • func (f Func) Detect(p Page) Outcome
• type MarkerSet
  • func DefaultMarkers() MarkerSet
• type Outcome
  • func (o Outcome) String() string
• type Page

Constants

const DefaultMinBody = 256

DefaultMinBody is the body size, in bytes after trimming whitespace, below which ShortBody calls a successful HTML page Empty.

Functions

func NeedsJS

func NeedsJS(p Page) bool

NeedsJS reports whether a successful HTML page looks like it has to be rendered by a browser before it holds any content: an application shell (scripts and almost no text), or a page asking for JavaScript with little else on it.

Types

type Chain

type Chain []Detector

Chain runs detectors in order and returns the first outcome that is not OK. An empty chain returns OK.

func (Chain) Detect

func (c Chain) Detect(p Page) Outcome

Detect implements Detector.

type Detector

type Detector interface {
	Detect(p Page) Outcome
}

Detector classifies a page.

func Default

func Default() Detector

Default is the generic chain: challenge markers first, so a 403 that carries a captcha reads as a captcha, then status codes, then a short body.

func Markers

func Markers() Detector

Markers returns a detector using DefaultMarkers.

func MarkersFrom

func MarkersFrom(m MarkerSet) Detector

MarkersFrom returns a detector that looks for the given markers. Captcha markers are checked before block markers.

func ShortBody

func ShortBody(minBytes int) Detector

ShortBody reports Empty for a 2xx HTML (or untyped) response whose body, trimmed of whitespace, is shorter than min bytes. Other content types are left alone because a short JSON or text answer is often legitimate.

func Status

func Status() Detector

Status reports Blocked for 403, 429 and 999 (the code some sites use for "go away"). Server errors are not blocks; the retry policy handles them.

type Func

type Func func(p Page) Outcome

Func adapts a function to a Detector.

func (Func) Detect

func (f Func) Detect(p Page) Outcome

Detect implements Detector.

type MarkerSet

type MarkerSet struct {
	CaptchaTitles []string
	BlockTitles   []string
	CaptchaTokens []string
	BlockTokens   []string
	// SmallPage is the visible-text length, in bytes, under which body
	// tokens are trusted on a 2xx page.
	SmallPage int
}

MarkerSet is the vocabulary Markers looks for. Every string is matched case-insensitively.

Title phrases are decisive: a page titled "Just a moment..." is a challenge whatever else it holds. Body tokens are not, because the same scripts and widgets appear on ordinary pages (a reCAPTCHA on a login form, Cloudflare's bot-management script on every page it fronts); they count only when the page is also an error or carries little visible text.

func DefaultMarkers

func DefaultMarkers() MarkerSet

DefaultMarkers covers Cloudflare, Akamai, Imperva/Incapsula, DataDome, PerimeterX/HUMAN, reCAPTCHA, hCaptcha and Turnstile.

type Outcome

type Outcome int

Outcome is the classification of one page.

const (
	OK Outcome = iota
	Blocked
	Captcha
	Empty
)

The outcomes, in no particular order of severity.

func (Outcome) String

func (o Outcome) String() string

String returns the outcome's lowercase name.

type Page

type Page struct {
	URL    string
	Status int
	Header http.Header
	Body   []byte
}

Page is what a Detector looks at.