crawlora-deadweb

README

crawlora-deadweb

Is a domain genuinely dead — or just blocking your bot? Tell them apart from one passive probe.

crawlora-deadweb is a small, dependency-free CLI (and Go library) that probes a domain and classifies it alive / redirect / blocked / dead, with the reason. It tells a domain that's gone (no DNS, nothing listening) from one that's alive but refusing automated clients (403 / 429 / anti-bot). Most "dead link" checkers conflate the two — and that's exactly the error behind the myth that ~27% of the web is dead. It isn't.

It is a classifier, not an unblocker. It does a DNS lookup, a TCP connect, and one honest GET /, reads the response, and labels it. It never logs in, submits a form, solves a challenge, or tries to defeat anything.

Classification runs locally and open, from the public response. For the measured browser-fingerprint arm — re-probing a blocked domain with a real Chrome TLS/JA3 client across the proxied fleet to see which "blocked" sites are actually reachable — add --browser, which calls Crawlora's hosted engine.

This powers, and is the open companion to, the Dead-Web Index — a reachability census of the top 10 million domains that found ~14% genuinely dead, not the usual 27.6% (most "dead" is anti-bot blocking or a served error).

What the labels mean

• alive — a usable HTTP response (2xx, or a 4xx/5xx the server answered — a response isn't death).
• redirect — ended on an unresolved redirect.
• blocked — the host is up but won't serve us: anti-bot / auth / rate-limit, or it accepts a TCP connection but won't complete HTTP (tarpit / strict TLS).
• dead — no DNS resolution, a refused/reset connection, or nothing listening. Genuinely gone.

Install

# from source (Go 1.23+)
go install github.com/Crawlora-org/crawlora-deadweb@latest

# or clone + build
git clone https://github.com/Crawlora-org/crawlora-deadweb
cd crawlora-deadweb && go build -o crawlora-deadweb .

Prebuilt Linux / macOS / Windows binaries are published via GitHub Releases.

Usage

crawlora-deadweb [flags] <domain> [domain...]

$ crawlora-deadweb grooveshark.com reuters.com
grooveshark.com
outcome dead
reason  dns_failed — genuinely unreachable

reuters.com
outcome blocked
reason  forbidden (403) — alive but refusing this client
        (run with --browser for the measured browser-fingerprint arm)

--json emits NDJSON (one compact object per line) — pipe straight into jq -c or a data pipeline.

Batch / pipelines. Pass many domains as args, or pipe a list on stdin (one per line; blank lines and #-comments ignored). Domains are probed in parallel (--concurrency, default 8):

cat domains.txt | crawlora-deadweb --json --concurrency 50 > results.ndjson
printf 'grooveshark.com\nexample.com\n' | crawlora-deadweb

Each JSON record matches the open dataset schema: domain, tld, outcome, reason, first_status, final_status, final_url, scheme, hops, parked.

The browser arm (optional, hosted)

The local probe is a polite HTTP request from your IP, so "blocked" is an upper bound — a vendor refusing a datacenter client ≠ the site being unreachable. For the measured tier — what actually gets through with a real browser fingerprint and the proxied fleet — add --browser:

export CRAWLORA_API_KEY=...   # get one at https://crawlora.net
crawlora-deadweb --browser reuters.com
# outcome blocked  reason forbidden (403) ← local
# ── measured (Crawlora API) ──
# browser arm: reachable ✓           ← live measurement

Flags

Flag	Default	Description
--json	false	Output NDJSON (one object per line).
--browser	false	Also fetch the measured browser-fingerprint arm from the Crawlora API (needs a key).
--timeout	20s	Per-attempt timeout for the local probe.
--retries	1	Retries on transient (timeout / reset) failures.
--concurrency	8	Domains probed in parallel (batch mode).
--api-key	$CRAWLORA_API_KEY	Crawlora API key (for --browser).
--api-base	$CRAWLORA_API_BASE or https://api.crawlora.net/api/v1	API base URL.
--user-agent	Chrome UA	User-Agent for the local probe.
--version		Print version and exit.

How classification works

A DNS lookup across 8 public resolvers (IPv4-first, so an IPv4-only vantage doesn't dead-label a dual-stack host), then HTTPS-first with HTTP fallback, redirects followed (max 10), and one retry on transient failures. When HTTP doesn't answer, a raw TCP-connect check on :443/:80 separates "up but unresponsive" (blocked) from "gone" (dead). The honest rule: a received status is never dead — 404/410/5xx → alive (the server answered), 403/429/anti-bot (incl. the non-standard 999) → blocked, only no-DNS / refused / nothing-listening → dead.

It's a lower bound on "alive": homepage-level, from your vantage, passive. Deep pages and residential vantages reach more — that's what --browser measures.

Use as a library

import "github.com/Crawlora-org/crawlora-deadweb/classify"

res := classify.Inspect(ctx, "example.com", classify.Options{})
fmt.Println(res.Outcome, res.Reason) // alive ok

Scope & ethics

Each probe is an unauthenticated GET / to a public homepage (plus DNS/TCP checks) — the same request any browser makes. No authentication is bypassed, no login-only or robots.txt-disallowed paths are fetched, and no page content is republished. It's a classifier, not a bypass — respect each site's Terms of Service and robots.txt, and only probe what you're authorized to.

License

MIT. Built by Crawlora. See the open Dead-Web Index and dataset.

Documentation

Overview

Command crawlora-deadweb classifies a domain's reachability as alive / redirect / blocked / dead — telling a genuinely gone host from one that is alive but blocking your bot. It is the open companion to the Crawlora Dead-Web Index (https://crawlora.net/dead-web-index).