crawler

Documentation

Overview

Command crawler is the Quert command-line web crawler.

It crawls from seed URLs, following discovered links breadth-first (bounded by max depth and max pages), respecting robots.txt and per-host rate limits, and writes each extracted page as one JSON object per line (JSONL) to the output.