crawl

README

crawl

crawl is a tool for crawling and understanding websites. It has two goals.

First, crawling with local machines is often resource-intensive. This applies even more so to the analysis of collected data. Crawl assumes that you intend to do analysis on other, non-local platform. Specifically, it's tailored to use with BigQuery.

To that end, crawl maintains the absolute minimum state necessary to complete a crawl. A crawl of a 10,000 page site might use < 20 MB RAM. Crawling 1,000,000 pages might use < 1 GB RAM.

Second, most crawling tools either constrain you to pre-defined analysis, or only allow you to export flattened data. Or both. But as a graph structure, flattening a crawl limits the types of analysis the consumer of the data can do themselves.

crawl produces nested data. One row is generated for each unique crawlable URL encountered. That row contains nested information about one-to-many relationships. All links that appear on the page are embedded in this record. All hreflang relationships described on the page are embedd in this record. This allows for complete flexibility of analysis for the end user.

Installation

go get -u github.com/benjaminestes/crawl/...

Use

Site mode

crawl site my_config.json >out.txt

Running crawl sends a row of newline-delimited JSON to os.Stdout. In the example above, we're sending that to out.txt.

List mode

crawl list my_config.json <list.txt >out.txt

Configuration

The repository includes an example config.json file. This lists all of the available options with reasonable default values. In particular, you should think about these options:

• Start: The fully-qualified URL from which you want to start crawling. Unlike other crawlers you may have used, this choice does not affect the scope of the crawl.
• Include: An array of regular expressions that a URL must match in order to be crawler. If there is no valid Include expression, nothing will be crawled.
• Exclude: An array of regular expressions that filter the URLs to be crawled.

To summarize, a URL must match at least one Include but none of the Exclude rules in order to be crawled.

If you're crawling in list mode, the MaxDepth, Include, and Exclude options will be ignored.

Use with BigQuery

Run crawl schema >schema.json to get a BigQuery-compatible schema definition file. The file is automatically generated (via go generate) from the structure of the result object generated by the crawler, so it should always be up-to-date.

If you find an incompatibility between the output schema file and the data produced from a crawl, please flag as a bug on GitHub.

License

MIT