crawler

module
Version: v0.0.0-...-091bbc5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 19, 2019 License: MIT

README

Simple web crawler

Please take a look at the tests to validate the operation of the crawler. I've created a simple website and hosted it locally for the tests. This is used to test the Crawl(..)

A lot need be done to actually crawl a website as it needs handling various corner cases and specified link variations in web pages.

Directories

Path Synopsis
src

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL