crawler

module

v0.0.0-...-091bbc5 Latest Latest Go to latest Published: Sep 19, 2019 License: MIT

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

README ¶

Simple web crawler

Please take a look at the tests to validate the operation of the crawler. I've created a simple website and hosted it locally for the tests. This is used to test the Crawl(..)

A lot need be done to actually crawl a website as it needs handling various corner cases and specified link variations in web pages.

Directories ¶

Path	Synopsis
src
crawler

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL