Simple web crawler
Please take a look at the tests to validate the operation of the crawler. I've created a simple website and hosted it locally for the tests. This is used to test the Crawl(..)
A lot need be done to actually crawl a website as it needs handling various corner cases and specified link variations in web pages.