github.com/

gocrawl is a polite, slim and concurrent web crawler written in Go.

Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document.

Package purell offers URL normalization as described on the wikipedia page: http://en.wikipedia.org/wiki/URL_normalization

Package pat implements a simple URL pattern muxer

Package negroni is an idiomatic approach to web middleware in Go.

Package redis is a client for the Redis database.

A go port of the ruby dotenv library (https://github.com/bkeepers/dotenv) Examples/readme can be found on the github page at https://github.com/joho/godotenv The TL;DR is that you make a .env file that looks something like SOME_ENV_VAR=somevalue and then in your go code you can call godotenv.Load() and all the env vars declared in .env will be avaiable through os.Getenv("SOME_ENV_VAR")

Package gopengraph For a full guide visit https://github.com/roperzh/gopengraph

The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/wc/robots.html with various extensions.