Imports github.com/temoto/robotstxt-go golang.org/x/net/html github.com/mohae/firkin/queue Standard library imports bytes errors fmt io math/rand net/http net/url sort strings sync time