webxcrawler_go

README

WebXCrawler

WebXCrawler is a fast static crawler to crawl a website and get all the links.

Installation

If you have Go installed and configured (i.e. with $GOPATH/bin in your $PATH):

▶ go get -u github.com/shivamsaraswat/webxcrawler_go

Using the WebXCrawler

To run the WebXCrawler on a website, use the '-u' flag and provide the URL as an argument:

▶ ./webxcrawler_go -u URL

For an overview of all commands use the following command:

██╗    ██╗███████╗██████╗ ██╗  ██╗ ██████╗██████╗  █████╗ ██╗    ██╗██╗     ███████╗██████╗
██║    ██║██╔════╝██╔══██╗╚██╗██╔╝██╔════╝██╔══██╗██╔══██╗██║    ██║██║     ██╔════╝██╔══██╗
██║ █╗ ██║█████╗  ██████╔╝ ╚███╔╝ ██║     ██████╔╝███████║██║ █╗ ██║██║     █████╗  ██████╔╝
██║███╗██║██╔══╝  ██╔══██╗ ██╔██╗ ██║     ██╔══██╗██╔══██║██║███╗██║██║     ██╔══╝  ██╔══██╗
╚███╔███╔╝███████╗██████╔╝██╔╝ ██╗╚██████╗██║  ██║██║  ██║╚███╔███╔╝███████╗███████╗██║  ██║
 ╚══╝╚══╝ ╚══════╝╚═════╝ ╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝ ╚══╝╚══╝ ╚══════╝╚══════╝╚═╝  ╚═╝
                                            Coded with Love by Shivam Saraswat (@cybersapien)
WebXCrawler is a fast static crawler to crawl a website and get all the links.

Usage:
  ./webxcrawler_go [flags]

Flags:
   -u, -url string     target url
   -d, -depth int      maximum depth to crawl (default 2)
   -t, -threads int    number of threads to use (default 10)
   -o, -output string  file to write output to
   -V, -version        show webxcrawler version number and exit