command module
Version: v0.0.0-...-ba9c0dd Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Dec 22, 2015 License: MIT Imports: 5 Imported by: 0


Software License GoDoc

A golang crawler


Getting started

You can grab this package with the following command:

go get

And then build it:

cd ${GOPATH%/}/src/
go build


A main file is provided to suit most needs. This is a web crawler that saves crawled HTML content to the disk, after removing the HTML tags (almost all of them) and keeping only the "meaningful" content. I highly advise you to custom this main file or to create your own fetcher or processor to suit your need. You can then reuse the crawler to do the work concurrently.

From the -h flag:

Usage of ./crawler:
  -start string
        address to start from (default "")
  -levels int
        depth of the web crawl (default 50)
  -savePath string
        where to saved crawled resources (default "/tmp/crawl/")
  -concurrentFetchers int
        number of fetchers to run concurrently (default 30)
        keep the fragment part of an URL. Example #top
        keep the query part of an URL. Example ?foo=bar
        do not crawl resources that are stored on another domain. (default true)


Some components are abstracted behind interfaces, or contracts as I like to call them. We have got 2 of them:

  • A Fetcher is in charge of fetching a resource and finding children resources to explore after. The Fetcher is also in charge of determining if a children resource should be explored or not
  • A Processor is in charge of processing a fetched resource

If you want to customise things, you will need to implement at least one of these 2 interfaces and write a custom main file. Of course you can use the provided main file and implementations to get you started.


The Go Gopher

There is no documentation for this package.

Source Files


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL