crawler

Documentation

Index

• func Crawl(endpoint config.Endpoint, nThreads int, depth uint8)
• func Do(work chan *Page, results chan *Result)
• type Page
• type Result

Functions

func Crawl

func Crawl(endpoint config.Endpoint, nThreads int, depth uint8)

Crawl spawns worker threads to make requests to the endpoint. It takes the results and appends them to the set of links, making sure that the depth does not exceed the supplied parameter. After 10 seconds, if the set does not enlarge, we stop crawling.

func Do

func Do(work chan *Page, results chan *Result)

Do makes an HTTP request to the URL supplied in the work pool, parses it if it follows the HTML schema, looks for "form" and "a" tags and sends back the URLs associated respectively.

Types

type Page

type Page struct {
	Location string
	Depth    uint8
}

Page represents a page that is crawled. It includes the depth at which it was discovered.

type Result

type Result struct {
	*Page
	Err error
}

Result is returned by the Do function. It contains an embedded pointer to a page that gets discovered with its respective depth. The Err field may indicate any error associated while crawling.