sicra

Documentation

Index

• func Crawler(scrapURL, userAgent, allowDomain, baseAuth, uriFilter, noIndexRule string, ...) *scrapeURL
• func GenerateSiteMap(fileName string, urls []string) error
• func GenerateTxt(fileName string, urls []string) error

Functions

func Crawler

func Crawler(
	scrapURL, userAgent, allowDomain, baseAuth, uriFilter, noIndexRule string,
	paralScan, maxDepth, timeoutResp int,
	delay int64,
	asyncScan, addError, skipNoIndex, verbose bool,
) *scrapeURL

Crawler takes as input the parameters to scan. Returns URL scanning statistics and a list of links for sitemap generation. By default, pages containing 'meta name="robots" content="noindex,nofollow"' are ignored

func GenerateSiteMap

func GenerateSiteMap(fileName string, urls []string) error

GenerateSiteMap will write the crawled url to given file

func GenerateTxt

func GenerateTxt(fileName string, urls []string) error

GenerateTxt generate txt file for error list url or skiped noindex