crawler

Documentation

Index

• func Crawler(target string, txt string, html string, delayTime int, concurrency int, ...) ([]string, []scanner.SecretMatched, []scanner.EndpointMatched, ...)
• func CreateColly(delayTime int, concurrency int, cache bool, timeout int, intensive bool, ...) *colly.Collector
• func EndpointsMatch(target string, endpointsFile []string) []scanner.EndpointMatched
• func ErrorsMatch(url string, body string) []scanner.ErrorMatched
• func GenerateRandomUserAgent() string
• func GetRequest(target string) (string, error)
• func HeadRequest(target string) (string, error)
• func IgnoreMatch(url string, ignoreSlice []string) bool
• func PostRequest(target string, data map[string]string) (string, error)
• func RetrieveBody(target string) string
• func SecretsMatch(url string, body string, secretsFile []string) []scanner.SecretMatched

Functions

func Crawler

func Crawler(target string, txt string, html string, delayTime int, concurrency int,
	ignore string, ignoreTxt string, cache bool, timeout int, intensive bool, rua bool,
	proxy string, secrets bool, secretsFile []string, plain bool, endpoints bool,
	endpointsFile []string, fileType int, headers map[string]string,
	errors bool) ([]string, []scanner.SecretMatched, []scanner.EndpointMatched,
	[]scanner.FileTypeMatched, []scanner.ErrorMatched)

Crawler it's the actual crawler engine. It controls all the behaviours of a scan (event handlers, secrets, errors, extensions and endpoints scanning)

func CreateColly

func CreateColly(delayTime int, concurrency int, cache bool, timeout int,
	intensive bool, rua bool, proxy string) *colly.Collector

CreateColly takes as input all the settings needed to instantiate a new Colly Collector object and it returns this object.

func EndpointsMatch

func EndpointsMatch(target string, endpointsFile []string) []scanner.EndpointMatched

EndpointsMatch check if an endpoint matches a juicy parameter

func ErrorsMatch

func ErrorsMatch(url string, body string) []scanner.ErrorMatched

ErrorsMatch hunts for extensions

func GenerateRandomUserAgent

func GenerateRandomUserAgent() string

GenerateRandomUserAgent generates a random user agent (can be Chrome or Firefox).

func GetRequest

func GetRequest(target string) (string, error)

GetRequest performs a GET request and return a string (the body of the response)

func HeadRequest

func HeadRequest(target string) (string, error)

HeadRequest performs a HEAD request and return a string (the headers of the response)

func IgnoreMatch

func IgnoreMatch(url string, ignoreSlice []string) bool

IgnoreMatch checks if the URL should be ignored or not.

func PostRequest

func PostRequest(target string, data map[string]string) (string, error)

PostRequest performs a POST request and return a string (the body of the response) the map in the input should contains the data fields and values in this way for example: { email: test@example.com, password: stupid_pwd }

func RetrieveBody

func RetrieveBody(target string) string

RetrieveBody retrieves the body (in the response) of a url

func SecretsMatch

func SecretsMatch(url string, body string, secretsFile []string) []scanner.SecretMatched

SecretsMatch checks if a body matches some secrets