spider

Documentation

Overview

This file contains the http functions used by the scum spider

This file contains the spider struct and correlated functions for the scum program

Index

• func InitHTTP(followRedirects bool, proxy string, timeout int) error
• func Request(method string, URL string, agent string) (*http.Response, error)
• type Spider
  • func NewSpider(rootURL string, proxy string, agent string, rLevel int, threads int, ...) *Spider
  • func (s *Spider) Run()

Functions

func InitHTTP

func InitHTTP(followRedirects bool, proxy string, timeout int) error

InitHTTP initialises the http client

func Request

func Request(method string, URL string, agent string) (*http.Response, error)

Request will do the given Request with the previously initialised client and return the response

Types

type Spider

type Spider struct {
	// contains filtered or unexported fields
}

func NewSpider

func NewSpider(rootURL string,
	proxy string,
	agent string,
	rLevel int,
	threads int,
	includeSubs bool,
	includeEx bool,
	includeJS bool,
	includeCSS bool,
	includeIMG bool) *Spider

NewSpider will create a new Spider and return a pointer to the created struct

func (*Spider) Run

func (s *Spider) Run()

Run will run the scum webspider