spider

Documentation

Overview

Package spider goes through the site tree and outputs the meta tree data consumable for reporters

Index

• type Progress
• type Result
• type Spider
  • func New(path, domains string) *Spider
  • func (spider *Spider) Crawl() (progress chan *Progress)
  • func (spider Spider) Get() (*Result, error)
  • func (spider *Spider) Validate() (err error)

Types

type Progress

type Progress struct {
	Data  *Result
	Error error
}

Progress intermediate data

type Result

type Result struct {
	Assets   map[string][]string `json:"assets"`
	URL      string              `json:"url"`
	Name     string              `json:"name"`
	Links    []string            `json:"links"`
	Broken   []string            `json:"broken"`
	Children []*Result           `json:"children"`
	// contains filtered or unexported fields
}

Result spider data that we eventually return

type Spider

type Spider struct {
	Result *Result
	Error  error

	Progress chan *Progress
	// contains filtered or unexported fields
}

Spider configuration

func New

func New(path, domains string) *Spider

New returns new instance of Spider

func (*Spider) Crawl

func (spider *Spider) Crawl() (progress chan *Progress)

Crawl visits the sites and collects data from them

func (Spider) Get

func (spider Spider) Get() (*Result, error)

Get final result

func (*Spider) Validate

func (spider *Spider) Validate() (err error)

Validate spider input