xcrawl

README

xcrawl

"Because you have done this, you are more cursed than all the livestock, and more than all the earth's animals, You'll crawl on your belly and eat outdated URLs as long as you live." ~ Genesis 3:14

Xtensible crawler propagates through links with search constraints defined in yaml files

Documentation

Overview

Package crawl ... Is a web crawler

Index

• func StaticRequest(link string) (dom *stew.Stew, err error)
• type Crawler
  • func NewJson(jsonParams []byte) *Crawler
  • func NewYaml(ymlParams []byte) *Crawler
  • func (this *Crawler) Crawl(URI string, visited map[string]struct{})
  • func (this *Crawler) InjectRec(rec RecFunc)
  • func (this *Crawler) InjectReq(req ReqFunc)
• type PageInfo
• type RecFunc
• type ReqFunc
  • func GetDynamicRequest(execPath string) ReqFunc

Functions

func StaticRequest

func StaticRequest(link string) (dom *stew.Stew, err error)

StaticRequest ... Construct stew dom tree from page before javascript execution Default Request Injectable

Types

type Crawler

type Crawler struct {
	MaxDepth     uint     `yaml:"depth",json:"depth"`
	SameHost     bool     `yaml:"same_host",json:"sameHost"`
	ContainsTags []string `yaml:"contains_tags",json:"containsTags"`
	// contains filtered or unexported fields
}

Crawler ... Is the filter and record parameters

func NewJson

func NewJson(jsonParams []byte) *Crawler

NewJson ... Marshal json options as Crawler parameters

func NewYaml

func NewYaml(ymlParams []byte) *Crawler

NewYaml ... Marshal yaml options as Crawler parameters

func (*Crawler) Crawl

func (this *Crawler) Crawl(URI string, visited map[string]struct{})

Crawl ... Visits all pages starting from input URI

func (*Crawler) InjectRec

func (this *Crawler) InjectRec(rec RecFunc)

InjectRec ... Adds record functor

func (*Crawler) InjectReq

func (this *Crawler) InjectReq(req ReqFunc)

InjectReq ... Adds request functor

type PageInfo

type PageInfo struct {
	DOM  *stew.Stew
	Link string
	Refs map[string]struct{}
}

PageInfo ... Represents useful data for a single page

type RecFunc

type RecFunc func(*sync.WaitGroup, *PageInfo)

type ReqFunc

type ReqFunc func(string) (*stew.Stew, error)

func GetDynamicRequest

func GetDynamicRequest(execPath string) ReqFunc

GetDynamicRequest ... Returns DynamicRequest, which Construct stew dom from page after javascript execution