scraper

Documentation

Index

• type Parameters
• type Scraper
  • func New(parameters Parameters) *Scraper
  • func (s *Scraper) GetWebsite(secure bool) string
  • func (s *Scraper) Log(v ...interface{})
  • func (s *Scraper) Scrape() ([]string, error)

Types

type Parameters

type Parameters struct {
	Website string

	MaxDepth int
	Timeout  int

	Recursively         bool
	Async               bool
	JS                  bool
	FollowExternalLinks bool
	Debug               bool

	// Objects to scrape
	Emails bool
}

Configuration for the scrape.

type Scraper

type Scraper struct {
	Parameters
}

Goal of the scrape.

func New

func New(parameters Parameters) *Scraper

Initiate new scraper.

func (*Scraper) GetWebsite

func (s *Scraper) GetWebsite(secure bool) string

func (*Scraper) Log

func (s *Scraper) Log(v ...interface{})

func (*Scraper) Scrape

func (s *Scraper) Scrape() ([]string, error)

Scrape is responsible for main scraping logic.