standard

Documentation

Overview

Package standard implements the functionality for a non-headless crawler. It uses net/http for making requests and goquery for scraping web page HTML.

Index

• type Crawler
  • func New(options *types.CrawlerOptions) (*Crawler, error)
  • func (c *Crawler) Close() error
  • func (c *Crawler) Crawl(rootURL string) error

Types

type Crawler

type Crawler struct {
	*common.Shared
}

Crawler is a standard crawler instance

func New

func New(options *types.CrawlerOptions) (*Crawler, error)

New returns a new standard crawler instance

func (*Crawler) Close

func (c *Crawler) Close() error

Close closes the crawler process

func (*Crawler) Crawl

func (c *Crawler) Crawl(rootURL string) error

Crawl crawls a URL with the specified options